developer_uid: rirv938
submission_id: chaiml-235b-sft-new-rm-_21184_v1
model_name: chaiml-235b-sft-new-rm-_21184_v1
model_group: ChaiML/235b_sft_new_rm_l
status: torndown
timestamp: 2026-02-19T07:21:47+00:00
num_battles: 11148
num_wins: 6081
celo_rating: 1330.09
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-new-rm-_21184_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-15
win_ratio: 0.5454790096878364
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'You:', '<|im_end|>', '###', '<|im_start|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-21184-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-21184-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-19484-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-19484-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-68639-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-68639-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-74662-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-74662-v1-uploader to finish
chaiml-235b-sft-new-rm-42002-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-19484-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-19484-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16...
chaiml-235b-sft-new-rm-21184-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-21184-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-21184-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-21184-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16...
chaiml-235b-sft-new-rm-74662-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-74662-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-74662-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-74662-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16...
chaiml-235b-sft-new-rm-68639-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-68639-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-68639-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-68639-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16...
chaiml-235b-sft-new-rm-42002-v1-uploader: Downloaded in 42.416s
chaiml-235b-sft-new-rm-42002-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 in 42.949s
chaiml-235b-sft-new-rm-42002-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloaded in 40.938s
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 in 41.624s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloaded in 41.633s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-68639-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 in 42.164s
chaiml-235b-sft-new-rm-42002-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloaded in 41.836s
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 in 42.376s
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-42002-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-42002-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloaded in 42.491s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/generation_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 in 43.074s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/.gitattributes
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/merges.txt
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/vocab.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/generation_config.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-new-rm-19484-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors?partNumber=74&uploadId=2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY-": write tcp 10.0.27.120:42634->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: ERROR "cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY- caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000858b68de70f88057-006992a26a-1579ff623c-default, host id:
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00001-of-00027.safetensors
Job chaiml-235b-sft-new-rm-42002-v1-uploader completed after 977.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-42002-v1-uploader
Pipeline stage VLLMUploader completed in 979.90s
run pipeline stage %s
Running pipeline stage VLLMTemplater
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00020-of-00027.safetensors
Pipeline stage VLLMTemplater completed in 0.70s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00024-of-00027.safetensors
run pipeline stage %s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00007-of-00027.safetensors
Running pipeline stage VLLMDeployer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00002-of-00027.safetensors
Creating inference service chaiml-235b-sft-new-rm-42002-v1
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00004-of-00027.safetensors
Waiting for inference service chaiml-235b-sft-new-rm-42002-v1 to be ready
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00017-of-00027.safetensors
Job chaiml-235b-sft-new-rm-68639-v1-uploader completed after 992.0s with status: succeeded
Job chaiml-235b-sft-new-rm-21184-v1-uploader completed after 1001.88s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-68639-v1-uploader
Stopping job with name chaiml-235b-sft-new-rm-21184-v1-uploader
Pipeline stage VLLMUploader completed in 994.71s
Pipeline stage VLLMUploader completed in 1004.07s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.74s
Job chaiml-235b-sft-new-rm-74662-v1-uploader completed after 990.29s with status: succeeded
Pipeline stage VLLMTemplater completed in 0.77s
run pipeline stage %s
Stopping job with name chaiml-235b-sft-new-rm-74662-v1-uploader
run pipeline stage %s
Running pipeline stage VLLMDeployer
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 993.75s
Creating inference service chaiml-235b-sft-new-rm-68639-v1
Creating inference service chaiml-235b-sft-new-rm-21184-v1
run pipeline stage %s
Waiting for inference service chaiml-235b-sft-new-rm-68639-v1 to be ready
Running pipeline stage VLLMTemplater
Waiting for inference service chaiml-235b-sft-new-rm-21184-v1 to be ready
Pipeline stage VLLMTemplater completed in 0.89s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-74662-v1
Waiting for inference service chaiml-235b-sft-new-rm-74662-v1 to be ready
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors
Job chaiml-235b-sft-new-rm-19484-v1-uploader completed after 1036.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-19484-v1-uploader
Pipeline stage VLLMUploader completed in 1039.33s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.75s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-19484-v1
Waiting for inference service chaiml-235b-sft-new-rm-19484-v1 to be ready
Inference service chaiml-235b-sft-new-rm-21184-v1 ready after 284.21195912361145s
Pipeline stage VLLMDeployer completed in 287.49s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.110910654067993s
Received healthy response to inference request in 2.182664394378662s
Received healthy response to inference request in 2.320479154586792s
Received healthy response to inference request in 2.2272443771362305s
Received healthy response to inference request in 2.196024179458618s
Received healthy response to inference request in 2.432786464691162s
Received healthy response to inference request in 2.1420376300811768s
Received healthy response to inference request in 2.075230598449707s
Received healthy response to inference request in 2.3608663082122803s
Received healthy response to inference request in 2.5839426517486572s
Received healthy response to inference request in 1.9517178535461426s
Received healthy response to inference request in 2.15575909614563s
Received healthy response to inference request in 2.4736087322235107s
Received healthy response to inference request in 2.333919048309326s
Received healthy response to inference request in 2.1538188457489014s
Received healthy response to inference request in 2.025940179824829s
Received healthy response to inference request in 2.0790276527404785s
Received healthy response to inference request in 2.042165756225586s
Received healthy response to inference request in 2.234696865081787s
Received healthy response to inference request in 2.1628029346466064s
Received healthy response to inference request in 2.5693507194519043s
Received healthy response to inference request in 2.2414157390594482s
Received healthy response to inference request in 1.9806458950042725s
Received healthy response to inference request in 2.428105592727661s
Received healthy response to inference request in 2.4217584133148193s
Received healthy response to inference request in 1.9162476062774658s
Received healthy response to inference request in 2.3511404991149902s
Received healthy response to inference request in 2.0151288509368896s
Received healthy response to inference request in 2.1614203453063965s
Received healthy response to inference request in 2.088926076889038s
30 requests
0 failed requests
5th percentile: 1.964735472202301
10th percentile: 2.011680555343628
20th percentile: 2.0686176300048826
30th percentile: 2.126104164123535
40th percentile: 2.15915584564209
50th percentile: 2.18934428691864
60th percentile: 2.2373844146728517
70th percentile: 2.3390854835510253
80th percentile: 2.423027849197388
90th percentile: 2.4831829309463505
95th percentile: 2.5773762822151185
99th percentile: 2.958089933395386
mean time: 2.247326103846232
Pipeline stage StressChecker completed in 101.54s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 4.31s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_21184_v1 status is now deployed due to DeploymentManager action
chaiml-235b-sft-new-rm-_21184_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-235b-sft-new-rm-_21184_v1 status is now torndown due to DeploymentManager action