chaiml-235b-sft-new-rm-_74662

developer_uid: rirv938
submission_id: chaiml-235b-sft-new-rm-_74662_v1
model_name: chaiml-235b-sft-new-rm-_74662_v1
model_group: ChaiML/235b_sft_new_rm_l
status: torndown
timestamp: 2026-02-19T07:32:03+00:00
num_battles: 11311
num_wins: 6228
celo_rating: 1333.71
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-235b-sft-new-rm-_74662_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-15
win_ratio: 0.5506144461144019
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|im_start|>', '</s>', '###', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-235b-sft-new-rm-74662-v1-uploader
Waiting for job on chaiml-235b-sft-new-rm-74662-v1-uploader to finish
chaiml-235b-sft-new-rm-42002-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-19484-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-19484-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-19484-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16...
chaiml-235b-sft-new-rm-21184-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-21184-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-21184-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-21184-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16...
chaiml-235b-sft-new-rm-74662-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-74662-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-74662-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-74662-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16...
chaiml-235b-sft-new-rm-68639-v1-uploader: Using quantization_mode: w4a16
chaiml-235b-sft-new-rm-68639-v1-uploader: Repo ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already ends in W4A16. Skipping...
chaiml-235b-sft-new-rm-68639-v1-uploader: Checking if ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 already exists in ChaiML
chaiml-235b-sft-new-rm-68639-v1-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloading snapshot of ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16...
chaiml-235b-sft-new-rm-42002-v1-uploader: Downloaded in 42.416s
chaiml-235b-sft-new-rm-42002-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e1-W4A16 in 42.949s
chaiml-235b-sft-new-rm-42002-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: Downloaded in 40.938s
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e3-W4A16 in 41.624s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: Downloaded in 41.633s
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-68639-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e4-W4A16 in 42.164s
chaiml-235b-sft-new-rm-42002-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: Downloaded in 41.836s
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-68639-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e2-W4A16 in 42.376s
chaiml-235b-sft-new-rm-19484-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-42002-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-42002-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-19484-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-19484-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-19484-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-74662-v1-uploader: Downloaded in 42.491s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/generation_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader: Processed model ChaiML/235b_sft_new_rm_lexical_10k_e5-W4A16 in 43.074s
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/.gitattributes
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: creating bucket guanaco-vllm-models
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/config.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/merges.txt
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/vocab.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-21184-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-235b-sft-new-rm-21184-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default
chaiml-235b-sft-new-rm-74662-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/generation_config.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-74662-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model.safetensors.index.json
chaiml-235b-sft-new-rm-74662-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer_config.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/added_tokens.json
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/chat_template.jinja
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/vocab.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/generation_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/special_tokens_map.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/.gitattributes
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/merges.txt
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/quantization_config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/config.json
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/tokenizer.json
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00027-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-235b-sft-new-rm-19484-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors?partNumber=74&uploadId=2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY-": write tcp 10.0.27.120:42634->216.153.53.63:443: write: connection reset by peer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00020-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: ERROR "cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~KnzMM_iHrH6pMtMPh-RsoJxVK81EBY- caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000858b68de70f88057-006992a26a-1579ff623c-default, host id:
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-42002-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-42002-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00003-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00015-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00011-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00001-of-00027.safetensors
Job chaiml-235b-sft-new-rm-42002-v1-uploader completed after 977.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-42002-v1-uploader
Pipeline stage VLLMUploader completed in 979.90s
run pipeline stage %s
Running pipeline stage VLLMTemplater
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00020-of-00027.safetensors
Pipeline stage VLLMTemplater completed in 0.70s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00024-of-00027.safetensors
run pipeline stage %s
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00007-of-00027.safetensors
Running pipeline stage VLLMDeployer
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00002-of-00027.safetensors
Creating inference service chaiml-235b-sft-new-rm-42002-v1
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00025-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00004-of-00027.safetensors
Waiting for inference service chaiml-235b-sft-new-rm-42002-v1 to be ready
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00013-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00002-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00010-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00001-of-00027.safetensors
chaiml-235b-sft-new-rm-68639-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-68639-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00009-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00017-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00021-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00016-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00019-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00018-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00004-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00026-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00006-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00024-of-00027.safetensors
chaiml-235b-sft-new-rm-74662-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-74662-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00008-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00012-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00023-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00005-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00014-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00022-of-00027.safetensors
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00007-of-00027.safetensors
chaiml-235b-sft-new-rm-21184-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-21184-v1/default/model-00017-of-00027.safetensors
Job chaiml-235b-sft-new-rm-68639-v1-uploader completed after 992.0s with status: succeeded
Job chaiml-235b-sft-new-rm-21184-v1-uploader completed after 1001.88s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-68639-v1-uploader
Stopping job with name chaiml-235b-sft-new-rm-21184-v1-uploader
Pipeline stage VLLMUploader completed in 994.71s
Pipeline stage VLLMUploader completed in 1004.07s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.74s
Job chaiml-235b-sft-new-rm-74662-v1-uploader completed after 990.29s with status: succeeded
Pipeline stage VLLMTemplater completed in 0.77s
run pipeline stage %s
Stopping job with name chaiml-235b-sft-new-rm-74662-v1-uploader
run pipeline stage %s
Running pipeline stage VLLMDeployer
Running pipeline stage VLLMDeployer
Pipeline stage VLLMUploader completed in 993.75s
Creating inference service chaiml-235b-sft-new-rm-68639-v1
Creating inference service chaiml-235b-sft-new-rm-21184-v1
run pipeline stage %s
Waiting for inference service chaiml-235b-sft-new-rm-68639-v1 to be ready
Running pipeline stage VLLMTemplater
Waiting for inference service chaiml-235b-sft-new-rm-21184-v1 to be ready
Pipeline stage VLLMTemplater completed in 0.89s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-74662-v1
Waiting for inference service chaiml-235b-sft-new-rm-74662-v1 to be ready
chaiml-235b-sft-new-rm-19484-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-235b-sft-new-rm-19484-v1/default/model-00013-of-00027.safetensors
Job chaiml-235b-sft-new-rm-19484-v1-uploader completed after 1036.9s with status: succeeded
Stopping job with name chaiml-235b-sft-new-rm-19484-v1-uploader
Pipeline stage VLLMUploader completed in 1039.33s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.75s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-235b-sft-new-rm-19484-v1
Waiting for inference service chaiml-235b-sft-new-rm-19484-v1 to be ready
Inference service chaiml-235b-sft-new-rm-21184-v1 ready after 284.21195912361145s
Pipeline stage VLLMDeployer completed in 287.49s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.110910654067993s
Received healthy response to inference request in 2.182664394378662s
Received healthy response to inference request in 2.320479154586792s
Received healthy response to inference request in 2.2272443771362305s
Received healthy response to inference request in 2.196024179458618s
Received healthy response to inference request in 2.432786464691162s
Received healthy response to inference request in 2.1420376300811768s
Received healthy response to inference request in 2.075230598449707s
Received healthy response to inference request in 2.3608663082122803s
Received healthy response to inference request in 2.5839426517486572s
Received healthy response to inference request in 1.9517178535461426s
Received healthy response to inference request in 2.15575909614563s
Received healthy response to inference request in 2.4736087322235107s
Received healthy response to inference request in 2.333919048309326s
Received healthy response to inference request in 2.1538188457489014s
Received healthy response to inference request in 2.025940179824829s
Received healthy response to inference request in 2.0790276527404785s
Received healthy response to inference request in 2.042165756225586s
Received healthy response to inference request in 2.234696865081787s
Received healthy response to inference request in 2.1628029346466064s
Received healthy response to inference request in 2.5693507194519043s
Received healthy response to inference request in 2.2414157390594482s
Received healthy response to inference request in 1.9806458950042725s
Received healthy response to inference request in 2.428105592727661s
Received healthy response to inference request in 2.4217584133148193s
Received healthy response to inference request in 1.9162476062774658s
Received healthy response to inference request in 2.3511404991149902s
Received healthy response to inference request in 2.0151288509368896s
Received healthy response to inference request in 2.1614203453063965s
Received healthy response to inference request in 2.088926076889038s
30 requests
0 failed requests
5th percentile: 1.964735472202301
10th percentile: 2.011680555343628
20th percentile: 2.0686176300048826
30th percentile: 2.126104164123535
40th percentile: 2.15915584564209
50th percentile: 2.18934428691864
60th percentile: 2.2373844146728517
70th percentile: 2.3390854835510253
80th percentile: 2.423027849197388
90th percentile: 2.4831829309463505
95th percentile: 2.5773762822151185
99th percentile: 2.958089933395386
mean time: 2.247326103846232
Pipeline stage StressChecker completed in 101.54s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 4.31s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_21184_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-235b-sft-new-rm-42002-v1 ready after 994.670592546463s
Pipeline stage VLLMDeployer completed in 998.02s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.0977299213409424s
Received healthy response to inference request in 2.399458408355713s
Received healthy response to inference request in 2.5763421058654785s
Received healthy response to inference request in 2.858030319213867s
Received healthy response to inference request in 2.44118595123291s
Received healthy response to inference request in 2.8757450580596924s
Received healthy response to inference request in 2.8443284034729004s
Received healthy response to inference request in 2.5446155071258545s
Received healthy response to inference request in 2.5604970455169678s
Received healthy response to inference request in 2.429673910140991s
Received healthy response to inference request in 2.3868467807769775s
Received healthy response to inference request in 2.496626853942871s
Received healthy response to inference request in 2.4134621620178223s
Received healthy response to inference request in 2.3911209106445312s
Received healthy response to inference request in 2.56704044342041s
Received healthy response to inference request in 2.476635217666626s
Received healthy response to inference request in 2.3877668380737305s
Received healthy response to inference request in 2.4317421913146973s
Received healthy response to inference request in 2.619509696960449s
Received healthy response to inference request in 2.391740083694458s
Received healthy response to inference request in 2.451599597930908s
Received healthy response to inference request in 2.3814611434936523s
Received healthy response to inference request in 2.496241807937622s
Received healthy response to inference request in 2.6144020557403564s
Received healthy response to inference request in 2.5028927326202393s
Received healthy response to inference request in 2.4484102725982666s
Received healthy response to inference request in 2.4050216674804688s
Received healthy response to inference request in 2.511706590652466s
Received healthy response to inference request in 2.409762144088745s
Received healthy response to inference request in 2.383235454559326s
30 requests
0 failed requests
5th percentile: 2.3848605513572694
10th percentile: 2.387674832344055
20th percentile: 2.397914743423462
30th percentile: 2.4123521566390993
40th percentile: 2.437408447265625
50th percentile: 2.464117407798767
60th percentile: 2.4991332054138184
70th percentile: 2.5493799686431884
80th percentile: 2.583954095840454
90th percentile: 2.845698595046997
95th percentile: 2.867773425579071
99th percentile: 3.03335431098938
mean time: 2.5264943758646647
Pipeline stage StressChecker completed in 112.65s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.81s
Shutdown handler de-registered
chaiml-235b-sft-new-rm-_42002_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-235b-sft-new-rm-74662-v1 ready after 1094.186842918396s
Pipeline stage VLLMDeployer completed in 1096.88s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.643380641937256s
Received healthy response to inference request in 2.374591588973999s
Inference service chaiml-235b-sft-new-rm-68639-v1 ready after 1106.5658824443817s
Pipeline stage VLLMDeployer completed in 1109.31s
Received healthy response to inference request in 2.337110757827759s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9210622310638428s
Received healthy response to inference request in 2.5777587890625s
Received healthy response to inference request in 2.569279193878174s
Received healthy response to inference request in 2.4409821033477783s
Received healthy response to inference request in 2.4259932041168213s
Received healthy response to inference request in 2.349668264389038s
Received healthy response to inference request in 2.4582436084747314s
Received healthy response to inference request in 2.280062675476074s
Received healthy response to inference request in 2.466123580932617s
Received healthy response to inference request in 2.5523171424865723s
Received healthy response to inference request in 2.5020205974578857s
Received healthy response to inference request in 2.3359944820404053s
Received healthy response to inference request in 2.4120028018951416s
Received healthy response to inference request in 2.344644546508789s
Received healthy response to inference request in 2.406651735305786s
Received healthy response to inference request in 2.372013807296753s
Received healthy response to inference request in 2.523146152496338s
Received healthy response to inference request in 2.3930835723876953s
Received healthy response to inference request in 2.4304440021514893s
Received healthy response to inference request in 2.4950525760650635s
Received healthy response to inference request in 2.533607244491577s
Received healthy response to inference request in 2.3914802074432373s
Received healthy response to inference request in 2.3361949920654297s
Received healthy response to inference request in 2.3968145847320557s
Received healthy response to inference request in 2.398986339569092s
Received healthy response to inference request in 2.3470561504364014s
Received healthy response to inference request in 2.3250985145568848s
Received healthy response to inference request in 2.4120969772338867s
Received healthy response to inference request in 2.5817148685455322s
Received healthy response to inference request in 2.3362741470336914s
Received healthy response to inference request in 2.4230263233184814s
Received healthy response to inference request in 2.728562831878662s
Received healthy response to inference request in 2.31996750831604s
Received healthy response to inference request in 2.4035210609436035s
Received healthy response to inference request in 2.3654749393463135s
Received healthy response to inference request in 2.378087043762207s
Received healthy response to inference request in 2.481503963470459s
Received healthy response to inference request in 2.489112138748169s
Received healthy response to inference request in 2.7940869331359863s
Received healthy response to inference request in 2.5736591815948486s
Received healthy response to inference request in 2.4851763248443604s
Received healthy response to inference request in 2.8361728191375732s
Received healthy response to inference request in 2.5728437900543213s
Received healthy response to inference request in 2.4424192905426025s
Received healthy response to inference request in 2.3410980701446533s
Received healthy response to inference request in 2.510648727416992s
Received healthy response to inference request in 2.506194591522217s
Received healthy response to inference request in 2.505237102508545s
Received healthy response to inference request in 2.33840012550354s
Received healthy response to inference request in 2.3653564453125s
Received healthy response to inference request in 2.352787971496582s
Received healthy response to inference request in 2.436540365219116s
Received healthy response to inference request in 2.316542863845825s
Received healthy response to inference request in 2.4103496074676514s
30 requests
0 failed requests
5th percentile: 2.33009192943573
10th percentile: 2.337019181251526
Received healthy response to inference request in 2.4209046363830566s
20th percentile: 2.372768259048462
30th percentile: 2.409240245819092
40th percentile: 2.4248064517974854
50th percentile: 2.447391986846924
60th percentile: 2.4829729080200194
70th percentile: 2.506860589981079
80th percentile: 2.5407416343688967
90th percentile: 2.5878814458847046
Received healthy response to inference request in 2.7723793983459473s
95th percentile: 2.7262691020965573
99th percentile: 2.8842393946647644
mean time: 2.477195469538371
Pipeline stage StressChecker completed in 99.11s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 2.4736390113830566s
run_pipeline:run_in_cloud %s
30 requests
starting trigger_guanaco_pipeline args=%s
0 failed requests
5th percentile: 2.325296092033386
triggered trigger_guanaco_pipeline args=%s
10th percentile: 2.3362461805343626
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.02s
20th percentile: 2.343935251235962
Shutdown handler de-registered
30th percentile: 2.351852059364319
chaiml-235b-sft-new-rm-_74662_v1 status is now deployed due to DeploymentManager action
chaiml-235b-sft-new-rm-_74662_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-235b-sft-new-rm-_74662_v1 status is now torndown due to DeploymentManager action