chaiml-muster-v0-q235b-_52842

developer_uid: zonemercy
submission_id: chaiml-muster-v0-q235b-_52842_v3
model_name: chaiml-muster-v0-q235b-_52842_v3
model_group: ChaiML/muster-v0-q235b-l
status: torndown
timestamp: 2026-02-08T14:41:28+00:00
num_battles: 12042
num_wins: 6412
celo_rating: 1326.42
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/muster-v0-q235b-lr1e4ep2r64g4
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 72
reward_model: chaiml-prm-kimi-v1-300k_92220_v7
display_name: chaiml-muster-v0-q235b-_52842_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/muster-v0-q235b-lr1e4ep2r64g4
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-04
win_ratio: 0.5324696894203621
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '</think>', '</s>', '<|user|>', '<|im_end|>', '<|assistant|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 72}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-muster-v0-q235b-52842-v3-uploader
Waiting for job on chaiml-muster-v0-q235b-52842-v3-uploader to finish
chaiml-muster-v0-q235b-52842-v3-uploader: Using quantization_mode: w4a16
chaiml-muster-v0-q235b-52842-v3-uploader: Checking if ChaiML/muster-v0-q235b-lr1e4ep2r64g4-W4A16 already exists in ChaiML
chaiml-muster-v0-q235b-52842-v3-uploader: 
Fetching 39 files:   0%|          | 0/39 [00:00<?, ?it/s]
Fetching 39 files:   3%|▎         | 1/39 [00:00<00:08,  4.24it/s]
Fetching 39 files:  15%|█▌        | 6/39 [00:00<00:01, 17.03it/s]
Fetching 39 files:  21%|██        | 8/39 [00:19<01:39,  3.20s/it]
Fetching 39 files:  26%|██▌       | 10/39 [00:19<01:04,  2.22s/it]
Fetching 39 files:  26%|██▌       | 10/39 [00:30<01:04,  2.22s/it]
Fetching 39 files:  38%|███▊      | 15/39 [00:34<01:01,  2.58s/it]
Fetching 39 files:  41%|████      | 16/39 [00:34<00:51,  2.26s/it]
Fetching 39 files:  46%|████▌     | 18/39 [00:34<00:36,  1.72s/it]
Fetching 39 files:  51%|█████▏    | 20/39 [00:39<00:34,  1.83s/it]
Fetching 39 files:  59%|█████▉    | 23/39 [00:50<00:40,  2.55s/it]
Fetching 39 files:  62%|██████▏   | 24/39 [00:52<00:37,  2.49s/it]
Fetching 39 files:  64%|██████▍   | 25/39 [00:52<00:29,  2.09s/it]
Fetching 39 files:  67%|██████▋   | 26/39 [00:54<00:26,  2.02s/it]
Fetching 39 files:  72%|███████▏  | 28/39 [00:57<00:21,  1.92s/it]
Fetching 39 files:  74%|███████▍  | 29/39 [00:58<00:15,  1.59s/it]
Fetching 39 files:  79%|███████▉  | 31/39 [01:02<00:13,  1.72s/it]
Fetching 39 files: 100%|██████████| 39/39 [01:02<00:00,  1.59s/it]
chaiml-muster-v0-q235b-52842-v3-uploader: Downloaded in 62.292s
chaiml-muster-v0-q235b-52842-v3-uploader: Processed model ChaiML/muster-v0-q235b-lr1e4ep2r64g4 in 63.048s
chaiml-muster-v0-q235b-52842-v3-uploader: creating bucket guanaco-vllm-models
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-muster-v0-q235b-52842-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-muster-v0-q235b-52842-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-muster-v0-q235b-52842-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-muster-v0-q235b-52842-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-muster-v0-q235b-52842-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-muster-v0-q235b-52842-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/special_tokens_map.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/chat_template.jinja
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/.gitattributes
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/tokenizer_config.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/generation_config.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/added_tokens.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/quantization_config.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/config.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/merges.txt
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model.safetensors.index.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/vocab.json
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/tokenizer.json
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-muster-v0-q235b-52842-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00006-of-00027.safetensors?partNumber=2&uploadId=2~rvHZJmaNhaVr_W7vn4bWjjV6A7GXSbv": write tcp 10.2.139.9:45236->216.153.53.63:443: write: connection reset by peer
chaiml-muster-v0-q235b-52842-v3-uploader: ERROR "cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00006-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~rvHZJmaNhaVr_W7vn4bWjjV6A7GXSbv caused by: SignatureDoesNotMatch:  status code: 403, request id: tx0000035fdca4679df468e-006982e39d-14a24de22e-default, host id:
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00013-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00001-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00023-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00005-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00022-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00015-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00004-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00016-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00018-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00009-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00025-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00026-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00002-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00017-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00008-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00024-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00020-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00019-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00011-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00003-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00012-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00007-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v3-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/.gitattributes": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/added_tokens.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/chat_template.jinja": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/config.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/generation_config.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/merges.txt": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00001-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00002-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00003-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00004-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00005-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00007-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00008-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00009-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00010-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00011-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00012-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00013-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00014-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00015-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00016-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00017-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00018-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00019-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00020-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00021-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00022-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00023-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00024-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00025-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00026-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00027-of-00027.safetensors": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model.safetensors.index.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/quantization_config.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/special_tokens_map.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/tokenizer.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/tokenizer_config.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/vocab.json": object size matches
chaiml-muster-v0-q235b-52842-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v3/model-00006-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
Job chaiml-muster-v0-q235b-52842-v3-uploader completed after 496.49s with status: succeeded
Stopping job with name chaiml-muster-v0-q235b-52842-v3-uploader
Pipeline stage VLLMUploader completed in 497.28s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-muster-v0-q235b-52842-v3
Waiting for inference service chaiml-muster-v0-q235b-52842-v3 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-muster-v0-q235b-52842-v3 ready after 580.4140622615814s
Pipeline stage VLLMDeployer completed in 580.87s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8816051483154297s
Received healthy response to inference request in 1.9329454898834229s
Received healthy response to inference request in 1.8904502391815186s
Received healthy response to inference request in 1.8619868755340576s
Received healthy response to inference request in 1.7319347858428955s
Received healthy response to inference request in 1.9098079204559326s
Received healthy response to inference request in 1.7304670810699463s
Received healthy response to inference request in 1.84016752243042s
Received healthy response to inference request in 1.7397480010986328s
Received healthy response to inference request in 1.7628114223480225s
Received healthy response to inference request in 1.7165660858154297s
Received healthy response to inference request in 1.7866580486297607s
Received healthy response to inference request in 1.7689085006713867s
Received healthy response to inference request in 1.7930877208709717s
Received healthy response to inference request in 1.9524621963500977s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.8361599445343018s
Received healthy response to inference request in 1.7773208618164062s
Received healthy response to inference request in 1.8159775733947754s
Received healthy response to inference request in 1.8567800521850586s
Received healthy response to inference request in 1.7304065227508545s
Received healthy response to inference request in 1.7568793296813965s
Received healthy response to inference request in 1.8403973579406738s
Received healthy response to inference request in 1.732198715209961s
Received healthy response to inference request in 1.7954039573669434s
Received healthy response to inference request in 1.8028461933135986s
Received healthy response to inference request in 1.7763421535491943s
Received healthy response to inference request in 1.889521598815918s
Received healthy response to inference request in 1.7862861156463623s
Received healthy response to inference request in 1.986750602722168s
Received healthy response to inference request in 1.7954998016357422s
30 requests
0 failed requests
5th percentile: 1.7304337739944458
10th percentile: 1.7317880153656007
20th percentile: 1.7534530639648438
30th percentile: 1.774112057685852
40th percentile: 1.7865092754364014
50th percentile: 1.7954518795013428
60th percentile: 1.8240505218505858
70th percentile: 1.8453121662139893
80th percentile: 1.8831884384155273
90th percentile: 1.9121216773986818
95th percentile: 1.943679678440094
99th percentile: 1.9768069648742677
mean time: 1.8159459273020426
Pipeline stage StressChecker completed in 58.40s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-muster-v0-q235b-_52842_v3 status is now deployed due to DeploymentManager action
chaiml-muster-v0-q235b-_52842_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-muster-v0-q235b-_52842_v3 status is now torndown due to DeploymentManager action