chaiml-muster-v2-q235b-_77862

developer_uid: zonemercy
submission_id: chaiml-muster-v2-q235b-_77862_v2
model_name: chaiml-muster-v2-q235b-_77862_v2
model_group: ChaiML/muster-v2-q235b-l
status: inactive
timestamp: 2026-02-10T15:26:54+00:00
num_battles: 11534
num_wins: 6177
celo_rating: 1332.12
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/muster-v2-q235b-lr1e4ep2r64g4
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-muster-v2-q235b-_77862_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/muster-v2-q235b-lr1e4ep2r64g4
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-10
win_ratio: 0.5355470782035721
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '<|assistant|>', '<|user|>', '<|im_end|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-muster-v2-q235b-77862-v2-uploader
Waiting for job on chaiml-muster-v2-q235b-77862-v2-uploader to finish
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
chaiml-muster-v2-q235b-77862-v2-uploader: 
Fetching 39 files:   0%|          | 0/39 [00:00<?, ?it/s]
Fetching 39 files:   3%|▎         | 1/39 [00:00<00:10,  3.51it/s]
Fetching 39 files:  15%|█▌        | 6/39 [00:00<00:01, 18.73it/s]
Fetching 39 files:  15%|█▌        | 6/39 [00:19<00:01, 18.73it/s]
Fetching 39 files:  18%|█▊        | 7/39 [00:32<03:28,  6.52s/it]
Fetching 39 files:  21%|██        | 8/39 [01:59<12:35, 24.38s/it]
Fetching 39 files:  54%|█████▍    | 21/39 [02:02<01:28,  4.92s/it]
Fetching 39 files:  64%|██████▍   | 25/39 [02:17<01:05,  4.65s/it]
Fetching 39 files:  67%|██████▋   | 26/39 [02:18<00:55,  4.30s/it]
Fetching 39 files:  72%|███████▏  | 28/39 [02:22<00:42,  3.87s/it]
Fetching 39 files:  74%|███████▍  | 29/39 [02:28<00:41,  4.15s/it]
Fetching 39 files:  77%|███████▋  | 30/39 [02:29<00:33,  3.69s/it]
Fetching 39 files: 100%|██████████| 39/39 [02:29<00:00,  3.84s/it]
chaiml-muster-v2-q235b-77862-v2-uploader: Downloaded in 149.832s
chaiml-muster-v2-q235b-77862-v2-uploader: Processed model ChaiML/muster-v2-q235b-lr1e4ep2r64g4 in 150.372s
chaiml-muster-v2-q235b-77862-v2-uploader: creating bucket guanaco-vllm-models
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v2-q235b-77862-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-muster-v2-q235b-77862-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v2-q235b-77862-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v2-q235b-77862-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v2-q235b-77862-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v2-q235b-77862-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-muster-v2-q235b-77862-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-muster-v2-q235b-77862-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-muster-v2-q235b-77862-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-muster-v2-q235b-77862-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-muster-v2-q235b-77862-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/added_tokens.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/chat_template.jinja
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/special_tokens_map.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/tokenizer_config.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/generation_config.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/quantization_config.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/.gitattributes
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/config.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/merges.txt
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/vocab.json
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model.safetensors.index.json
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/tokenizer.json
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Retrying (%r) after connection broken by '%r': %s
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG retryable error: ServiceUnavailable: Service Unavailable
chaiml-muster-v2-q235b-77862-v2-uploader: 	status code: 503, request id: , host id: 
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG retryable error: ServiceUnavailable: Service Unavailable
chaiml-muster-v2-q235b-77862-v2-uploader: 	status code: 503, request id: , host id: 
chaiml-muster-v2-q235b-77862-v2-uploader: ERROR "cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00009-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~LS2xlaZoa8zRI_rW3819sdos7Effj8R caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000007d95755cc8444cf1-00698b219d-14a26732ad-default, host id:
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v2-q235b-77862-v2-uploader: ERROR "cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00024-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~OQuKMzhROip3Vw4MaGr3Iftjwc90H2x caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000002e4051ebc88424f5-00698b2193-14a26732ad-default, host id:
HTTP Request: %s %s "%s %d %s"
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00027-of-00027.safetensors
Failed to get response for submission blend_honas_2026-02-03: ('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-2a6f-69d4-linear-w01_v7/predict', '{"detail":"1 validation error for RuntimeResponse\\npredictions\\n  Field required [type=missing, input_value={\'detail\': \\"503, message=...linear-w01_v7/predict\'\\"}, input_type=dict]\\n    For further information visit https://errors.pydantic.dev/2.12/v/missing"}')
HTTP Request: %s %s "%s %d %s"
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00011-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00012-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00015-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00006-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00021-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00013-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00014-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00022-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00004-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00017-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00019-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00002-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00020-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00005-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00003-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00025-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00026-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00010-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00008-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00023-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00001-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00007-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00016-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00018-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/.gitattributes": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/added_tokens.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/chat_template.jinja": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/config.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/generation_config.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/merges.txt": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00001-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00002-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00003-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00004-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00005-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00006-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00007-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00008-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00010-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00011-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00012-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00013-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00014-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00015-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00016-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00017-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00018-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00019-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00020-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00021-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00022-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00023-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00025-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00026-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00027-of-00027.safetensors": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model.safetensors.index.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/quantization_config.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/special_tokens_map.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/tokenizer.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/tokenizer_config.json": object size matches
chaiml-muster-v2-q235b-77862-v2-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/vocab.json": object size matches
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00009-of-00027.safetensors
chaiml-muster-v2-q235b-77862-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v2-q235b-77862-v2/model-00024-of-00027.safetensors
Job chaiml-muster-v2-q235b-77862-v2-uploader completed after 5096.34s with status: succeeded
Stopping job with name chaiml-muster-v2-q235b-77862-v2-uploader
Pipeline stage VLLMUploader completed in 5096.96s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-muster-v2-q235b-77862-v2
Waiting for inference service chaiml-muster-v2-q235b-77862-v2 to be ready
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-muster-v2-q235b-77862-v2 ready after 640.3983697891235s
Pipeline stage VLLMDeployer completed in 641.69s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2205357551574707s
Received healthy response to inference request in 2.3725056648254395s
Received healthy response to inference request in 2.117520809173584s
Received healthy response to inference request in 2.014451265335083s
Received healthy response to inference request in 2.2203900814056396s
Received healthy response to inference request in 2.4447572231292725s
Received healthy response to inference request in 1.9870736598968506s
Received healthy response to inference request in 1.99845552444458s
Received healthy response to inference request in 2.41432785987854s
Received healthy response to inference request in 2.44562029838562s
Received healthy response to inference request in 2.2016117572784424s
Received healthy response to inference request in 2.054523229598999s
Received healthy response to inference request in 2.6230063438415527s
read tcp 127.0.0.1:42108->127.0.0.1:8080: read: connection reset by peer
Received unhealthy response to inference request!
Received healthy response to inference request in 2.011765480041504s
Received healthy response to inference request in 1.9323747158050537s
Received healthy response to inference request in 2.252962827682495s
Received healthy response to inference request in 2.4008657932281494s
Received healthy response to inference request in 2.0785934925079346s
Received healthy response to inference request in 2.3858697414398193s
Received healthy response to inference request in 1.9657642841339111s
Received healthy response to inference request in 2.38281512260437s
Received healthy response to inference request in 1.964522361755371s
Received healthy response to inference request in 2.034196376800537s
Received healthy response to inference request in 2.1497256755828857s
Received healthy response to inference request in 2.3143935203552246s
Received healthy response to inference request in 1.997626543045044s
Received healthy response to inference request in 1.9861083030700684s
Received healthy response to inference request in 2.222071409225464s
Received healthy response to inference request in 2.3235208988189697s
30 requests
1 failed requests
5th percentile: 1.9468411564826966
10th percentile: 1.965640091896057
20th percentile: 1.9955159664154052
30th percentile: 2.0136455297470093
40th percentile: 2.0689653873443605
50th percentile: 2.175668716430664
60th percentile: 2.221150016784668
70th percentile: 2.3171317338943482
80th percentile: 2.38342604637146
90th percentile: 2.4173707962036133
95th percentile: 2.4452319145202637
99th percentile: 2.5715643906593324
mean time: 2.1248135805130004
%s, retrying in %s seconds...
Received healthy response to inference request in 2.0329959392547607s
Received healthy response to inference request in 2.259993076324463s
Received healthy response to inference request in 2.21954345703125s
Received healthy response to inference request in 2.1374149322509766s
Received healthy response to inference request in 1.9747719764709473s
Received healthy response to inference request in 2.195943593978882s
Received healthy response to inference request in 2.0812177658081055s
Received healthy response to inference request in 2.4862985610961914s
Received healthy response to inference request in 2.2827601432800293s
Received healthy response to inference request in 2.142066478729248s
Received healthy response to inference request in 1.9534368515014648s
Received healthy response to inference request in 2.6578352451324463s
Received healthy response to inference request in 2.5154190063476562s
Received healthy response to inference request in 2.0078635215759277s
Received healthy response to inference request in 1.9621975421905518s
Received healthy response to inference request in 4.23717737197876s
Received healthy response to inference request in 2.208422899246216s
Received healthy response to inference request in 2.0441207885742188s
Received healthy response to inference request in 2.0976943969726562s
Received healthy response to inference request in 1.9529316425323486s
Received healthy response to inference request in 2.1167328357696533s
Received healthy response to inference request in 2.071767807006836s
Received healthy response to inference request in 2.194859743118286s
Received healthy response to inference request in 2.090059518814087s
Received healthy response to inference request in 2.0801100730895996s
Received healthy response to inference request in 2.186530351638794s
Received healthy response to inference request in 2.0876879692077637s
Received healthy response to inference request in 2.1186881065368652s
Received healthy response to inference request in 2.1000678539276123s
Received healthy response to inference request in 2.204782485961914s
30 requests
0 failed requests
5th percentile: 1.957379162311554
10th percentile: 1.9735145330429078
20th percentile: 2.0418958187103273
30th percentile: 2.0808854579925535
40th percentile: 2.0946404457092287
50th percentile: 2.1177104711532593
60th percentile: 2.1598520278930664
70th percentile: 2.1985952615737916
80th percentile: 2.2276333808898925
90th percentile: 2.489210605621338
95th percentile: 2.5937479376792902
99th percentile: 3.77916815519333
mean time: 2.2233797311782837
Pipeline stage StressChecker completed in 139.47s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.58s
Shutdown handler de-registered
chaiml-muster-v2-q235b-_77862_v2 status is now deployed due to DeploymentManager action
chaiml-muster-v2-q235b-_77862_v2 status is now inactive due to auto deactivation removed underperforming models