chaiml-muster-v0-q235b_52842

developer_uid: zonemercy
submission_id: chaiml-muster-v0-q235b_52842_v13
model_name: chaiml-muster-v0-q235b_52842_v13
model_group: ChaiML/muster-v0-q235b-l
status: torndown
timestamp: 2026-02-12T20:01:09+00:00
num_battles: 10630
num_wins: 5688
celo_rating: 1332.21
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/muster-v0-q235b-lr1e4ep2r64g4
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-muster-v0-q235b_52842_v13
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/muster-v0-q235b-lr1e4ep2r64g4
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-09
win_ratio: 0.5350893697083725
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####', '</think>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-muster-v0-q235b-52842-v13-uploader
Waiting for job on chaiml-muster-v0-q235b-52842-v13-uploader to finish
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v0-q235b-52842-v13-uploader: Using quantization_mode: w4a16
chaiml-muster-v0-q235b-52842-v13-uploader: Checking if ChaiML/muster-v0-q235b-lr1e4ep2r64g4-W4A16 already exists in ChaiML
chaiml-muster-v0-q235b-52842-v13-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-muster-v0-q235b-52842-v13-uploader: Downloading snapshot of ChaiML/muster-v0-q235b-lr1e4ep2r64g4-W4A16...
chaiml-muster-v0-q235b-52842-v13-uploader: 
Fetching 39 files:   0%|          | 0/39 [00:00<?, ?it/s]
Fetching 39 files:   3%|▎         | 1/39 [00:00<00:10,  3.49it/s]
Fetching 39 files:  18%|█▊        | 7/39 [00:13<01:05,  2.06s/it]
Fetching 39 files:  21%|██        | 8/39 [00:18<01:19,  2.56s/it]
Fetching 39 files:  38%|███▊      | 15/39 [00:28<00:42,  1.78s/it]
Fetching 39 files:  41%|████      | 16/39 [00:31<00:46,  2.00s/it]
Fetching 39 files:  44%|████▎     | 17/39 [00:32<00:39,  1.81s/it]
Fetching 39 files:  46%|████▌     | 18/39 [00:38<00:52,  2.50s/it]
Fetching 39 files:  51%|█████▏    | 20/39 [00:40<00:38,  2.01s/it]
Fetching 39 files:  59%|█████▉    | 23/39 [00:46<00:32,  2.00s/it]
Fetching 39 files:  62%|██████▏   | 24/39 [00:48<00:30,  2.04s/it]
Fetching 39 files:  64%|██████▍   | 25/39 [00:50<00:28,  2.02s/it]
Fetching 39 files:  67%|██████▋   | 26/39 [00:58<00:43,  3.35s/it]
Fetching 39 files:  79%|███████▉  | 31/39 [01:04<00:16,  2.01s/it]
Fetching 39 files: 100%|██████████| 39/39 [01:04<00:00,  1.65s/it]
chaiml-muster-v0-q235b-52842-v13-uploader: Downloaded in 64.488s
chaiml-muster-v0-q235b-52842-v13-uploader: Processed model ChaiML/muster-v0-q235b-lr1e4ep2r64g4 in 65.016s
chaiml-muster-v0-q235b-52842-v13-uploader: creating bucket guanaco-vllm-models
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v13-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-muster-v0-q235b-52842-v13-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v13-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v13-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v13-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v0-q235b-52842-v13-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-muster-v0-q235b-52842-v13-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-muster-v0-q235b-52842-v13-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-muster-v0-q235b-52842-v13-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-muster-v0-q235b-52842-v13-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-muster-v0-q235b-52842-v13-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/.gitattributes
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/chat_template.jinja
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/special_tokens_map.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/added_tokens.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/tokenizer_config.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/generation_config.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/config.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/quantization_config.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/merges.txt
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/vocab.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/tokenizer.json
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model.safetensors.index.json
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00006-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00012-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00009-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00019-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00015-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00014-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00017-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00020-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00011-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00008-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00016-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00001-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00013-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00025-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00024-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00003-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00005-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00002-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00007-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00026-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00010-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00018-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00022-of-00027.safetensors
chaiml-muster-v0-q235b-52842-v13-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v0-q235b-52842-v13/model-00004-of-00027.safetensors
Job chaiml-muster-v0-q235b-52842-v13-uploader completed after 647.69s with status: succeeded
Stopping job with name chaiml-muster-v0-q235b-52842-v13-uploader
Pipeline stage VLLMUploader completed in 648.09s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-muster-v0-q235b-52842-v13
Waiting for inference service chaiml-muster-v0-q235b-52842-v13 to be ready
Failed to get response for submission chaiml-llama31-mer-v2-_44570_v66: ('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-02f4-69d4-linear-w01_v8/predict', '{"detail":"1 validation error for RuntimeResponse\\npredictions\\n  Field required [type=missing, input_value={\'detail\': \'1 validation ...tic.dev/2.11/v/missing\'}, input_type=dict]\\n    For further information visit https://errors.pydantic.dev/2.12/v/missing"}')
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-muster-v0-q235b-52842-v13 ready after 702.9868919849396s
Pipeline stage VLLMDeployer completed in 703.37s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.4234793186187744s
Received healthy response to inference request in 2.099055051803589s
Received healthy response to inference request in 2.3366031646728516s
Received healthy response to inference request in 2.02972412109375s
Received healthy response to inference request in 2.0090792179107666s
Received healthy response to inference request in 1.9097042083740234s
Received healthy response to inference request in 1.9979898929595947s
Received healthy response to inference request in 2.2116668224334717s
Received healthy response to inference request in 2.3856096267700195s
Received healthy response to inference request in 2.316627264022827s
Received healthy response to inference request in 2.582383155822754s
Received healthy response to inference request in 2.4647982120513916s
Received healthy response to inference request in 2.260566234588623s
Received healthy response to inference request in 2.280977725982666s
Received healthy response to inference request in 1.9983463287353516s
Received healthy response to inference request in 2.5893077850341797s
Received healthy response to inference request in 2.050670623779297s
Received healthy response to inference request in 1.9795353412628174s
Received healthy response to inference request in 2.105691909790039s
Received healthy response to inference request in 2.443502902984619s
Received healthy response to inference request in 2.094851016998291s
Received healthy response to inference request in 2.522597074508667s
Received healthy response to inference request in 2.0416102409362793s
Received healthy response to inference request in 2.1182148456573486s
Received healthy response to inference request in 2.5109736919403076s
Received healthy response to inference request in 2.0945851802825928s
Received healthy response to inference request in 2.3743231296539307s
Received healthy response to inference request in 2.20220947265625s
Received healthy response to inference request in 2.0147504806518555s
30 requests
1 failed requests
5th percentile: 1.9878398895263671
10th percentile: 1.9983106851577759
20th percentile: 2.026729393005371
30th percentile: 2.081410813331604
40th percentile: 2.103037166595459
50th percentile: 2.206938147544861
60th percentile: 2.2952375411987305
70th percentile: 2.377709078788757
80th percentile: 2.447761964797974
90th percentile: 2.5285756826400756
95th percentile: 2.586191701889038
99th percentile: 15.05405305385591
mean time: 2.8198240359624225
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9310636520385742s
Received healthy response to inference request in 2.5428075790405273s
Received healthy response to inference request in 2.307943105697632s
Received healthy response to inference request in 2.0547828674316406s
Received healthy response to inference request in 2.311689615249634s
Received healthy response to inference request in 2.2329320907592773s
Received healthy response to inference request in 2.4675076007843018s
Received healthy response to inference request in 2.119061231613159s
Received healthy response to inference request in 1.892719030380249s
Received healthy response to inference request in 2.7005996704101562s
Received healthy response to inference request in 2.008322238922119s
Received healthy response to inference request in 2.043686628341675s
Received healthy response to inference request in 2.504676580429077s
Received healthy response to inference request in 2.0586142539978027s
Received healthy response to inference request in 2.0165364742279053s
Received healthy response to inference request in 2.2016170024871826s
Received healthy response to inference request in 2.4658567905426025s
Received healthy response to inference request in 1.9686307907104492s
Received healthy response to inference request in 2.2751407623291016s
Received healthy response to inference request in 2.6609981060028076s
Received healthy response to inference request in 1.9883739948272705s
Received healthy response to inference request in 1.9827816486358643s
Received healthy response to inference request in 2.1313939094543457s
Received healthy response to inference request in 2.2009494304656982s
Received healthy response to inference request in 2.0725135803222656s
Received healthy response to inference request in 2.562920570373535s
Received healthy response to inference request in 2.096195697784424s
Received healthy response to inference request in 2.060673236846924s
Received healthy response to inference request in 2.1242871284484863s
Received healthy response to inference request in 1.9251043796539307s
30 requests
0 failed requests
5th percentile: 1.9277860522270203
10th percentile: 1.9648740768432618
20th percentile: 2.0043325901031492
30th percentile: 2.0514539957046507
40th percentile: 2.0677774429321287
50th percentile: 2.1216741800308228
60th percentile: 2.201216459274292
70th percentile: 2.2849814653396607
80th percentile: 2.4661869525909426
90th percentile: 2.544818878173828
95th percentile: 2.616863214969635
99th percentile: 2.6891152167320254
mean time: 2.1970126549402873
Pipeline stage StressChecker completed in 156.62s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.63s
Shutdown handler de-registered
chaiml-muster-v0-q235b_52842_v13 status is now deployed due to DeploymentManager action
chaiml-muster-v0-q235b_52842_v13 status is now inactive due to auto deactivation removed underperforming models
chaiml-muster-v0-q235b_52842_v13 status is now torndown due to DeploymentManager action