chaiml-muster-v3a-kakit-_3784

developer_uid: zonemercy
submission_id: chaiml-muster-v3a-kakit-_3784_v3
model_name: chaiml-muster-v3a-kakit-_3784_v3
model_group: ChaiML/muster-v3a-kakit-
status: inactive
timestamp: 2026-02-15T10:17:20+00:00
num_battles: 10657
num_wins: 5613
celo_rating: 1318.71
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/muster-v3a-kakit-q235b-lr1e4ep1r64g4
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-muster-v3a-kakit-_3784_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/muster-v3a-kakit-q235b-lr1e4ep1r64g4
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-15
win_ratio: 0.5266960683119076
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|user|>', '</think>', '</s>', '<|assistant|>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-muster-v3a-kakit-3784-v3-uploader
Waiting for job on chaiml-muster-v3a-kakit-3784-v3-uploader to finish
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-muster-v3a-kakit-3784-v3-uploader: Using quantization_mode: w4a16
chaiml-muster-v3a-kakit-3784-v3-uploader: Checking if ChaiML/muster-v3a-kakit-q235b-lr1e4ep1r64g4-W4A16 already exists in ChaiML
chaiml-muster-v3a-kakit-3784-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-muster-v3a-kakit-3784-v3-uploader: Downloading snapshot of ChaiML/muster-v3a-kakit-q235b-lr1e4ep1r64g4-W4A16...
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v3a-kakit-3784-v3-uploader: Downloaded in 46.352s
chaiml-muster-v3a-kakit-3784-v3-uploader: Processed model ChaiML/muster-v3a-kakit-q235b-lr1e4ep1r64g4 in 46.986s
chaiml-muster-v3a-kakit-3784-v3-uploader: creating bucket guanaco-vllm-models
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v3a-kakit-3784-v3-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-muster-v3a-kakit-3784-v3-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v3a-kakit-3784-v3-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v3a-kakit-3784-v3-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v3a-kakit-3784-v3-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-muster-v3a-kakit-3784-v3-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-muster-v3a-kakit-3784-v3-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-muster-v3a-kakit-3784-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-muster-v3a-kakit-3784-v3-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-muster-v3a-kakit-3784-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-muster-v3a-kakit-3784-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/added_tokens.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/generation_config.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/chat_template.jinja
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/.gitattributes
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/config.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/special_tokens_map.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/tokenizer_config.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/quantization_config.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/vocab.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/merges.txt
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model.safetensors.index.json
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/tokenizer.json
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-muster-v3a-kakit-3784-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00018-of-00027.safetensors?partNumber=36&uploadId=2~k4TDKb5Tt1z5gjDl9EvsY3G_uKzjCoY": write tcp 10.2.139.36:57098->216.153.53.63:443: write: connection reset by peer
chaiml-muster-v3a-kakit-3784-v3-uploader: ERROR "cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00018-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~k4TDKb5Tt1z5gjDl9EvsY3G_uKzjCoY caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000a79ea6c9a3380bde-00699186a5-1569a6ec03-default, host id:
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00027-of-00027.safetensors
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-muster-v3a-kakit-3784-v3-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00001-of-00027.safetensors?partNumber=87&uploadId=2~4yuJgWg9xjosNd28VYICNb_m8ljNLLf": write tcp 10.2.139.36:60594->216.153.53.63:443: write: connection reset by peer
chaiml-muster-v3a-kakit-3784-v3-uploader: ERROR "cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00001-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~4yuJgWg9xjosNd28VYICNb_m8ljNLLf caused by: SignatureDoesNotMatch:  status code: 403, request id: tx0000026a4a3aeeb5a8885-00699187bb-15698d1afd-default, host id:
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00016-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00005-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00022-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00007-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00010-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00009-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00025-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00019-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00026-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00023-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00017-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00008-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00006-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00013-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00012-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00004-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00003-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00014-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00011-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00020-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00002-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00024-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00015-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/.gitattributes": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/added_tokens.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/chat_template.jinja": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/config.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/generation_config.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/merges.txt": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00002-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00003-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00004-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00005-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00006-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00007-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00008-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00009-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00010-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00011-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00012-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00013-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00014-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00015-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00016-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00017-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00019-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00020-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00021-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00022-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00023-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00024-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00025-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00026-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00027-of-00027.safetensors": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model.safetensors.index.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/quantization_config.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/special_tokens_map.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/tokenizer.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/tokenizer_config.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/vocab.json": object size matches
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00001-of-00027.safetensors
chaiml-muster-v3a-kakit-3784-v3-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-muster-v3a-kakit-3784-v3/default/model-00018-of-00027.safetensors
Job chaiml-muster-v3a-kakit-3784-v3-uploader completed after 705.41s with status: succeeded
Stopping job with name chaiml-muster-v3a-kakit-3784-v3-uploader
Pipeline stage VLLMUploader completed in 705.82s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-muster-v3a-kakit-3784-v3
Waiting for inference service chaiml-muster-v3a-kakit-3784-v3 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-muster-v3a-kakit-3784-v3 ready after 771.1893212795258s
Pipeline stage VLLMDeployer completed in 771.57s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.556675910949707s
Received healthy response to inference request in 2.0430259704589844s
Received healthy response to inference request in 1.8494553565979004s
Received healthy response to inference request in 1.98032546043396s
Received healthy response to inference request in 2.1225333213806152s
Received healthy response to inference request in 1.8652029037475586s
Received healthy response to inference request in 2.2301337718963623s
Received healthy response to inference request in 2.0235698223114014s
Received healthy response to inference request in 2.2509233951568604s
Received healthy response to inference request in 2.2006540298461914s
Received healthy response to inference request in 1.9237236976623535s
Received healthy response to inference request in 2.9639804363250732s
Received healthy response to inference request in 2.3228938579559326s
Received healthy response to inference request in 2.145066022872925s
Received healthy response to inference request in 2.665104627609253s
Received healthy response to inference request in 2.4027411937713623s
Received healthy response to inference request in 2.4511096477508545s
Received healthy response to inference request in 2.285212755203247s
Received healthy response to inference request in 2.3018743991851807s
Received healthy response to inference request in 2.17692232131958s
Received healthy response to inference request in 1.7991406917572021s
Received healthy response to inference request in 2.3685903549194336s
Received healthy response to inference request in 2.104097604751587s
Received healthy response to inference request in 2.005618095397949s
Received healthy response to inference request in 2.057563543319702s
Received healthy response to inference request in 1.9236724376678467s
Received healthy response to inference request in 2.1880135536193848s
Received healthy response to inference request in 2.169123649597168s
Received healthy response to inference request in 2.095334053039551s
Received healthy response to inference request in 1.9442734718322754s
30 requests
0 failed requests
5th percentile: 1.8565417528152466
10th percentile: 1.9178254842758178
20th percentile: 1.9731150627136231
30th percentile: 2.0371891260147095
40th percentile: 2.1005921840667723
50th percentile: 2.1570948362350464
60th percentile: 2.1930697441101072
70th percentile: 2.2612102031707764
80th percentile: 2.332033157348633
90th percentile: 2.46166627407074
95th percentile: 2.616311705112457
99th percentile: 2.8773064517974856
mean time: 2.1805518786112468
Pipeline stage StressChecker completed in 72.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-muster-v3a-kakit-_3784_v3 status is now deployed due to DeploymentManager action
chaiml-muster-v3a-kakit-_3784_v3 status is now inactive due to auto deactivation removed underperforming models