developer_uid: acehao-chai
submission_id: chaiml-kimid-v9-opusdv_23365_v24
model_name: chaiml-kimid-v9-opusdv_23365_v24
model_group: ChaiML/kimid-v9-opusdv1-
status: torndown
timestamp: 2026-03-09T18:32:17+00:00
num_battles: 14617
num_wins: 7907
celo_rating: 1324.65
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-kimid-v9-opusdv_23365_v24
ineligible_reason: max_output_tokens!=64
is_internal_developer: False
language_model: ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01
model_size: 19B
ranking_group: single
us_pacific_date: 2026-03-06
win_ratio: 0.540945474447561
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '<|user|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v24-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v24-uploader to finish
chaiml-kimid-v9-opusdv-23365-v24-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v24-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v24-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v24-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-kimid-v9-opusdv-23365-v24-uploader: Downloaded in 172.981s
chaiml-kimid-v9-opusdv-23365-v24-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 173.684s
chaiml-kimid-v9-opusdv-23365-v24-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v24-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v24-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v24-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v24-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/config.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model.safetensors.index.json
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v9-opusdv-23365-v24-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00016-of-00027.safetensors?partNumber=12&uploadId=2~7xtJd03Flw4j7CB8mXDBU3KCwlatE3i": read tcp 10.0.23.216:38922->216.153.53.63:443: read: connection reset by peer
chaiml-kimid-v9-opusdv-23365-v24-uploader: ERROR "cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00016-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~7xtJd03Flw4j7CB8mXDBU3KCwlatE3i caused by: SignatureDoesNotMatch:  status code: 403, request id: tx0000044e3eeea8d265c8b-006995fc2d-1569a6ec03-default, host id:
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00027-of-00027.safetensors
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v24-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/.gitattributes": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/added_tokens.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/chat_template.jinja": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/config.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/generation_config.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/merges.txt": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model.safetensors.index.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/quantization_config.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/special_tokens_map.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/tokenizer.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/tokenizer_config.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/vocab.json": object size matches
chaiml-kimid-v9-opusdv-23365-v24-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v24/default/model-00016-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-23365-v24-uploader completed after 446.95s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v24-uploader
Pipeline stage VLLMUploader completed in 447.68s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.21s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v24
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v24 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-grpo-q235b-kimid_37540_v1: HTTPConnectionPool(host='chaiml-grpo-q235b-kimid-37540-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-kimid-v9-opusdv-23365-v24 ready after 633.6393709182739s
Pipeline stage VLLMDeployer completed in 634.43s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0228095054626465s
Received healthy response to inference request in 1.8956551551818848s
Received healthy response to inference request in 2.026089906692505s
Received healthy response to inference request in 1.937640905380249s
Received healthy response to inference request in 3.223639726638794s
Received healthy response to inference request in 2.056368827819824s
Received healthy response to inference request in 2.268141508102417s
Received healthy response to inference request in 2.154297351837158s
Received healthy response to inference request in 2.1929843425750732s
Received healthy response to inference request in 1.9165515899658203s
Received healthy response to inference request in 2.140000343322754s
Received healthy response to inference request in 1.949073076248169s
Received healthy response to inference request in 2.096768379211426s
Received healthy response to inference request in 1.9149436950683594s
Received healthy response to inference request in 2.2331736087799072s
Received healthy response to inference request in 2.1133720874786377s
Received healthy response to inference request in 2.0386416912078857s
Received healthy response to inference request in 1.879903793334961s
Received healthy response to inference request in 2.0363662242889404s
Received healthy response to inference request in 1.9699456691741943s
Received healthy response to inference request in 2.010455369949341s
Received healthy response to inference request in 1.9311389923095703s
Received healthy response to inference request in 1.976738452911377s
Received healthy response to inference request in 2.084238052368164s
Received healthy response to inference request in 2.046945333480835s
Received healthy response to inference request in 2.1088483333587646s
Received healthy response to inference request in 2.108058214187622s
Received healthy response to inference request in 2.31950044631958s
Received healthy response to inference request in 2.0695197582244873s
Received healthy response to inference request in 2.137061357498169s
30 requests
0 failed requests
5th percentile: 1.9043349981307984
10th percentile: 1.9163908004760741
20th percentile: 1.946786642074585
30th percentile: 2.000340294837952
40th percentile: 2.032255697250366
50th percentile: 2.0516570806503296
60th percentile: 2.089250183105469
70th percentile: 2.1102054595947264
80th percentile: 2.1428597450256346
90th percentile: 2.236670398712158
95th percentile: 2.2963889241218567
99th percentile: 2.9614393353462227
mean time: 2.095295723279317
Pipeline stage StressChecker completed in 68.14s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v24 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v24 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_23365_v24 status is now torndown due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v24 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v24 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_23365_v24 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v24 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_23365_v24 status is now torndown due to DeploymentManager action