developer_uid: zonemercy
submission_id: chaiml-opusd-v4-g46-lr1_17132_v2
model_name: chaiml-opusd-v4-g46-lr1_17132_v2
model_group: ChaiML/opusd-v4-g46-lr1e
status: torndown
timestamp: 2026-03-07T17:32:22+00:00
num_battles: 11533
num_wins: 6323
celo_rating: 1341.69
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-opusd-v4-g46-lr1_17132_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-04
win_ratio: 0.5482528396774473
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}.<|user|>\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-opusd-v4-g46-lr1-17132-v2-uploader
Waiting for job on chaiml-opusd-v4-g46-lr1-17132-v2-uploader to finish
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: Using quantization_mode: none
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: Downloading snapshot of ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8...
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: Downloaded in 195.282s
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: Processed model ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8 in 428.000s
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: creating bucket guanaco-vllm-models
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-g46-lr1-17132-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/generation_config.json
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/.gitattributes
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/config.json
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/chat_template.jinja
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/recipe.yaml
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/tokenizer.json
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00072-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00071-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00064-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00027-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00040-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00066-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00058-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00003-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00001-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00022-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00063-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00029-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00048-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00067-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00037-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00059-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00054-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00032-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00028-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00012-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00038-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00049-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00025-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00009-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00043-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00006-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00062-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00030-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00013-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00008-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00036-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00044-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00016-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00017-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00069-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00047-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00019-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00060-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00002-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00056-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00051-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00021-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00039-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00061-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00045-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00024-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00050-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00057-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00031-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00014-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00004-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00033-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00065-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00068-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00052-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00042-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00046-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00011-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00007-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00035-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00055-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00015-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00053-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00005-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00020-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00026-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00041-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00023-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00034-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00070-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00018-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-17132-v2-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-17132-v2/default/model-00010-of-00072.safetensors
Job chaiml-opusd-v4-g46-lr1-17132-v2-uploader completed after 647.63s with status: succeeded
Stopping job with name chaiml-opusd-v4-g46-lr1-17132-v2-uploader
Pipeline stage VLLMUploader completed in 648.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.27s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-opusd-v4-g46-lr1-17132-v2
Waiting for inference service chaiml-opusd-v4-g46-lr1-17132-v2 to be ready
Inference service chaiml-opusd-v4-g46-lr1-17132-v2 ready after 1642.1853649616241s
Pipeline stage VLLMDeployer completed in 1642.75s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.2046916484832764s
Received healthy response to inference request in 1.8325443267822266s
Received healthy response to inference request in 4.1044793128967285s
Received healthy response to inference request in 2.0610694885253906s
Received healthy response to inference request in 1.937274694442749s
Received healthy response to inference request in 1.9825823307037354s
Received healthy response to inference request in 1.9337289333343506s
Received healthy response to inference request in 2.2229650020599365s
Received healthy response to inference request in 1.8670058250427246s
Received healthy response to inference request in 2.2265844345092773s
Received healthy response to inference request in 2.1304919719696045s
Received healthy response to inference request in 1.9710485935211182s
Received healthy response to inference request in 2.1483192443847656s
Received healthy response to inference request in 1.960522174835205s
Received healthy response to inference request in 2.0459461212158203s
Received healthy response to inference request in 1.8149733543395996s
Received healthy response to inference request in 2.149151563644409s
Received healthy response to inference request in 2.010442018508911s
Received healthy response to inference request in 2.0619709491729736s
Received healthy response to inference request in 2.02223801612854s
Received healthy response to inference request in 2.2159245014190674s
Received healthy response to inference request in 2.032949924468994s
Received healthy response to inference request in 2.064056634902954s
Received healthy response to inference request in 1.9878222942352295s
Received healthy response to inference request in 2.312100410461426s
Received healthy response to inference request in 1.8688924312591553s
Received healthy response to inference request in 2.1089940071105957s
Received healthy response to inference request in 1.980656623840332s
Received healthy response to inference request in 2.089155912399292s
30 requests
1 failed requests
5th percentile: 1.8480520009994508
10th percentile: 1.8687037706375123
20th percentile: 1.9558726787567138
30th percentile: 1.9820046186447144
40th percentile: 2.0175196170806884
50th percentile: 2.0535078048706055
60th percentile: 2.074096345901489
70th percentile: 2.135840153694153
80th percentile: 2.2069382190704347
90th percentile: 2.235136032104492
95th percentile: 3.2979088068008373
99th percentile: 15.483002977371228
mean time: 2.7159717241923014
%s, retrying in %s seconds...
Received healthy response to inference request in 1.863130807876587s
Received healthy response to inference request in 2.1838786602020264s
Received healthy response to inference request in 1.9040546417236328s
Received healthy response to inference request in 2.150085210800171s
Received healthy response to inference request in 1.956038475036621s
Received healthy response to inference request in 2.149627685546875s
Received healthy response to inference request in 1.9959051609039307s
Received healthy response to inference request in 2.0133426189422607s
Received healthy response to inference request in 1.9074184894561768s
Received healthy response to inference request in 2.0290281772613525s
Received healthy response to inference request in 2.0633935928344727s
Received healthy response to inference request in 2.0613503456115723s
Received healthy response to inference request in 2.057427406311035s
Received healthy response to inference request in 2.013667106628418s
Received healthy response to inference request in 1.9560296535491943s
Received healthy response to inference request in 1.8056931495666504s
Received healthy response to inference request in 1.9876432418823242s
Received healthy response to inference request in 2.1076934337615967s
Received healthy response to inference request in 2.2316553592681885s
Received healthy response to inference request in 2.0293774604797363s
Received healthy response to inference request in 2.0295920372009277s
Received healthy response to inference request in 2.039144277572632s
Received healthy response to inference request in 1.923781156539917s
Received healthy response to inference request in 2.4034013748168945s
Received healthy response to inference request in 2.024061441421509s
Received healthy response to inference request in 1.8977012634277344s
Received healthy response to inference request in 1.9170787334442139s
Received healthy response to inference request in 1.9589958190917969s
Received healthy response to inference request in 1.9047214984893799s
Received healthy response to inference request in 2.2474026679992676s
30 requests
0 failed requests
5th percentile: 1.8786875128746032
10th percentile: 1.903419303894043
20th percentile: 1.9151466846466065
30th percentile: 1.9560358285903932
40th percentile: 1.992600393295288
50th percentile: 2.0188642740249634
60th percentile: 2.029463291168213
70th percentile: 2.058604288101196
80th percentile: 2.1160802841186523
90th percentile: 2.1886563301086426
95th percentile: 2.240316379070282
99th percentile: 2.3581617498397827
mean time: 2.0270773649215696
Pipeline stage StressChecker completed in 152.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.91s
Shutdown handler de-registered
chaiml-opusd-v4-g46-lr1_17132_v2 status is now deployed due to DeploymentManager action
chaiml-opusd-v4-g46-lr1_17132_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-opusd-v4-g46-lr1_17132_v2 status is now torndown due to DeploymentManager action