developer_uid: zonemercy
submission_id: chaiml-opusd-v4-g46-lr1_69112_v6
model_name: chaiml-opusd-v4-g46-lr1_69112_v6
model_group: ChaiML/opusd-v4-g46-lr1e
status: torndown
timestamp: 2026-03-08T12:33:20+00:00
num_battles: 11557
num_wins: 6286
celo_rating: 1322.17
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/opusd-v4-g46-lr1e4ep1r64b16
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-opusd-v4-g46-lr1_69112_v6
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/opusd-v4-g46-lr1e4ep1r64b16
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-05
win_ratio: 0.5439127801332526
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|user|>', '<|im_end|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n<think></think>\n{message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-opusd-v4-g46-lr1-69112-v6-uploader
Connection pool is full, discarding connection: %s. Connection pool size: %s
Waiting for job on chaiml-opusd-v4-g46-lr1-69112-v6-uploader to finish
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Using quantization_mode: fp8
Failed to get response for submission chaiml-csfs-v3-3-dpo-lr5e_937_v5: ('http://guanaco-model-mesh-load-balancer.model-mesh.kchai-google-us-east4.chaiverse.com/models/chaiml-csfs-v3-3-dpo-lr5e_937_v5/predict', '{"detail":"1 validation error for RuntimeResponse\\npredictions\\n  Field required [type=missing, input_value={\'detail\': \\"Cannot connec...(\'10.8.31.105\', 8080)]\\"}, input_type=dict]\\n    For further information visit https://errors.pydantic.dev/2.12/v/missing"}')
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Checking if ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8 already exists in ChaiML
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Downloading snapshot of ChaiML/opusd-v4-g46-lr1e4ep1r64b16-FP8...
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Downloaded in 150.143s
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Processed model ChaiML/opusd-v4-g46-lr1e4ep1r64b16 in 153.595s
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: creating bucket guanaco-vllm-models
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-opusd-v4-g46-lr1-69112-v6-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/.gitattributes
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/chat_template.jinja
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/generation_config.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/config.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/special_tokens_map.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/tokenizer_config.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/recipe.yaml
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model.safetensors.index.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/tokenizer.json
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00072-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00071-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00071-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00035-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00041-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00049-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00025-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00061-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00024-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00022-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00027-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00050-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00004-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00068-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00001-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00042-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00026-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00054-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00056-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00060-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00045-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00017-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00048-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00039-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00014-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00003-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00065-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00008-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00020-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00051-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00013-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00044-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00043-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00021-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00007-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00016-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00005-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00015-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00036-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00047-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00040-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00010-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00037-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00057-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00052-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00067-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00066-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00032-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00031-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00028-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00030-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00023-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00009-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00019-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00002-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00012-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00062-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00038-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00034-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00058-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00053-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00033-of-00072.safetensors
chaiml-opusd-v4-g46-lr1-69112-v6-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-opusd-v4-g46-lr1-69112-v6/default/model-00011-of-00072.safetensors
Job chaiml-opusd-v4-g46-lr1-69112-v6-uploader completed after 298.01s with status: succeeded
Stopping job with name chaiml-opusd-v4-g46-lr1-69112-v6-uploader
Pipeline stage VLLMUploader completed in 298.60s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.93s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-opusd-v4-g46-lr1-69112-v6
Waiting for inference service chaiml-opusd-v4-g46-lr1-69112-v6 to be ready
Inference service chaiml-opusd-v4-g46-lr1-69112-v6 ready after 582.3654465675354s
Pipeline stage VLLMDeployer completed in 582.80s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8790028095245361s
Received healthy response to inference request in 2.057586669921875s
Received healthy response to inference request in 2.526106595993042s
Received healthy response to inference request in 2.1458775997161865s
Received healthy response to inference request in 2.4882493019104004s
Received healthy response to inference request in 2.055389642715454s
Received healthy response to inference request in 1.895559310913086s
Received healthy response to inference request in 2.1031606197357178s
Received healthy response to inference request in 2.1367523670196533s
Received healthy response to inference request in 1.9396929740905762s
Received healthy response to inference request in 2.5677475929260254s
Received healthy response to inference request in 2.273350238800049s
Received healthy response to inference request in 1.945199728012085s
Received healthy response to inference request in 1.9265587329864502s
Received healthy response to inference request in 1.9743890762329102s
Received healthy response to inference request in 2.0448083877563477s
Received healthy response to inference request in 2.4331178665161133s
Received healthy response to inference request in 2.061448335647583s
Received healthy response to inference request in 2.5599606037139893s
Received healthy response to inference request in 1.8536889553070068s
Received healthy response to inference request in 2.106550455093384s
Received healthy response to inference request in 2.0743861198425293s
Received healthy response to inference request in 2.7080657482147217s
Received healthy response to inference request in 1.7181203365325928s
Received healthy response to inference request in 1.9801137447357178s
Received healthy response to inference request in 1.7805678844451904s
Received healthy response to inference request in 1.8806746006011963s
Received healthy response to inference request in 2.095956563949585s
Received healthy response to inference request in 1.838280200958252s
Received healthy response to inference request in 2.0430197715759277s
30 requests
0 failed requests
5th percentile: 1.806538426876068
10th percentile: 1.8521480798721313
20th percentile: 1.892582368850708
30th percentile: 1.9435477018356324
40th percentile: 2.017857360839844
50th percentile: 2.0564881563186646
60th percentile: 2.0830142974853514
70th percentile: 2.1156110286712644
80th percentile: 2.3053037643432623
90th percentile: 2.529491996765137
95th percentile: 2.5642434477806093
99th percentile: 2.667373483181
mean time: 2.1031127611796063
Pipeline stage StressChecker completed in 77.90s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
Shutdown handler de-registered
chaiml-opusd-v4-g46-lr1_69112_v6 status is now deployed due to DeploymentManager action
chaiml-opusd-v4-g46-lr1_69112_v6 status is now inactive due to auto deactivation removed underperforming models
Deleting key chaiml-opusd-v4-g46-mas-11495-v5/default/recipe.yaml from bucket guanaco-vllm-models
chaiml-opusd-v4-g46-lr1_69112_v6 status is now torndown due to DeploymentManager action