developer_uid: rirv938
submission_id: chaiml-sft-qwen-235b-ro_85751_v1
model_name: chaiml-sft-qwen-235b-ro_85751_v1
model_group: ChaiML/sft_qwen_235b_rob
status: torndown
timestamp: 2026-02-23T12:06:50+00:00
num_battles: 12516
num_wins: 6898
celo_rating: 1344.15
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 96
reward_model: default
display_name: chaiml-sft-qwen-235b-ro_85751_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-05
win_ratio: 0.5511345477788431
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'You:', '<|im_end|>', '<|im_start|>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 96}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-85751-v1-uploader to finish
chaiml-sft-qwen-235b-ro-60246-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-60246-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-85751-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Downloaded to shared memory in 136.714s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Downloaded to shared memory in 136.876s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed in 203.108s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-85751-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/added_tokens.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/.gitattributes
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/chat_template.jinja
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/generation_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/special_tokens_map.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/quantization_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/merges.txt
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/vocab.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model.safetensors.index.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-sft-qwen-235b-ro-85751-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors?partNumber=4&uploadId=2~IPH1kA31PD94HtAEHIVGf3v3_IP6Ftr": write tcp 10.0.5.146:34102->216.153.53.63:443: write: broken pipe
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ERROR "cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~IPH1kA31PD94HtAEHIVGf3v3_IP6Ftr caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000e76ac69b8882baf5-006984d033-14a26732ad-default, host id:
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00027-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_1_version-int4-mixed in 319.655s
chaiml-sft-qwen-235b-ro-60246-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-60246-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/.gitattributes
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/added_tokens.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/tokenizer_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/generation_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/chat_template.jinja
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/special_tokens_map.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/quantization_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/merges.txt
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model.safetensors.index.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/tokenizer.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/vocab.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00012-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00009-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00018-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00003-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00010-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00026-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00006-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00021-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00007-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00001-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00005-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00002-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00024-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00011-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00027-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00016-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00023-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00004-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00015-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00020-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00019-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00008-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00017-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00014-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-85751-v1-uploader completed after 466.09s with status: failed
Stopping job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-85751-v1-uploader to finish
chaiml-sft-qwen-235b-ro-85751-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-85751-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00020-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00014-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00004-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00001-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00016-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00026-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00011-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00009-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00008-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00015-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00005-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00019-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00023-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00006-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00024-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00003-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00010-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00018-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00017-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00022-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-60246-v1-uploader completed after 592.96s with status: succeeded
Stopping job with name chaiml-sft-qwen-235b-ro-60246-v1-uploader
Pipeline stage VLLMUploader completed in 594.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.42s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-sft-qwen-235b-ro-60246-v1
Waiting for inference service chaiml-sft-qwen-235b-ro-60246-v1 to be ready
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Downloaded to shared memory in 154.023s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed in 217.621s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-85751-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/.gitattributes": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/added_tokens.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/chat_template.jinja": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/generation_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/merges.txt": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00001-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00002-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00003-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00004-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00005-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00006-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00007-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00008-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00009-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00010-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00011-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00012-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00013-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00014-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00015-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00016-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00017-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00018-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00019-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00020-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00021-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00022-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00023-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00024-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00026-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00027-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model.safetensors.index.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/quantization_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/special_tokens_map.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/vocab.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-85751-v1-uploader completed after 272.29s with status: succeeded
Stopping job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Pipeline stage VLLMUploader completed in 741.77s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.41s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-sft-qwen-235b-ro-85751-v1
Waiting for inference service chaiml-sft-qwen-235b-ro-85751-v1 to be ready
Inference service chaiml-sft-qwen-235b-ro-60246-v1 ready after 456.3508563041687s
Pipeline stage VLLMDeployer completed in 457.89s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.6831576824188232s
Received healthy response to inference request in 3.2752230167388916s
Received healthy response to inference request in 3.0649478435516357s
Received healthy response to inference request in 2.948944330215454s
Received healthy response to inference request in 2.940086603164673s
Received healthy response to inference request in 2.9528658390045166s
Received healthy response to inference request in 3.2884461879730225s
Received healthy response to inference request in 2.8155863285064697s
Received healthy response to inference request in 2.8440632820129395s
Received healthy response to inference request in 2.712222099304199s
Received healthy response to inference request in 2.797752618789673s
Received healthy response to inference request in 2.8235769271850586s
Received healthy response to inference request in 3.006124973297119s
Received healthy response to inference request in 2.7054388523101807s
Received healthy response to inference request in 2.8429062366485596s
Received healthy response to inference request in 2.9872934818267822s
Received healthy response to inference request in 2.910615921020508s
Received healthy response to inference request in 2.824007987976074s
Received healthy response to inference request in 2.733128547668457s
Received healthy response to inference request in 2.9522318840026855s
Received healthy response to inference request in 2.9221689701080322s
Received healthy response to inference request in 3.055537223815918s
Received healthy response to inference request in 2.839881658554077s
Received healthy response to inference request in 3.421782970428467s
Received healthy response to inference request in 3.0641307830810547s
Received healthy response to inference request in 3.1991145610809326s
Received healthy response to inference request in 2.8516061305999756s
Received healthy response to inference request in 2.769779920578003s
Received healthy response to inference request in 2.980849504470825s
Received healthy response to inference request in 3.051243543624878s
30 requests
0 failed requests
5th percentile: 2.721630001068115
10th percentile: 2.7661147832870485
20th percentile: 2.821978807449341
30th percentile: 2.8419988632202147
40th percentile: 2.887012004852295
50th percentile: 2.9445154666900635
60th percentile: 2.96405930519104
70th percentile: 3.0196605443954465
80th percentile: 3.064294195175171
90th percentile: 3.2765453338623045
95th percentile: 3.3617814183235164
99th percentile: 3.6073590159416202
mean time: 2.9754905303319297
Pipeline stage StressChecker completed in 111.56s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.99s
Shutdown handler de-registered
chaiml-sft-qwen-235b-ro_60246_v1 status is now deployed due to DeploymentManager action
Inference service chaiml-sft-qwen-235b-ro-85751-v1 ready after 477.170859336853s
Pipeline stage VLLMDeployer completed in 478.48s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6163299083709717s
Received healthy response to inference request in 3.5707409381866455s
Received healthy response to inference request in 2.881340980529785s
Received healthy response to inference request in 2.9864344596862793s
Received healthy response to inference request in 3.177905559539795s
Received healthy response to inference request in 3.2262215614318848s
Received healthy response to inference request in 2.9961657524108887s
Received healthy response to inference request in 2.937823534011841s
Received healthy response to inference request in 2.7989325523376465s
Received healthy response to inference request in 2.79569411277771s
Received healthy response to inference request in 2.860351800918579s
Received healthy response to inference request in 2.947758436203003s
Received healthy response to inference request in 2.7759082317352295s
Received healthy response to inference request in 2.699169158935547s
Received healthy response to inference request in 2.740800619125366s
Received healthy response to inference request in 2.8345115184783936s
Received healthy response to inference request in 2.936521053314209s
Received healthy response to inference request in 2.822934865951538s
Received healthy response to inference request in 2.822862386703491s
Received healthy response to inference request in 2.794018507003784s
Received healthy response to inference request in 2.851808547973633s
Received healthy response to inference request in 3.0107414722442627s
Received healthy response to inference request in 3.1506989002227783s
Received healthy response to inference request in 2.9304070472717285s
Received healthy response to inference request in 2.9765665531158447s
Received healthy response to inference request in 2.6953346729278564s
Received healthy response to inference request in 2.8741283416748047s
Received healthy response to inference request in 2.525390148162842s
Received healthy response to inference request in 2.7734034061431885s
Received healthy response to inference request in 2.712207794189453s
30 requests
0 failed requests
5th percentile: 2.65188205242157
10th percentile: 2.6987857103347777
20th percentile: 2.766882848739624
30th percentile: 2.795191431045532
40th percentile: 2.8229058742523194
50th percentile: 2.856080174446106
60th percentile: 2.9009674072265623
70th percentile: 2.9408040046691895
80th percentile: 2.988380718231201
90th percentile: 3.15341956615448
95th percentile: 3.2044793605804442
99th percentile: 3.4708303189277654
mean time: 2.890770427385966
Pipeline stage StressChecker completed in 96.80s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.40s
Shutdown handler de-registered
chaiml-sft-qwen-235b-ro_85751_v1 status is now deployed due to DeploymentManager action
chaiml-sft-qwen-235b-ro_85751_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-sft-qwen-235b-ro_85751_v1 status is now inactive due to Froze recruitment for AB test 0206_kullback
chaiml-sft-qwen-235b-ro_85751_v1 status is now torndown due to DeploymentManager action