developer_uid: rirv938
submission_id: chaiml-sft-qwen-235b-ro_60246_v1
model_name: chaiml-sft-qwen-235b-ro_60246_v1
model_group: ChaiML/sft_qwen_235b_rob
status: inactive
timestamp: 2026-02-05T18:57:49+00:00
num_battles: 12938
num_wins: 7122
celo_rating: 1340.0
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/sft_qwen_235b_rob_new_1_version-int4-mixed
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 96
reward_model: default
display_name: chaiml-sft-qwen-235b-ro_60246_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/sft_qwen_235b_rob_new_1_version-int4-mixed
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-05
win_ratio: 0.5504714793631164
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 60, 'presence_penalty': 0.1, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_start|>', '</s>', 'You:', '<|im_end|>', '###'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 96}
formatter: {'memory_template': '<|im_start|>system\nRespond as a high quality storyteller.<|im_end|>\n<|im_start|>user\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '<|im_end|>\n<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-sft-qwen-235b-ro-60246-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-60246-v1-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-85751-v1-uploader to finish
chaiml-sft-qwen-235b-ro-60246-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-60246-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-60246-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-85751-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Downloaded to shared memory in 136.714s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Downloaded to shared memory in 136.876s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed in 203.108s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-85751-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/added_tokens.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/.gitattributes
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/chat_template.jinja
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/generation_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/special_tokens_map.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/quantization_config.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/merges.txt
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/vocab.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model.safetensors.index.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-sft-qwen-235b-ro-85751-v1-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors?partNumber=4&uploadId=2~IPH1kA31PD94HtAEHIVGf3v3_IP6Ftr": write tcp 10.0.5.146:34102->216.153.53.63:443: write: broken pipe
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ERROR "cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~IPH1kA31PD94HtAEHIVGf3v3_IP6Ftr caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000e76ac69b8882baf5-006984d033-14a26732ad-default, host id:
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00027-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_1_version-int4-mixed in 319.655s
chaiml-sft-qwen-235b-ro-60246-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-60246-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-60246-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/.gitattributes
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/added_tokens.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/tokenizer_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/generation_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/chat_template.jinja
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/special_tokens_map.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/quantization_config.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/merges.txt
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model.safetensors.index.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/tokenizer.json
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/vocab.json
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00012-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00009-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00018-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00003-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00010-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00026-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00006-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00021-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00007-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00001-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00005-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00002-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00024-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00011-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00027-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00016-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00023-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00004-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00015-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00020-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00019-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00008-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00017-of-00027.safetensors
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00014-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-85751-v1-uploader completed after 466.09s with status: failed
Stopping job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Waiting for job on chaiml-sft-qwen-235b-ro-85751-v1-uploader to finish
chaiml-sft-qwen-235b-ro-85751-v1-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-sft-qwen-235b-ro-85751-v1-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  https://mk1.ai                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  The license key for the current software has been verified as      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  belonging to:                                                      ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Chai Research Corp.                                                ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ║                                                                     ║
chaiml-sft-qwen-235b-ro-85751-v1-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00020-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00014-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00004-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00001-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00016-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00026-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00011-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00009-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00008-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00015-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00005-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00019-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00023-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00006-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00024-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00003-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00010-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00018-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00017-of-00027.safetensors
chaiml-sft-qwen-235b-ro-60246-v1-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-60246-v1/model-00022-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-60246-v1-uploader completed after 592.96s with status: succeeded
Stopping job with name chaiml-sft-qwen-235b-ro-60246-v1-uploader
Pipeline stage VLLMUploader completed in 594.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.42s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-sft-qwen-235b-ro-60246-v1
Waiting for inference service chaiml-sft-qwen-235b-ro-60246-v1 to be ready
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Downloaded to shared memory in 154.023s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Processed model ChaiML/sft_qwen_235b_rob_new_2_version-int4-mixed in 217.621s
chaiml-sft-qwen-235b-ro-85751-v1-uploader: creating bucket guanaco-vllm-models
chaiml-sft-qwen-235b-ro-85751-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-sft-qwen-235b-ro-85751-v1-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/.gitattributes": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/added_tokens.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/chat_template.jinja": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/generation_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/merges.txt": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00001-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00002-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00003-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00004-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00005-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00006-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00007-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00008-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00009-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00010-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00011-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00012-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00013-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00014-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00015-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00016-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00017-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00018-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00019-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00020-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00021-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00022-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00023-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00024-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00026-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00027-of-00027.safetensors": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model.safetensors.index.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/quantization_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/special_tokens_map.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/tokenizer_config.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/vocab.json": object size matches
chaiml-sft-qwen-235b-ro-85751-v1-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-sft-qwen-235b-ro-85751-v1/model-00025-of-00027.safetensors
Job chaiml-sft-qwen-235b-ro-85751-v1-uploader completed after 272.29s with status: succeeded
Stopping job with name chaiml-sft-qwen-235b-ro-85751-v1-uploader
Pipeline stage VLLMUploader completed in 741.77s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.41s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-sft-qwen-235b-ro-85751-v1
Waiting for inference service chaiml-sft-qwen-235b-ro-85751-v1 to be ready
Inference service chaiml-sft-qwen-235b-ro-60246-v1 ready after 456.3508563041687s
Pipeline stage VLLMDeployer completed in 457.89s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.6831576824188232s
Received healthy response to inference request in 3.2752230167388916s
Received healthy response to inference request in 3.0649478435516357s
Received healthy response to inference request in 2.948944330215454s
Received healthy response to inference request in 2.940086603164673s
Received healthy response to inference request in 2.9528658390045166s
Received healthy response to inference request in 3.2884461879730225s
Received healthy response to inference request in 2.8155863285064697s
Received healthy response to inference request in 2.8440632820129395s
Received healthy response to inference request in 2.712222099304199s
Received healthy response to inference request in 2.797752618789673s
Received healthy response to inference request in 2.8235769271850586s
Received healthy response to inference request in 3.006124973297119s
Received healthy response to inference request in 2.7054388523101807s
Received healthy response to inference request in 2.8429062366485596s
Received healthy response to inference request in 2.9872934818267822s
Received healthy response to inference request in 2.910615921020508s
Received healthy response to inference request in 2.824007987976074s
Received healthy response to inference request in 2.733128547668457s
Received healthy response to inference request in 2.9522318840026855s
Received healthy response to inference request in 2.9221689701080322s
Received healthy response to inference request in 3.055537223815918s
Received healthy response to inference request in 2.839881658554077s
Received healthy response to inference request in 3.421782970428467s
Received healthy response to inference request in 3.0641307830810547s
Received healthy response to inference request in 3.1991145610809326s
Received healthy response to inference request in 2.8516061305999756s
Received healthy response to inference request in 2.769779920578003s
Received healthy response to inference request in 2.980849504470825s
Received healthy response to inference request in 3.051243543624878s
30 requests
0 failed requests
5th percentile: 2.721630001068115
10th percentile: 2.7661147832870485
20th percentile: 2.821978807449341
30th percentile: 2.8419988632202147
40th percentile: 2.887012004852295
50th percentile: 2.9445154666900635
60th percentile: 2.96405930519104
70th percentile: 3.0196605443954465
80th percentile: 3.064294195175171
90th percentile: 3.2765453338623045
95th percentile: 3.3617814183235164
99th percentile: 3.6073590159416202
mean time: 2.9754905303319297
Pipeline stage StressChecker completed in 111.56s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.99s
Shutdown handler de-registered
chaiml-sft-qwen-235b-ro_60246_v1 status is now deployed due to DeploymentManager action
chaiml-sft-qwen-235b-ro_60246_v1 status is now inactive due to auto deactivation removed underperforming models