developer_uid: rirv938
submission_id: chaiml-qwen-235b-95p-5f_92674_v3
model_name: chaiml-qwen-235b-95p-5f_92674_v3
model_group: ChaiML/qwen_235B_95p_5f_
status: torndown
timestamp: 2026-01-31T19:18:42+00:00
num_battles: 12637
num_wins: 7025
celo_rating: 1337.47
family_friendly_score: 0.5658000000000001
family_friendly_standard_error: 0.007009570029609519
submission_type: basic
model_repo: ChaiML/qwen_235B_95p_5f_reward_dpo_10k_samples_opusdv1-int4-mixed
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 4
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-qwen-235b-95p-5f_92674_v3
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/qwen_235B_95p_5f_reward_dpo_10k_samples_opusdv1-int4-mixed
model_size: 19B
ranking_group: single
us_pacific_date: 2026-01-28
win_ratio: 0.5559072564690987
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '</s>', '<|im_end|>', '<|user|>', '</think>', '####'], 'max_input_tokens': 2048, 'best_of': 4, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-qwen-235b-95p-5f-92674-v3-uploader
Waiting for job on chaiml-qwen-235b-95p-5f-92674-v3-uploader to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v17-uploader to finish
chaiml-qwen-235b-95p-5f-92674-v3-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-qwen-235b-95p-5f-92674-v3-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║                                                                     ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: no job control in this shell
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-kimid-v9-opusdv-83165-v17-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  https://mk1.ai                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  The license key for the current software has been verified as      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  belonging to:                                                      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Chai Research Corp.                                                ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-qwen-235b-95p-5f-92674-v3-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Version: 0.30.6+torch280                                           ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  https://mk1.ai                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  The license key for the current software has been verified as      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  belonging to:                                                      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Chai Research Corp.                                                ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Downloaded to shared memory in 147.854s
chaiml-kimid-v9-opusdv-83165-v17-uploader: Downloaded to shared memory in 158.502s
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Processed model ChaiML/qwen_235B_95p_5f_reward_dpo_10k_samples_opusdv1-int4-mixed in 200.603s
chaiml-qwen-235b-95p-5f-92674-v3-uploader: creating bucket guanaco-vllm-models
chaiml-qwen-235b-95p-5f-92674-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-qwen-235b-95p-5f-92674-v3-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/added_tokens.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/chat_template.jinja
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/.gitattributes
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/special_tokens_map.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/tokenizer_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/generation_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/merges.txt
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/quantization_config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/config.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/vocab.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model.safetensors.index.json
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 209.817s
chaiml-kimid-v9-opusdv-83165-v17-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v17-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/.gitattributes
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/added_tokens.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/chat_template.jinja
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/README.md
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/generation_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/special_tokens_map.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/quantization_config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/config.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/merges.txt
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/vocab.json
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-kimid-v9-opusdv-83165-v17-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors?partNumber=18&uploadId=2~DmM0zErtkMt4Aca_Qr0bke0RFJzHK0Y": write tcp 10.0.46.246:56160->216.153.53.63:443: write: connection reset by peer
chaiml-kimid-v9-opusdv-83165-v17-uploader: ERROR "cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~DmM0zErtkMt4Aca_Qr0bke0RFJzHK0Y caused by: SignatureDoesNotMatch:  status code: 403, request id: tx00000a460ae867d43fc2d-00697a46d7-14a19326cd-default, host id:
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00010-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00015-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00025-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00007-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00021-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00013-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00022-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00009-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00016-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00004-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00020-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00015-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00012-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00017-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00008-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00019-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00011-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00003-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00010-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00001-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00004-of-00027.safetensors
chaiml-qwen-235b-95p-5f-92674-v3-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-qwen-235b-95p-5f-92674-v3/model-00025-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v17-uploader completed after 773.46s with status: failed
Stopping job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v17-uploader to finish
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v17-uploader: bash: no job control in this shell
chaiml-kimid-v9-opusdv-83165-v17-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v9-opusdv-83165-v17-uploader:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
Job chaiml-qwen-235b-95p-5f-92674-v3-uploader completed after 781.19s with status: succeeded
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███░█████░███   ░███  ███     ░███                               ║
Stopping job with name chaiml-qwen-235b-95p-5f-92674-v3-uploader
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
Pipeline stage VLLMUploader completed in 783.26s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   ░███      ░███   ░███ ░░███    ░███                               ║
run pipeline stage %s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║   █████     █████  █████ ░░████  █████                              ║
Running pipeline stage VLLMTemplater
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
Pipeline stage VLLMTemplater completed in 0.64s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Version: 0.30.6+torch280                                           ║
run pipeline stage %s
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Features: FLYWHEEL, CUDA                                           ║
Running pipeline stage VLLMDeployer
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
Creating inference service chaiml-qwen-235b-95p-5f-92674-v3
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  https://mk1.ai                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
Waiting for inference service chaiml-qwen-235b-95p-5f-92674-v3 to be ready
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  The license key for the current software has been verified as      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  belonging to:                                                      ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Chai Research Corp.                                                ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ║                                                                     ║
chaiml-kimid-v9-opusdv-83165-v17-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v17-uploader: Downloaded to shared memory in 151.290s
chaiml-kimid-v9-opusdv-83165-v17-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 214.294s
chaiml-kimid-v9-opusdv-83165-v17-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v17-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v17-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/.gitattributes": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/README.md": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/added_tokens.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/chat_template.jinja": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/generation_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/merges.txt": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00001-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00002-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00003-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00004-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00005-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00006-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00007-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00008-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00009-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00010-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00011-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00012-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00013-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00014-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00015-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00017-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00018-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00019-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00020-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00021-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00022-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00023-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00024-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00025-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00026-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00027-of-00027.safetensors": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model.safetensors.index.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/quantization_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/special_tokens_map.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/tokenizer_config.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: DEBUG "sync /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/vocab.json": object size matches
chaiml-kimid-v9-opusdv-83165-v17-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v17/model-00016-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v17-uploader completed after 263.3s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-83165-v17-uploader
Pipeline stage VLLMUploader completed in 1040.86s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.55s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-83165-v17
Waiting for inference service chaiml-kimid-v9-opusdv-83165-v17 to be ready
Inference service chaiml-qwen-235b-95p-5f-92674-v3 ready after 437.5312707424164s
Pipeline stage VLLMDeployer completed in 439.37s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9888055324554443s
Received healthy response to inference request in 2.9072659015655518s
Received healthy response to inference request in 2.1958885192871094s
Received healthy response to inference request in 2.5546646118164062s
Received healthy response to inference request in 2.4633312225341797s
Received healthy response to inference request in 2.2723567485809326s
Received healthy response to inference request in 2.280489444732666s
Received healthy response to inference request in 2.222970962524414s
Received healthy response to inference request in 2.385891914367676s
Received healthy response to inference request in 2.530067205429077s
Received healthy response to inference request in 2.144638776779175s
Received healthy response to inference request in 2.317072868347168s
Received healthy response to inference request in 2.339120864868164s
Received healthy response to inference request in 2.862046957015991s
Received healthy response to inference request in 2.1863958835601807s
Received healthy response to inference request in 2.3233320713043213s
Received healthy response to inference request in 2.3718531131744385s
Received healthy response to inference request in 2.2574870586395264s
Received healthy response to inference request in 2.201815366744995s
Received healthy response to inference request in 2.8437156677246094s
Received healthy response to inference request in 2.494262456893921s
Received healthy response to inference request in 2.2720706462860107s
Received healthy response to inference request in 2.2844760417938232s
Received healthy response to inference request in 2.2059648036956787s
Received healthy response to inference request in 2.626243829727173s
Received healthy response to inference request in 2.4604499340057373s
Received healthy response to inference request in 2.7456774711608887s
Received healthy response to inference request in 2.344508171081543s
Received healthy response to inference request in 3.046015501022339s
Received healthy response to inference request in 2.5127720832824707s
30 requests
0 failed requests
5th percentile: 2.1906675696372986
10th percentile: 2.2012226819992065
20th percentile: 2.250583839416504
30th percentile: 2.278049635887146
40th percentile: 2.32082839012146
50th percentile: 2.3581806421279907
60th percentile: 2.4616024494171143
70th percentile: 2.5179606199264524
80th percentile: 2.6501305580139163
90th percentile: 2.8665688514709475
95th percentile: 2.9521126985549926
99th percentile: 3.0294246101379394
mean time: 2.454721721013387
Pipeline stage StressChecker completed in 90.53s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-qwen-235b-95p-5f_92674_v3 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2948.70s
Shutdown handler de-registered
chaiml-qwen-235b-95p-5f_92674_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-qwen-235b-95p-5f_92674_v3 status is now torndown due to DeploymentManager action