developer_uid: rirv938
submission_id: chaiml-q235b-opus-v3-ju_23012_v1
model_name: chaiml-q235b-opus-v3-ju_23012_v1
model_group: ChaiML/q235b_opus_v3_jud
status: torndown
timestamp: 2026-04-05T18:52:19+00:00
num_battles: 10596
num_wins: 5709
celo_rating: 1319.96
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_opus_v3_judging-step396-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 1821417132032.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-opus-v3-ju_23012_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_opus_v3_judging-step396-merged
model_size: 1821B
ranking_group: single
us_pacific_date: 2026-04-02
win_ratio: 0.538788221970555
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|im_end|>', '<|user|>', '<|assistant|>', '####', '</think>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-opus-v3-ju-23012-v1-uploader
Waiting for job on chaiml-q235b-opus-v3-ju-23012-v1-uploader to finish
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Checking if ChaiML/q235b_opus_v3_judging-step396-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Downloading snapshot of ChaiML/q235b_opus_v3_judging-step396-merged...
2026-04-02T15:52:36.900419+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T15:53:36.994392+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T15:54:37.111437+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Downloaded in 160.014s
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Applying quantization...
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:55:31 INFO __init__.py L202: Patched transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock -> auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock
2026-04-02T15:55:37.199013+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:55:52 INFO base.py L448: `enable_opt_rtn` is turned on, set `--disable_opt_rtn` for higher speed at the cost of accuracy.
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:55:52 INFO base.py L486: using torch.bfloat16 for quantization tuning
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:55:53 INFO base.py L1573: Using predefined ignore_layers: ['mlp.gate']
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:55:55 INFO base.py L1081: start to compute imatrix
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:21 WARNING base.py L1201: MoE layer detected: optimized RTN is disabled for efficiency. Use `--enable_opt_rtn` to force-enable it for MoE layers.
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:23 INFO device.py L1468: 'peak_ram': 19.23GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:30 INFO device.py L1468: 'peak_ram': 20.55GB, 'peak_vram': 11.38GB
2026-04-02T15:56:37.281108+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:36 INFO device.py L1468: 'peak_ram': 21.91GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:48 INFO device.py L1468: 'peak_ram': 27.28GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:56:54 INFO device.py L1468: 'peak_ram': 27.28GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:01 INFO device.py L1468: 'peak_ram': 27.28GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:07 INFO device.py L1468: 'peak_ram': 27.28GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:17 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:24 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:30 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T15:57:37.367784+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:37 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:46 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:57:53 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:00 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:06 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:16 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:22 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:29 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:35 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T15:58:37.452711+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:45 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:51 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:58:58 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:04 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:14 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:20 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:27 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:33 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T15:59:37.557164+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:43 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:49 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 15:59:55 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:02 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:11 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:18 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:24 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:00:37.642961+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:31 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:41 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:47 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:00:54 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:00 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:16 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:22 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:01:37.739919+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:32 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:38 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:45 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:01:51 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:07 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:14 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:20 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:30 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:02:37.828003+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:36 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:43 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:49 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:02:59 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:06 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:12 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:18 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:28 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:03:38.007595+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:34 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:41 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:47 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:03:56 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:03 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:09 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:16 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:25 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:32 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:04:38.158974+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:38 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:44 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
Retrying (%r) after connection broken by '%r': %s
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:04:54 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:00 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:06 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:13 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:22 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:05:38.738649+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:29 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:35 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:42 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:51 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:05:58 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:04 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:13 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:20 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:26 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
2026-04-02T16:06:38.848476+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:32 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:42 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:48 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:06:54 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:01 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:10 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:16 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:23 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:29 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:07:38.967652+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:38 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:41 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:42 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict
chaiml-q235b-opus-v3-ju-23012-v1-uploader: 2026-04-02 16:07:42 INFO device.py L1468: 'peak_ram': 28.12GB, 'peak_vram': 11.38GB
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Checking if ChaiML/q235b_opus_v3_judging-step396-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Creating repo ChaiML/q235b_opus_v3_judging-step396-merged-W4A16 and uploading /dev/shm/model_output to it
2026-04-02T16:08:39.072199+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader:       
chaiml-q235b-opus-v3-ju-23012-v1-uploader: ---------- 2026-04-02 16:08:42 (0:01:00) ----------
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Files: hashed 32/32 (131.9G/131.9G) | pre-uploaded: 6/26 (24.5G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 20 | committing: 0 | waiting: 44
chaiml-q235b-opus-v3-ju-23012-v1-uploader: ---------------------------------------------------
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T16:09:39.261741+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader:       
chaiml-q235b-opus-v3-ju-23012-v1-uploader: ---------- 2026-04-02 16:09:42 (0:02:00) ----------
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Files: hashed 32/32 (131.9G/131.9G) | pre-uploaded: 26/26 (131.9G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-q235b-opus-v3-ju-23012-v1-uploader: ---------------------------------------------------
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Processed model ChaiML/q235b_opus_v3_judging-step396-merged in 1067.255s
chaiml-q235b-opus-v3-ju-23012-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-opus-v3-ju-23012-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-opus-v3-ju-23012-v1-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-opus-v3-ju-23012-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-opus-v3-ju-23012-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/config.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/tokenizer_config.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/generation_config.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/chat_template.jinja
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/quantization_config.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model.safetensors.index.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/tokenizer.json
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00025-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00025-of-00025.safetensors
2026-04-02T16:10:39.593158+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00010-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00010-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00023-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00023-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00020-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00020-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00003-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00003-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00005-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00005-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00021-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00021-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00012-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00012-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00014-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00014-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00004-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00004-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00006-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00006-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00018-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00018-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00022-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00022-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00009-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00009-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00002-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00002-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00024-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00024-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00019-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00019-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00011-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00011-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00017-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00017-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00015-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00015-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00007-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00007-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00016-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00016-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00013-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00013-of-00025.safetensors
chaiml-q235b-opus-v3-ju-23012-v1-uploader: cp /dev/shm/model_output/model-00008-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-23012-v1/default/model-00008-of-00025.safetensors
Job chaiml-q235b-opus-v3-ju-23012-v1-uploader completed after 1155.35s with status: succeeded
Stopping job with name chaiml-q235b-opus-v3-ju-23012-v1-uploader
Pipeline stage VLLMUploader completed in 1155.98s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.13s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.72s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-opus-v3-ju-23012-v1
Waiting for inference service chaiml-q235b-opus-v3-ju-23012-v1 to be ready
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:11:39.780675+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:12:39.917966+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:13:40.031325+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:14:40.142755+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T16:15:40.251750+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T16:16:40.363919+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:17:40.516863+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:18:40.625897+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:19:40.739428+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:20:40.844870+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:21:40.947850+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:22:41.058439+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:23:41.222471+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:24:41.329334+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:25:41.459756+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:26:41.579614+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:27:41.696205+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:28:41.812317+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:29:41.930653+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:30:42.093387+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:31:42.254078+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
2026-04-02T16:32:42.368536+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:33:42.513118+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T16:34:42.673167+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:35:42.792208+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
2026-04-02T16:36:43.026163+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Inference service chaiml-q235b-opus-v3-ju-23012-v1 ready after 1589.263953924179s
Pipeline stage VLLMDeployer completed in 1590.00s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.6358680725097656s
Received healthy response to inference request in 1.5923316478729248s
Received healthy response to inference request in 1.4689414501190186s
Received healthy response to inference request in 1.5975277423858643s
Received healthy response to inference request in 1.4599053859710693s
Received healthy response to inference request in 1.4716699123382568s
Received healthy response to inference request in 1.4569826126098633s
Received healthy response to inference request in 1.468790054321289s
2026-04-02T16:37:43.156300+00:00 monitor updated for chaiml-q235b-opus-v3-ju_23012_v1
Received healthy response to inference request in 1.5615715980529785s
Received healthy response to inference request in 2.0232553482055664s
Received healthy response to inference request in 2.0310680866241455s
Received healthy response to inference request in 1.501338005065918s
Received healthy response to inference request in 1.5428996086120605s
Received healthy response to inference request in 1.789381742477417s
Received healthy response to inference request in 1.5456390380859375s
Received healthy response to inference request in 1.612044095993042s
Received healthy response to inference request in 1.4923899173736572s
Received healthy response to inference request in 1.5698740482330322s
Received healthy response to inference request in 2.016319751739502s
Received healthy response to inference request in 3.5032708644866943s
Received healthy response to inference request in 2.1005828380584717s
Received healthy response to inference request in 1.6238934993743896s
Received healthy response to inference request in 1.609717607498169s
Received healthy response to inference request in 1.754638433456421s
Received healthy response to inference request in 1.480126142501831s
Received healthy response to inference request in 1.6607818603515625s
Received healthy response to inference request in 1.5088422298431396s
Received healthy response to inference request in 1.5752956867218018s
Received healthy response to inference request in 1.8928697109222412s
Received healthy response to inference request in 1.9118666648864746s
30 requests
0 failed requests
5th percentile: 1.4639034867286682
10th percentile: 1.4689263105392456
20th percentile: 1.489937162399292
30th percentile: 1.5326823949813841
40th percentile: 1.5665530681610107
50th percentile: 1.5949296951293945
60th percentile: 1.616783857345581
70th percentile: 1.7650614261627195
80th percentile: 1.9327572822570804
90th percentile: 2.038019561767578
95th percentile: 2.87206125259399
99th percentile: 3.597414882183075
mean time: 1.7819894552230835
Pipeline stage StressChecker completed in 58.15s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.90s
Shutdown handler de-registered
chaiml-q235b-opus-v3-ju_23012_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-opus-v3-ju_23012_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-opus-v3-ju_23012_v1 status is now torndown due to DeploymentManager action