chaiml-q235b-opus-judge_16335

developer_uid: rirv938
submission_id: chaiml-q235b-opus-judge_16335_v1
model_name: chaiml-q235b-opus-judge_16335_v1
model_group: ChaiML/q235b_opus_judge_
status: torndown
timestamp: 2026-04-05T10:52:08+00:00
num_battles: 11829
num_wins: 6352
celo_rating: 8393.47
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 1821417132032.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-opus-judge_16335_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged
model_size: 1821B
ranking_group: single
us_pacific_date: 2026-04-02
win_ratio: 0.5369853749260293
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|assistant|>', '</s>', '####', '</think>', '<|im_end|>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-opus-judge-16335-v1-uploader
Waiting for job on chaiml-q235b-opus-judge-16335-v1-uploader to finish
chaiml-q235b-opus-judge-16335-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-opus-judge-16335-v1-uploader: Checking if ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-judge-16335-v1-uploader: Downloading snapshot of ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged...
2026-04-02T07:22:41.128020+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
2026-04-02T07:23:41.257853+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
2026-04-02T07:24:41.712180+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: Downloaded in 180.013s
chaiml-q235b-opus-judge-16335-v1-uploader: Applying quantization...
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:25:28 INFO __init__.py L202: Patched transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock -> auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock[0m
2026-04-02T07:25:41.802911+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:25:51 INFO base.py L448: `enable_opt_rtn` is turned on, set `--disable_opt_rtn` for higher speed at the cost of accuracy.[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:25:51 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:25:51 INFO base.py L1573: Using predefined ignore_layers: ['mlp.gate'][0m
chaiml-q235b-opus-judge-16335-v1-uploader: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
chaiml-q235b-opus-judge-16335-v1-uploader: [33;1m2026-04-02 07:26:29 WARNING base.py L1201: MoE layer detected: optimized RTN is disabled for efficiency. Use `--enable_opt_rtn` to force-enable it for MoE layers.[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:26:31 INFO device.py L1468:  'peak_ram': 18.86GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:26:41.890378+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:26:43 INFO device.py L1468:  'peak_ram': 20.11GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:26:57 INFO device.py L1468:  'peak_ram': 21.36GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:27:15 INFO device.py L1468:  'peak_ram': 26.99GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:27:27 INFO device.py L1468:  'peak_ram': 26.99GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:27:41.972915+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:27:41 INFO device.py L1468:  'peak_ram': 26.99GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:27:53 INFO device.py L1468:  'peak_ram': 26.99GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:28:09 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:28:20 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:28:32 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:28:42.056716+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:28:46 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:29:01 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:29:15 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:29:28 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:29:42.142631+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:29:40 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:29:55 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:30:07 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:30:19 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:30:30 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:30:42.226170+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:30:46 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:30:58 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:31:09 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:31:22 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:31:37 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:31:42.323947+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:31:48 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:31:59 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:08 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:19 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:29 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:38 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:32:42.410718+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:48 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:32:58 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:04 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:10 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:17 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:26 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:33 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:39 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:33:42.513841+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:45 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:33:55 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:01 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:08 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
Failed to get request counts for guanaco-submitter. Falling back to default
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:18 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:24 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:31 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:37 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:34:42.598557+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:47 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:34:54 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:00 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:06 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:16 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:23 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:35 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:35:42.755153+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:45 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:51 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:35:57 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:04 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:13 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:19 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:26 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:32 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:36:42.840327+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:42 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:48 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:36:54 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:01 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:10 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:16 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:23 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:29 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:38 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:37:42.930555+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:45 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:51 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:37:57 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:07 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:13 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:19 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:26 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:35 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:41 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:38:43.017769+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:48 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:38:57 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:04 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:10 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:16 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:26 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:32 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
2026-04-02T07:39:43.241981+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:39 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:45 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:39:54 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:01 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:07 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:13 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:23 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:25 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [33;1m2026-04-02 07:40:26 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-q235b-opus-judge-16335-v1-uploader: [38;20m2026-04-02 07:40:26 INFO device.py L1468:  'peak_ram': 27.42GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-judge-16335-v1-uploader: Checking if ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-judge-16335-v1-uploader: Creating repo ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-opus-judge-16335-v1-uploader: ---------- 2026-04-02 07:40:27 (0:00:00) ----------
chaiml-q235b-opus-judge-16335-v1-uploader: Files:   hashed 7/32 (21.5M/131.9G) | pre-uploaded: 0/0 (0.0/131.9G) (+31 unsure) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-judge-16335-v1-uploader: Workers: hashing: 25 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 37
chaiml-q235b-opus-judge-16335-v1-uploader: ---------------------------------------------------
2026-04-02T07:40:43.347840+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-opus-judge-16335-v1-uploader: ---------- 2026-04-02 07:41:27 (0:01:00) ----------
chaiml-q235b-opus-judge-16335-v1-uploader: Files:   hashed 32/32 (131.9G/131.9G) | pre-uploaded: 9/26 (40.6G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-judge-16335-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 17 | committing: 0 | waiting: 47
chaiml-q235b-opus-judge-16335-v1-uploader: ---------------------------------------------------
2026-04-02T07:41:43.442638+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-opus-judge-16335-v1-uploader: ---------- 2026-04-02 07:42:27 (0:02:00) ----------
chaiml-q235b-opus-judge-16335-v1-uploader: Files:   hashed 32/32 (131.9G/131.9G) | pre-uploaded: 26/26 (131.9G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-judge-16335-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-q235b-opus-judge-16335-v1-uploader: ---------------------------------------------------
2026-04-02T07:42:43.535249+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader:                              Processed model ChaiML/q235b_opus_judge_dpo_with_repetition_fixed-step210-merged in 1213.002s
chaiml-q235b-opus-judge-16335-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-judge-16335-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-opus-judge-16335-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-judge-16335-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-judge-16335-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-judge-16335-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-judge-16335-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-opus-judge-16335-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-opus-judge-16335-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-opus-judge-16335-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-opus-judge-16335-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-opus-judge-16335-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/generation_config.json
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/tokenizer_config.json
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/chat_template.jinja
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/config.json
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/quantization_config.json
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/tokenizer.json
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model.safetensors.index.json
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: 65158320-082e-9f3f-9f5d-5853ca9ab24c, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: 1c28ef00-d07c-9c14-be47-03a624c595a5, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: 1de1a53e-627f-914a-94e1-0f1720c9900f, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: dd9e50e8-9646-942b-802f-abb493ec762e, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: 86009a3b-445d-911d-a264-f223e7f38e1d, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: 6f157a62-1a13-986d-adc3-4af4565dd4c2, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: DEBUG retryable error: InternalError: We encountered an internal error, please try again.
chaiml-q235b-opus-judge-16335-v1-uploader: 	status code: 500, request id: d320aec5-02d6-9241-9b33-180dd96cc536, host id: 
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00025-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00025-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00014-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00014-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00012-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00012-of-00025.safetensors
2026-04-02T07:43:43.626109+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00001-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00001-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00016-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00016-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00008-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00008-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00021-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00021-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00015-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00015-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00002-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00002-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00019-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00019-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00018-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00018-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00003-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00003-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00024-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00024-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00017-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00017-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00010-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00010-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00006-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00006-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00007-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00007-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00009-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00009-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00011-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00011-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00023-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00023-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00004-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00004-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00013-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00013-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00005-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00005-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00020-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00020-of-00025.safetensors
chaiml-q235b-opus-judge-16335-v1-uploader: cp /dev/shm/model_output/model-00022-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-judge-16335-v1/default/model-00022-of-00025.safetensors
Job chaiml-q235b-opus-judge-16335-v1-uploader completed after 1340.65s with status: succeeded
Stopping job with name chaiml-q235b-opus-judge-16335-v1-uploader
Pipeline stage VLLMUploader completed in 1341.17s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.92s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-opus-judge-16335-v1
Waiting for inference service chaiml-q235b-opus-judge-16335-v1 to be ready
2026-04-02T07:44:43.730567+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
2026-04-02T07:45:43.836029+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
2026-04-02T07:46:43.938203+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
Inference service chaiml-q235b-opus-judge-16335-v1 ready after 180.7584388256073s
Pipeline stage VLLMDeployer completed in 181.33s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.730393886566162s
Received healthy response to inference request in 1.706817388534546s
Received healthy response to inference request in 1.5725362300872803s
Received healthy response to inference request in 1.6293704509735107s
Received healthy response to inference request in 1.6031625270843506s
Received healthy response to inference request in 1.4988207817077637s
Received healthy response to inference request in 1.4854445457458496s
Received healthy response to inference request in 1.6590707302093506s
Received healthy response to inference request in 1.5244359970092773s
Received healthy response to inference request in 1.4591600894927979s
Received healthy response to inference request in 1.5708339214324951s
Received healthy response to inference request in 3.695284843444824s
Received healthy response to inference request in 1.4955966472625732s
Received healthy response to inference request in 1.4571316242218018s
Received healthy response to inference request in 1.516615867614746s
Received healthy response to inference request in 1.6562001705169678s
Received healthy response to inference request in 1.4737627506256104s
Received healthy response to inference request in 1.4948101043701172s
Received healthy response to inference request in 1.4688303470611572s
Received healthy response to inference request in 1.5503036975860596s
2026-04-02T07:47:44.031545+00:00 monitor updated for chaiml-q235b-opus-judge_16335_v1
Received healthy response to inference request in 1.701986312866211s
Received healthy response to inference request in 1.6965138912200928s
Received healthy response to inference request in 1.4535727500915527s
Received healthy response to inference request in 1.5147545337677002s
Received healthy response to inference request in 1.4793806076049805s
Received healthy response to inference request in 1.5403764247894287s
Received healthy response to inference request in 1.716766119003296s
Received healthy response to inference request in 1.8164746761322021s
Received healthy response to inference request in 1.5405728816986084s
Received healthy response to inference request in 1.6097736358642578s
30 requests
0 failed requests
5th percentile: 1.45804443359375
10th percentile: 1.4678633213043213
20th percentile: 1.4842317581176758
30th percentile: 1.4978535413742065
40th percentile: 1.5213079452514648
50th percentile: 1.545438289642334
60th percentile: 1.5847867488861083
70th percentile: 1.6374193668365478
80th percentile: 1.6976083755493163
90th percentile: 1.7267369747161867
95th percentile: 2.849820268154139
99th percentile: 3.720212264060974
mean time: 1.7106251478195191
Pipeline stage StressChecker completed in 54.90s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.66s
Shutdown handler de-registered
chaiml-q235b-opus-judge_16335_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-opus-judge_16335_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-opus-judge_16335_v1 status is now torndown due to DeploymentManager action