chaiml-q235b-opus-v3-ju_99162

developer_uid: rirv938
submission_id: chaiml-q235b-opus-v3-ju_99162_v1
model_name: chaiml-q235b-opus-v3-ju_99162_v1
model_group: ChaiML/q235b_opus_v3_jud
status: torndown
timestamp: 2026-04-05T18:22:12+00:00
num_battles: 11401
num_wins: 6126
celo_rating: 8389.84
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_opus_v3_judging-step198-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 1821417132032.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-opus-v3-ju_99162_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_opus_v3_judging-step198-merged
model_size: 1821B
ranking_group: single
us_pacific_date: 2026-04-02
win_ratio: 0.5373212876063503
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</think>', '<|assistant|>', '<|user|>', '####', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-opus-v3-ju-99162-v1-uploader
Waiting for job on chaiml-q235b-opus-v3-ju-99162-v1-uploader to finish
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Using quantization_mode: w4a16
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Checking if ChaiML/q235b_opus_v3_judging-step198-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Downloading snapshot of ChaiML/q235b_opus_v3_judging-step198-merged...
2026-04-02T15:52:41.427095+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T15:53:41.597519+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T15:54:41.791443+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Downloaded in 165.476s
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Applying quantization...
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:55:12 INFO __init__.py L202: Patched transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock -> auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:55:34 INFO base.py L448: `enable_opt_rtn` is turned on, set `--disable_opt_rtn` for higher speed at the cost of accuracy.[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:55:34 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:55:34 INFO base.py L1573: Using predefined ignore_layers: ['mlp.gate'][0m
2026-04-02T15:55:42.231215+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:55:37 INFO base.py L1081: start to compute imatrix[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [33;1m2026-04-02 15:56:09 WARNING base.py L1201: MoE layer detected: optimized RTN is disabled for efficiency. Use `--enable_opt_rtn` to force-enable it for MoE layers.[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:56:12 INFO device.py L1468:  'peak_ram': 19.31GB, 'peak_vram': 11.38GB[0m
2026-04-02T15:56:42.428807+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:56:37 INFO device.py L1468:  'peak_ram': 21.92GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:56:56 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:57:08 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:57:20 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:57:34 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 11.38GB[0m
2026-04-02T15:57:42.527261+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:57:52 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:58:04 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:58:15 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:58:28 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T15:58:42.637475+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:58:43 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:58:54 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:05 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:16 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:29 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:36 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T15:59:42.786263+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:43 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 15:59:50 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: HTTPConnectionPool(host='chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:00 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:08 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:17 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:24 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:33 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:40 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:00:42.877616+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:46 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:00:53 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:02 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:09 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:15 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:22 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:31 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:01:42.971299+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:38 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:44 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:01:51 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:00 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:07 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:13 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:20 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:29 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:36 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:02:43.072029+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:42 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:52 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:02:58 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:04 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:11 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:20 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:27 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:33 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:39 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:03:43.168891+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:49 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:03:55 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:02 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:08 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:18 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:24 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:30 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:04:43.262362+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:37 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:46 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:53 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:04:59 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:05 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:15 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:21 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:27 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:34 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:05:43.374440+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:43 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:49 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:05:56 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:02 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:12 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:18 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:24 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:31 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:06:43.474232+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:47 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:53 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:06:59 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:09 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:15 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:21 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:31 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:37 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:07:43.580772+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:44 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:50 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:07:59 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:06 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:12 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:18 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:28 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:34 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:41 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
2026-04-02T16:08:43.688552+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:47 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:57 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:08:59 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [33;1m2026-04-02 16:09:00 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: [38;20m2026-04-02 16:09:00 INFO device.py L1468:  'peak_ram': 28.15GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Checking if ChaiML/q235b_opus_v3_judging-step198-merged-W4A16 already exists in ChaiML
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Creating repo ChaiML/q235b_opus_v3_judging-step198-merged-W4A16 and uploading /dev/shm/model_output to it
chaiml-q235b-opus-v3-ju-99162-v1-uploader: ---------- 2026-04-02 16:09:00 (0:00:00) ----------
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Files:   hashed 7/32 (21.5M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+29 unsure) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Workers: hashing: 25 | get upload mode: 3 | pre-uploading: 0 | committing: 0 | waiting: 36
chaiml-q235b-opus-v3-ju-99162-v1-uploader: ---------------------------------------------------
2026-04-02T16:09:43.791548+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-opus-v3-ju-99162-v1-uploader: ---------- 2026-04-02 16:10:00 (0:01:00) ----------
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Files:   hashed 32/32 (131.9G/131.9G) | pre-uploaded: 17/26 (83.6G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 9 | committing: 0 | waiting: 55
chaiml-q235b-opus-v3-ju-99162-v1-uploader: ---------------------------------------------------
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:10:43.903665+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
chaiml-q235b-opus-v3-ju-99162-v1-uploader:                              Processed model ChaiML/q235b_opus_v3_judging-step198-merged in 1116.267s
chaiml-q235b-opus-v3-ju-99162-v1-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-opus-v3-ju-99162-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-opus-v3-ju-99162-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-opus-v3-ju-99162-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/config.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/generation_config.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/tokenizer_config.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/quantization_config.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/chat_template.jinja
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model.safetensors.index.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/tokenizer.json
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00025-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00025-of-00025.safetensors
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00015-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00015-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00003-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00003-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00021-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00021-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00020-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00020-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00009-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00009-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00016-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00016-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00007-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00007-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00013-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00013-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00008-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00008-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00004-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00004-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00019-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00019-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00001-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00001-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00010-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00010-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00002-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00002-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00012-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00012-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00011-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00011-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00006-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00006-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00018-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00018-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00017-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00017-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00014-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00014-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00024-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00024-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00005-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00005-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00023-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00023-of-00025.safetensors
chaiml-q235b-opus-v3-ju-99162-v1-uploader: cp /dev/shm/model_output/model-00022-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-opus-v3-ju-99162-v1/default/model-00022-of-00025.safetensors
2026-04-02T16:11:44.030240+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Job chaiml-q235b-opus-v3-ju-99162-v1-uploader completed after 1202.51s with status: succeeded
Stopping job with name chaiml-q235b-opus-v3-ju-99162-v1-uploader
Pipeline stage VLLMUploader completed in 1203.09s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.39s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-opus-v3-ju-99162-v1
Waiting for inference service chaiml-q235b-opus-v3-ju-99162-v1 to be ready
2026-04-02T16:12:44.143699+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:13:44.259571+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:14:44.383831+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:15:44.495710+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-02T16:16:44.619477+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:17:44.767484+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:18:44.986612+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:19:45.123861+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:20:45.257270+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:21:45.398225+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
2026-04-02T16:22:45.605329+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Retrying (%r) after connection broken by '%r': %s
2026-04-02T16:23:45.937332+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Inference service chaiml-q235b-opus-v3-ju-99162-v1 ready after 764.2417397499084s
Pipeline stage VLLMDeployer completed in 764.92s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.6687378883361816s
Received healthy response to inference request in 3.9834325313568115s
Received healthy response to inference request in 1.950645923614502s
Received healthy response to inference request in 1.5208885669708252s
2026-04-02T16:24:46.149617+00:00 monitor updated for chaiml-q235b-opus-v3-ju_99162_v1
Received healthy response to inference request in 1.4875950813293457s
Received healthy response to inference request in 1.9458868503570557s
Received healthy response to inference request in 1.4771900177001953s
Received healthy response to inference request in 1.775529384613037s
Received healthy response to inference request in 1.5322887897491455s
Received healthy response to inference request in 1.5360569953918457s
Received healthy response to inference request in 1.6294138431549072s
Received healthy response to inference request in 1.8163843154907227s
Received healthy response to inference request in 1.4680969715118408s
Received healthy response to inference request in 1.5244667530059814s
Received healthy response to inference request in 1.5381355285644531s
Received healthy response to inference request in 1.4919660091400146s
Received healthy response to inference request in 1.4906940460205078s
Received healthy response to inference request in 1.535872220993042s
Received healthy response to inference request in 1.510317087173462s
Received healthy response to inference request in 1.480093240737915s
Received healthy response to inference request in 1.620609998703003s
Received healthy response to inference request in 1.5523312091827393s
Received healthy response to inference request in 1.5492689609527588s
Received healthy response to inference request in 1.6692452430725098s
Received healthy response to inference request in 1.49684739112854s
Received healthy response to inference request in 1.615067958831787s
Received healthy response to inference request in 1.8578746318817139s
Received healthy response to inference request in 1.5790174007415771s
Received healthy response to inference request in 1.6166961193084717s
Received healthy response to inference request in 1.5735032558441162s
30 requests
0 failed requests
5th percentile: 1.4784964680671693
10th percentile: 1.4868448972702026
20th percentile: 1.495871114730835
30th percentile: 1.5233932971954345
40th percentile: 1.5359830856323242
50th percentile: 1.550800085067749
60th percentile: 1.593437623977661
70th percentile: 1.6232511520385742
80th percentile: 1.7837003707885744
90th percentile: 1.9463627576828002
95th percentile: 2.895596504211421
99th percentile: 3.8921710848808293
mean time: 1.7498051404953003
Pipeline stage StressChecker completed in 57.55s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.27s
Shutdown handler de-registered
chaiml-q235b-opus-v3-ju_99162_v1 status is now deployed due to DeploymentManager action
chaiml-q235b-opus-v3-ju_99162_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-opus-v3-ju_99162_v1 status is now torndown due to DeploymentManager action