chaiml-q235b-kimid-v2-c_39855

developer_uid: rirv938
submission_id: chaiml-q235b-kimid-v2-c_39855_v2
model_name: chaiml-q235b-kimid-v2-c_39855_v2
model_group: ChaiML/q235b_kimid_v2_cl
status: deployed
timestamp: 2026-04-03T17:51:01+00:00
num_battles: 344604
num_wins: 192003
celo_rating: 1344.31
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/q235b_kimid_v2_cliche_plus_original-step444-merged
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 1821417132032.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-q235b-kimid-v2-c_39855_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/q235b_kimid_v2_cliche_plus_original-step444-merged
model_size: 1821B
ranking_group: single
us_pacific_date: 2026-04-03
win_ratio: 0.5571699690079047
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</think>', '<|im_end|>', '####', '<|user|>', '<|assistant|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-kimid-v2-c-39855-v2-uploader
Waiting for job on chaiml-q235b-kimid-v2-c-39855-v2-uploader to finish
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Using quantization_mode: w4a16
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Checking if ChaiML/q235b_kimid_v2_cliche_plus_original-step444-merged-W4A16 already exists in ChaiML
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Downloading snapshot of ChaiML/q235b_kimid_v2_cliche_plus_original-step444-merged...
2026-04-03T17:26:51.320823+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
2026-04-03T17:27:51.422440+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
2026-04-03T17:28:51.621480+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Downloaded in 158.215s
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Applying quantization...
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:29:13 INFO __init__.py L202: Patched transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock -> auto_round.modeling.unfused_moe.qwen3_moe.LinearQwen3MoeSparseMoeBlock[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:29:28 INFO base.py L448: `enable_opt_rtn` is turned on, set `--disable_opt_rtn` for higher speed at the cost of accuracy.[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:29:28 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:29:28 INFO base.py L1573: Using predefined ignore_layers: ['mlp.gate'][0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:29:30 INFO base.py L1081: start to compute imatrix[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-04-03T17:29:51.728722+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [33;1m2026-04-03 17:30:01 WARNING base.py L1201: MoE layer detected: optimized RTN is disabled for efficiency. Use `--enable_opt_rtn` to force-enable it for MoE layers.[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:03 INFO device.py L1468:  'peak_ram': 19.17GB, 'peak_vram': 11.38GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:12 INFO device.py L1468:  'peak_ram': 20.54GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:20 INFO device.py L1468:  'peak_ram': 21.86GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:35 INFO device.py L1468:  'peak_ram': 27.24GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:44 INFO device.py L1468:  'peak_ram': 27.24GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:30:51.816871+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:30:53 INFO device.py L1468:  'peak_ram': 27.24GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:02 INFO device.py L1468:  'peak_ram': 27.24GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:15 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:21 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:28 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:35 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:47 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:31:51.901300+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:31:54 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:01 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:07 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:19 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:26 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:33 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:40 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:32:51.999371+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:51 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:32:58 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:05 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:11 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:23 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:30 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:37 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:33:52.096037+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:44 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:33:55 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:02 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:09 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:16 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:27 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:34 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:41 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:34:52.183634+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:48 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:34:59 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:06 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:13 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:20 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:31 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:38 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:45 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:35:52.294883+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:35:56 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:03 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:10 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:17 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:28 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:35 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:41 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:36:48 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:36:52.392174+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:00 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:07 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:13 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:20 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:32 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:38 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:37:52.492568+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:45 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:37:52 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:03 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:10 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:17 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:23 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:35 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:41 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:38:52.629686+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:38:48 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:06 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:13 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:19 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:26 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:37 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:44 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:39:52.790653+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:50 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:39:57 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:08 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:15 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:22 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
Failed to get request counts for guanaco-submitter. Falling back to default
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:28 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:39 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:46 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:40:53.155476+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:40:53 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:04 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:11 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:17 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:24 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:35 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:42 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:48 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
2026-04-03T17:41:53.376756+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:41:55 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:06 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:13 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:20 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:26 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:38 INFO device.py L1468:  'peak_ram': 28.08GB, 'peak_vram': 11.38GB[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: [38;20m2026-04-03 17:42:41 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------- 2026-04-03 17:42:43 (0:00:00) ----------
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Files:   hashed 7/32 (21.5M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+30 unsure) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Workers: hashing: 25 | get upload mode: 3 | pre-uploading: 0 | committing: 0 | waiting: 36
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------------------------------------------------
2026-04-03T17:42:53.503519+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------- 2026-04-03 17:43:43 (0:01:00) ----------
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Files:   hashed 32/32 (131.9G/131.9G) | pre-uploaded: 9/26 (40.6G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 17 | committing: 0 | waiting: 47
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------------------------------------------------
2026-04-03T17:43:53.610038+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------- 2026-04-03 17:44:43 (0:02:00) ----------
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Files:   hashed 32/32 (131.9G/131.9G) | pre-uploaded: 26/26 (131.9G/131.9G) | committed: 0/32 (0.0/131.9G) | ignored: 0
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-q235b-kimid-v2-c-39855-v2-uploader: ---------------------------------------------------
2026-04-03T17:44:54.086412+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader:                              Processed model ChaiML/q235b_kimid_v2_cliche_plus_original-step444-merged in 1111.957s
chaiml-q235b-kimid-v2-c-39855-v2-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-kimid-v2-c-39855-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-kimid-v2-c-39855-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-kimid-v2-c-39855-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-kimid-v2-c-39855-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00025-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00025-of-00025.safetensors
2026-04-03T17:45:54.489911+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00004-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00004-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00020-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00020-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00017-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00017-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00024-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00024-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00009-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00009-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00001-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00001-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00014-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00014-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00018-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00018-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00003-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00003-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00010-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00010-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00008-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00008-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00022-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00022-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00021-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00021-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00012-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00012-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00013-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00013-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00005-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00005-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00011-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00011-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00015-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00015-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00007-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00007-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00002-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00002-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00023-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00023-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00019-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00019-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00006-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00006-of-00025.safetensors
chaiml-q235b-kimid-v2-c-39855-v2-uploader: cp /dev/shm/model_output/model-00016-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v2-c-39855-v2/default/model-00016-of-00025.safetensors
Job chaiml-q235b-kimid-v2-c-39855-v2-uploader completed after 1229.5s with status: succeeded
Stopping job with name chaiml-q235b-kimid-v2-c-39855-v2-uploader
Pipeline stage VLLMUploader completed in 1230.31s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.03s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-kimid-v2-c-39855-v2
Waiting for inference service chaiml-q235b-kimid-v2-c-39855-v2 to be ready
2026-04-03T17:46:54.655671+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
2026-04-03T17:47:54.806029+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-04-03T17:48:55.038760+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
Inference service chaiml-q235b-kimid-v2-c-39855-v2 ready after 201.8702690601349s
Pipeline stage VLLMDeployer completed in 202.55s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8084793090820312s
Received healthy response to inference request in 1.4898269176483154s
Received healthy response to inference request in 1.4988646507263184s
2026-04-03T17:49:55.507744+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
Received healthy response to inference request in 3.72126841545105s
Received healthy response to inference request in 1.4805889129638672s
Received healthy response to inference request in 1.4609572887420654s
Received healthy response to inference request in 3.4757161140441895s
Received healthy response to inference request in 3.668642282485962s
Received healthy response to inference request in 1.5914652347564697s
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 1.6035985946655273s
Received healthy response to inference request in 1.5392751693725586s
Received healthy response to inference request in 1.4716014862060547s
Received healthy response to inference request in 1.5646319389343262s
Received healthy response to inference request in 1.7782576084136963s
Received healthy response to inference request in 1.4865984916687012s
Received healthy response to inference request in 3.6517152786254883s
Received healthy response to inference request in 1.6659724712371826s
Received healthy response to inference request in 2.2979512214660645s
Received healthy response to inference request in 1.4930226802825928s
Received healthy response to inference request in 1.5505518913269043s
Received healthy response to inference request in 1.5680501461029053s
Received healthy response to inference request in 1.9635069370269775s
Received healthy response to inference request in 1.6487298011779785s
Received healthy response to inference request in 1.597238302230835s
Received healthy response to inference request in 1.6820919513702393s
Received healthy response to inference request in 1.5986781120300293s
Received healthy response to inference request in 1.471888780593872s
Received healthy response to inference request in 1.498666524887085s
Received healthy response to inference request in 1.9002978801727295s
2026-04-03T17:50:55.765826+00:00 monitor updated for chaiml-q235b-kimid-v2-c_39855_v2
Received healthy response to inference request in 1.62900972366333s
30 requests
0 failed requests
5th percentile: 1.4717307686805725
10th percentile: 1.4797188997268678
20th percentile: 1.4923835277557373
30th percentile: 1.5271520137786865
40th percentile: 1.5666828632354737
50th percentile: 1.5979582071304321
60th percentile: 1.6368977546691894
70th percentile: 1.710941648483276
80th percentile: 2.030395793914796
90th percentile: 3.6534079790115355
95th percentile: 3.6975866556167603
99th percentile: 3.7831881499290465
mean time: 1.961904803911845
Pipeline stage StressChecker completed in 73.13s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.35s
Shutdown handler de-registered
chaiml-q235b-kimid-v2-c_39855_v2 status is now deployed due to DeploymentManager action