chaiml-gspo-glm47-combi_35241

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_35241_v1
model_name: chaiml-gspo-glm47-combi_35241_v1
model_group: ChaiML/gspo-glm47-combin
status: inactive
timestamp: 2026-04-03T03:17:32+00:00
num_battles: 10213
num_wins: 5743
celo_rating: 1332.22
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm721-mega-data-step500
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_35241_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm721-mega-data-step500
model_size: 24B
ranking_group: single
us_pacific_date: 2026-04-02
win_ratio: 0.562322530108685
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '####', '</s>', '<|assistant|>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-35241-v1-uploader
Waiting for job on chaiml-gspo-glm47-combi-35241-v1-uploader to finish
chaiml-gspo-glm47-combi-35241-v1-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-35241-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm721-mega-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-35241-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm721-mega-data-step500...
2026-04-02T23:24:41.429344+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:25:41.522402+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:26:41.658483+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:27:41.747478+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: Downloaded in 244.397s
2026-04-02T23:28:42.000883+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:29:42.111691+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:30:42.197371+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:31:42.347734+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:32:42.429342+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: Applying quantization...
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:33:24 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
2026-04-02T23:33:42.510195+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:33:57 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:12 INFO device.py L1468:  'peak_ram': 11.91GB, 'peak_vram': 1.44GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:18 INFO device.py L1468:  'peak_ram': 16.38GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:29 INFO device.py L1468:  'peak_ram': 16.94GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:34 INFO device.py L1468:  'peak_ram': 19.36GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:34:42.600814+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:44 INFO device.py L1468:  'peak_ram': 19.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:50 INFO device.py L1468:  'peak_ram': 22.17GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:34:55 INFO device.py L1468:  'peak_ram': 22.17GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:04 INFO device.py L1468:  'peak_ram': 22.71GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:09 INFO device.py L1468:  'peak_ram': 22.71GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:17 INFO device.py L1468:  'peak_ram': 23.36GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:23 INFO device.py L1468:  'peak_ram': 23.36GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:28 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:37 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:35:42.701900+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:43 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:48 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:35:57 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:02 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:11 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:17 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:26 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:39 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:36:42.783813+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:36:53 INFO device.py L1468:  'peak_ram': 23.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:37:08 INFO device.py L1468:  'peak_ram': 23.95GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:37:24 INFO device.py L1468:  'peak_ram': 23.95GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:37:36 INFO device.py L1468:  'peak_ram': 24.46GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:37:42.874600+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:37:52 INFO device.py L1468:  'peak_ram': 24.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:38:02 INFO device.py L1468:  'peak_ram': 25.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:38:13 INFO device.py L1468:  'peak_ram': 25.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:38:27 INFO device.py L1468:  'peak_ram': 25.94GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:38:38 INFO device.py L1468:  'peak_ram': 25.94GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:38:42.960660+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:38:52 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:39:03 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:39:14 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:39:29 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:39:43.089310+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:39:40 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:39:51 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:40:05 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:40:16 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:40:30 INFO device.py L1468:  'peak_ram': 26.9GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:40:43.203628+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:40:42 INFO device.py L1468:  'peak_ram': 27.61GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:40:53 INFO device.py L1468:  'peak_ram': 27.61GB, 'peak_vram': 1.59GB[0m
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:41:07 INFO device.py L1468:  'peak_ram': 28.66GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:41:19 INFO device.py L1468:  'peak_ram': 28.66GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:41:33 INFO device.py L1468:  'peak_ram': 29.47GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:41:43.353585+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:41:58 INFO device.py L1468:  'peak_ram': 29.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:42:12 INFO device.py L1468:  'peak_ram': 29.9GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:42:25 INFO device.py L1468:  'peak_ram': 31.06GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:42:36 INFO device.py L1468:  'peak_ram': 31.06GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:42:43.443118+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:42:50 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:43:01 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:43:15 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:43:26 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:43:37 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:43:43.536904+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:43:51 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:44:03 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:44:17 INFO device.py L1468:  'peak_ram': 33.08GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:44:29 INFO device.py L1468:  'peak_ram': 33.08GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:44:43.633051+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:44:41 INFO device.py L1468:  'peak_ram': 33.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:44:56 INFO device.py L1468:  'peak_ram': 33.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:45:08 INFO device.py L1468:  'peak_ram': 34.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:45:21 INFO device.py L1468:  'peak_ram': 34.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:45:34 INFO device.py L1468:  'peak_ram': 35.25GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:45:43.735791+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:45:44 INFO device.py L1468:  'peak_ram': 35.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:45:59 INFO device.py L1468:  'peak_ram': 36.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:46:19 INFO device.py L1468:  'peak_ram': 36.91GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:46:33 INFO device.py L1468:  'peak_ram': 36.91GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:46:43.869386+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:46:45 INFO device.py L1468:  'peak_ram': 37.72GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:47:00 INFO device.py L1468:  'peak_ram': 37.72GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:47:15 INFO device.py L1468:  'peak_ram': 37.72GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:47:27 INFO device.py L1468:  'peak_ram': 37.72GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:47:43.965120+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:47:45 INFO device.py L1468:  'peak_ram': 37.72GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:47:59 INFO device.py L1468:  'peak_ram': 37.8GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:48:12 INFO device.py L1468:  'peak_ram': 37.8GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:48:28 INFO device.py L1468:  'peak_ram': 39.07GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:48:44.182712+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:48:44 INFO device.py L1468:  'peak_ram': 39.07GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:06 INFO device.py L1468:  'peak_ram': 40.12GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:23 INFO device.py L1468:  'peak_ram': 40.12GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:34 INFO device.py L1468:  'peak_ram': 40.8GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:49:44.275769+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:46 INFO device.py L1468:  'peak_ram': 40.8GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:52 INFO device.py L1468:  'peak_ram': 41.84GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:49:58 INFO device.py L1468:  'peak_ram': 41.84GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:07 INFO device.py L1468:  'peak_ram': 42.88GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:13 INFO device.py L1468:  'peak_ram': 42.88GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:23 INFO device.py L1468:  'peak_ram': 44.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:41 INFO device.py L1468:  'peak_ram': 45.05GB, 'peak_vram': 1.59GB[0m
2026-04-02T23:50:44.586940+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:53 INFO device.py L1468:  'peak_ram': 45.05GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:50:58 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:51:05 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:51:10 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:51:20 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [33;1m2026-04-02 23:51:20 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: [38;20m2026-04-02 23:51:20 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-35241-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm721-mega-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-35241-v1-uploader: Creating repo ChaiML/gspo-glm47-combine-rm721-mega-data-step500-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------- 2026-04-02 23:51:21 (0:00:00) ----------
chaiml-gspo-glm47-combi-35241-v1-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+43 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-35241-v1-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------------------------------------------------
2026-04-02T23:51:44.747526+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------- 2026-04-02 23:52:21 (0:01:00) ----------
chaiml-gspo-glm47-combi-35241-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 11/40 (41.8G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-35241-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 29 | committing: 0 | waiting: 35
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------------------------------------------------
2026-04-02T23:52:44.853769+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------- 2026-04-02 23:53:21 (0:02:00) ----------
chaiml-gspo-glm47-combi-35241-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 39/40 (192.0G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-35241-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 63
chaiml-gspo-glm47-combi-35241-v1-uploader: ---------------------------------------------------
2026-04-02T23:53:45.349712+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm721-mega-data-step500 in 1780.076s
chaiml-gspo-glm47-combi-35241-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/config.json
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/quantization_config.json
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/tokenizer_config.json
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/tokenizer.json
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00038-of-00038.safetensors
2026-04-02T23:54:45.444515+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-35241-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-35241-v1/default/model-00024-of-00038.safetensors
Job chaiml-gspo-glm47-combi-35241-v1-uploader completed after 1878.33s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-35241-v1-uploader
Pipeline stage VLLMUploader completed in 1878.91s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.53s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.05s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-35241-v1
Waiting for inference service chaiml-gspo-glm47-combi-35241-v1 to be ready
2026-04-02T23:55:45.586423+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:56:45.736461+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:57:45.849873+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
2026-04-02T23:58:45.961143+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
Inference service chaiml-gspo-glm47-combi-35241-v1 ready after 271.54884028434753s
Pipeline stage VLLMDeployer completed in 272.16s
run pipeline stage %s
Running pipeline stage StressChecker
2026-04-02T23:59:46.072562+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
Received healthy response to inference request in 8.69994068145752s
Received healthy response to inference request in 8.817215204238892s
Received healthy response to inference request in 8.625854253768921s
Received healthy response to inference request in 2.1376471519470215s
Received healthy response to inference request in 2.3319857120513916s
Received healthy response to inference request in 9.006953001022339s
Received healthy response to inference request in 2.2368524074554443s
Received healthy response to inference request in 2.011502981185913s
Received healthy response to inference request in 8.646937608718872s
Received healthy response to inference request in 2.473752498626709s
Received healthy response to inference request in 2.037008762359619s
Received healthy response to inference request in 2.9205493927001953s
Received healthy response to inference request in 2.0873000621795654s
Received healthy response to inference request in 2.6405153274536133s
Received healthy response to inference request in 2.0985827445983887s
2026-04-03T00:00:46.168343+00:00 monitor updated for chaiml-gspo-glm47-combi_35241_v1
Received healthy response to inference request in 2.125955820083618s
Received healthy response to inference request in 2.2332122325897217s
Received healthy response to inference request in 2.0230538845062256s
Received healthy response to inference request in 2.2372395992279053s
Received healthy response to inference request in 2.096998453140259s
Received healthy response to inference request in 3.064730167388916s
Received healthy response to inference request in 2.049816370010376s
Received healthy response to inference request in 2.5699005126953125s
Received healthy response to inference request in 2.305415391921997s
Received healthy response to inference request in 2.02589750289917s
Received healthy response to inference request in 2.0117554664611816s
Received healthy response to inference request in 2.0840904712677s
Received healthy response to inference request in 2.061645269393921s
Received healthy response to inference request in 2.249332904815674s
Received healthy response to inference request in 2.1909942626953125s
30 requests
0 failed requests
5th percentile: 2.0168397545814516
10th percentile: 2.0256131410598757
20th percentile: 2.0592794895172117
30th percentile: 2.0940889358520507
40th percentile: 2.13297061920166
50th percentile: 2.235032320022583
60th percentile: 2.271765899658203
70th percentile: 2.50259690284729
80th percentile: 2.94938554763794
90th percentile: 8.652237915992737
95th percentile: 8.764441668987274
99th percentile: 8.95192903995514
mean time: 3.3367545366287232
Pipeline stage StressChecker completed in 104.13s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.92s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_35241_v1 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_35241_v1 status is now inactive due to auto deactivation removed underperforming models