chaiml-gspo-glm47-combi_30990

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_30990_v1
model_name: chaiml-gspo-glm47-combi_30990_v1
model_group: ChaiML/gspo-glm47-combin
status: torndown
timestamp: 2026-04-02T21:21:45+00:00
num_battles: 11971
num_wins: 6671
celo_rating: 1337.79
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm82-mega-data-step600
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_30990_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm82-mega-data-step600
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-30
win_ratio: 0.5572633865174171
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '####', '<|im_end|>', '<|user|>', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-30990-v1-uploader
Waiting for job on chaiml-gspo-glm47-combi-30990-v1-uploader to finish
chaiml-gspo-glm47-combi-30990-v1-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-30990-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-30990-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-mega-data-step600...
2026-03-30T19:11:30.691904+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:12:30.779765+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:13:30.868901+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:14:31.246008+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:15:31.507441+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: Downloaded in 265.802s
2026-03-30T19:16:32.195749+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:17:32.446502+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:18:33.257994+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:19:33.513620+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: Applying quantization...
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:20:25 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
2026-03-30T19:20:33.776901+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:21:02 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:21:22 INFO device.py L1468:  'peak_ram': 11.87GB, 'peak_vram': 1.44GB[0m
2026-03-30T19:21:33.878991+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:21:35 INFO device.py L1468:  'peak_ram': 16.3GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:21:54 INFO device.py L1468:  'peak_ram': 16.91GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:22:08 INFO device.py L1468:  'peak_ram': 19.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:22:26 INFO device.py L1468:  'peak_ram': 19.95GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:22:34.161287+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:22:39 INFO device.py L1468:  'peak_ram': 22.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:22:52 INFO device.py L1468:  'peak_ram': 22.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:23:09 INFO device.py L1468:  'peak_ram': 22.64GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:23:22 INFO device.py L1468:  'peak_ram': 22.64GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:23:34.469176+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:23:38 INFO device.py L1468:  'peak_ram': 23.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:23:49 INFO device.py L1468:  'peak_ram': 23.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:24:00 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:24:14 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:24:25 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:24:34.562187+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:24:35 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:24:49 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:25:00 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:25:14 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:25:25 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:25:34.922803+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:25:37 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:25:51 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:26:03 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:26:19 INFO device.py L1468:  'peak_ram': 23.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:26:29 INFO device.py L1468:  'peak_ram': 23.89GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:26:35.707816+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:26:40 INFO device.py L1468:  'peak_ram': 24.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:26:55 INFO device.py L1468:  'peak_ram': 24.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:27:08 INFO device.py L1468:  'peak_ram': 25.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:27:24 INFO device.py L1468:  'peak_ram': 25.23GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:27:35.795980+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:27:41 INFO device.py L1468:  'peak_ram': 26.0GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:27:55 INFO device.py L1468:  'peak_ram': 26.0GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:28:11 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:28:25 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:28:35.891524+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:28:38 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:28:52 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:29:05 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:29:21 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:29:36.033519+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:29:39 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:29:53 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:30:11 INFO device.py L1468:  'peak_ram': 26.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:30:22 INFO device.py L1468:  'peak_ram': 27.74GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:30:36.133103+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:30:35 INFO device.py L1468:  'peak_ram': 27.74GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:30:54 INFO device.py L1468:  'peak_ram': 28.84GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:31:07 INFO device.py L1468:  'peak_ram': 28.84GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:31:25 INFO device.py L1468:  'peak_ram': 29.61GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:31:36.424873+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:31:37 INFO device.py L1468:  'peak_ram': 29.61GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:31:49 INFO device.py L1468:  'peak_ram': 29.98GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:32:04 INFO device.py L1468:  'peak_ram': 29.98GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:32:18 INFO device.py L1468:  'peak_ram': 31.15GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:32:29 INFO device.py L1468:  'peak_ram': 31.15GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:32:36.679043+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:32:43 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:32:54 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:33:07 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:33:16 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:33:26 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:33:36.951265+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:33:40 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:33:50 INFO device.py L1468:  'peak_ram': 31.97GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:34:04 INFO device.py L1468:  'peak_ram': 33.03GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:34:15 INFO device.py L1468:  'peak_ram': 33.03GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:34:26 INFO device.py L1468:  'peak_ram': 33.4GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:34:37.217679+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:34:41 INFO device.py L1468:  'peak_ram': 33.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:34:52 INFO device.py L1468:  'peak_ram': 34.08GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:35:03 INFO device.py L1468:  'peak_ram': 34.08GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:35:17 INFO device.py L1468:  'peak_ram': 35.24GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:35:29 INFO device.py L1468:  'peak_ram': 35.24GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:35:37.492572+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:35:44 INFO device.py L1468:  'peak_ram': 36.3GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:35:58 INFO device.py L1468:  'peak_ram': 36.3GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:36:11 INFO device.py L1468:  'peak_ram': 36.95GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:36:27 INFO device.py L1468:  'peak_ram': 36.95GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:36:37.598174+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:36:44 INFO device.py L1468:  'peak_ram': 37.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:37:03 INFO device.py L1468:  'peak_ram': 37.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:37:16 INFO device.py L1468:  'peak_ram': 37.76GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:37:37.697775+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:37:30 INFO device.py L1468:  'peak_ram': 37.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:37:47 INFO device.py L1468:  'peak_ram': 37.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:38:03 INFO device.py L1468:  'peak_ram': 37.94GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:38:18 INFO device.py L1468:  'peak_ram': 37.94GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:38:37.797399+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:38:36 INFO device.py L1468:  'peak_ram': 38.69GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:38:49 INFO device.py L1468:  'peak_ram': 38.69GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:39:05 INFO device.py L1468:  'peak_ram': 39.71GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:39:20 INFO device.py L1468:  'peak_ram': 39.71GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:39:38.092124+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:39:35 INFO device.py L1468:  'peak_ram': 40.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:39:50 INFO device.py L1468:  'peak_ram': 40.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:40:03 INFO device.py L1468:  'peak_ram': 41.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:40:18 INFO device.py L1468:  'peak_ram': 41.35GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:40:38.346362+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:40:34 INFO device.py L1468:  'peak_ram': 42.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:40:47 INFO device.py L1468:  'peak_ram': 42.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:41:02 INFO device.py L1468:  'peak_ram': 43.41GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:41:17 INFO device.py L1468:  'peak_ram': 43.41GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:41:29 INFO device.py L1468:  'peak_ram': 44.25GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:41:38.634874+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:41:46 INFO device.py L1468:  'peak_ram': 44.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:41:59 INFO device.py L1468:  'peak_ram': 44.54GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:42:26 INFO device.py L1468:  'peak_ram': 44.54GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:42:34 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
2026-03-30T19:42:38.923290+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: [38;20m2026-03-30 19:42:35 INFO device.py L1468:  'peak_ram': 44.54GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-30990-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-30990-v1-uploader: Creating repo ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------- 2026-03-30 19:42:36 (0:00:00) ----------
chaiml-gspo-glm47-combi-30990-v1-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+44 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-30990-v1-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------------------------------------------------
chaiml-gspo-glm47-combi-30990-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------- 2026-03-30 19:43:36 (0:01:00) ----------
chaiml-gspo-glm47-combi-30990-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 20/37 (90.1G/197.4G) (+3 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-30990-v1-uploader: Workers: hashing: 0 | get upload mode: 3 | pre-uploading: 17 | committing: 0 | waiting: 44
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------------------------------------------------
2026-03-30T19:43:39.023437+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------- 2026-03-30 19:44:36 (0:02:00) ----------
chaiml-gspo-glm47-combi-30990-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 37/37 (181.3G/197.4G) (+3 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-30990-v1-uploader: Workers: hashing: 0 | get upload mode: 3 | pre-uploading: 0 | committing: 0 | waiting: 61
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------------------------------------------------
2026-03-30T19:44:39.122097+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader: HTTP Error 504 thrown while requesting POST https://huggingface.co/api/models/ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16/preupload/main
chaiml-gspo-glm47-combi-30990-v1-uploader: Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-combi-30990-v1-uploader: HTTP Error 504 thrown while requesting POST https://huggingface.co/api/models/ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16/preupload/main
chaiml-gspo-glm47-combi-30990-v1-uploader: Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-combi-30990-v1-uploader: HTTP Error 504 thrown while requesting POST https://huggingface.co/api/models/ChaiML/gspo-glm47-combine-rm82-mega-data-step600-W4A16/preupload/main
chaiml-gspo-glm47-combi-30990-v1-uploader: Retrying in 1s [Retry 1/5].
2026-03-30T19:45:39.222337+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------- 2026-03-30 19:45:36 (0:03:00) ----------
chaiml-gspo-glm47-combi-30990-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-30990-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-combi-30990-v1-uploader: ---------------------------------------------------
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
2026-03-30T19:46:39.330223+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
chaiml-gspo-glm47-combi-30990-v1-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm82-mega-data-step600 in 2125.422s
chaiml-gspo-glm47-combi-30990-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-30990-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-30990-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-30990-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-30990-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-30990-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-30990-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-30990-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-30990-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-30990-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-30990-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-30990-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/quantization_config.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/tokenizer_config.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/config.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/generation_config.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/chat_template.jinja
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/tokenizer.json
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-30990-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-30990-v1/default/model-00011-of-00038.safetensors
2026-03-30T19:47:39.448728+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
Job chaiml-gspo-glm47-combi-30990-v1-uploader completed after 2229.82s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-30990-v1-uploader
Pipeline stage VLLMUploader completed in 2230.45s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.11s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-30990-v1
Waiting for inference service chaiml-gspo-glm47-combi-30990-v1 to be ready
2026-03-30T19:48:39.940790+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-30T19:49:40.302520+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:50:40.600904+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:51:41.318991+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:52:41.458513+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:53:41.563569+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:54:41.690627+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:55:42.020228+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:56:42.293013+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:57:42.582353+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:58:42.878204+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T19:59:43.201366+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:00:43.498294+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:01:43.795298+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:02:44.307942+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:03:44.497356+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:04:44.968275+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
2026-03-30T20:05:45.834279+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
Inference service chaiml-gspo-glm47-combi-30990-v1 ready after 1103.2737510204315s
Pipeline stage VLLMDeployer completed in 1104.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.942630529403687s
Received healthy response to inference request in 9.10457730293274s
Received healthy response to inference request in 2.6220288276672363s
Received healthy response to inference request in 2.679842233657837s
Received healthy response to inference request in 2.3182125091552734s
Received healthy response to inference request in 2.285727024078369s
Received healthy response to inference request in 2.04966139793396s
Received healthy response to inference request in 2.4324045181274414s
2026-03-30T20:06:46.574053+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
Received healthy response to inference request in 2.0102415084838867s
Received healthy response to inference request in 2.496138095855713s
Received healthy response to inference request in 2.3059277534484863s
Received healthy response to inference request in 2.1570308208465576s
Received healthy response to inference request in 2.046889543533325s
Received healthy response to inference request in 2.149726152420044s
Received healthy response to inference request in 2.091938018798828s
Received healthy response to inference request in 2.289966583251953s
Received healthy response to inference request in 2.7352607250213623s
Received healthy response to inference request in 2.53262996673584s
Received healthy response to inference request in 2.172072172164917s
Received healthy response to inference request in 2.2231974601745605s
Received healthy response to inference request in 2.063326120376587s
Received healthy response to inference request in 2.1128625869750977s
Received healthy response to inference request in 2.5039219856262207s
Received healthy response to inference request in 2.256605386734009s
Received healthy response to inference request in 2.0910942554473877s
Received healthy response to inference request in 2.4811973571777344s
Received healthy response to inference request in 2.624896287918091s
Received healthy response to inference request in 2.2217490673065186s
Received healthy response to inference request in 2.4028773307800293s
Received healthy response to inference request in 2.106708526611328s
30 requests
0 failed requests
5th percentile: 2.0481368780136107
10th percentile: 2.061959648132324
20th percentile: 2.103754425048828
30th percentile: 2.1548394203186034
40th percentile: 2.2226181030273438
50th percentile: 2.287846803665161
60th percentile: 2.352078437805176
70th percentile: 2.485679578781128
80th percentile: 2.5505097389221194
90th percentile: 2.6853840827941897
95th percentile: 6.149314117431623
99th percentile: 9.057612738609315
mean time: 2.750378068288167
Pipeline stage StressChecker completed in 96.16s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
2026-03-30T20:07:46.700902+00:00 monitor updated for chaiml-gspo-glm47-combi_30990_v1
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.54s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_30990_v1 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_30990_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-gspo-glm47-combi_30990_v1 status is now torndown due to DeploymentManager action