chaiml-gspo-glm47-chai-_27711

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-chai-_27711_v1
model_name: chaiml-gspo-glm47-chai-_27711_v1
model_group: ChaiML/gspo-glm47-chai-r
status: torndown
timestamp: 2026-04-02T21:21:47+00:00
num_battles: 11422
num_wins: 6485
celo_rating: 1370.86
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-chai-rm-mega-data-step500
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-chai-_27711_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-chai-rm-mega-data-step500
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-30
win_ratio: 0.5677639642794607
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '####', '<|assistant|>', '<|im_end|>', '<|user|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-chai-27711-v1-uploader
Waiting for job on chaiml-gspo-glm47-chai-27711-v1-uploader to finish
chaiml-gspo-glm47-chai-27711-v1-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-chai-27711-v1-uploader: Checking if ChaiML/gspo-glm47-chai-rm-mega-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-chai-27711-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-mega-data-step500...
2026-03-30T19:10:37.560143+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:11:37.641535+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
2026-03-30T19:12:37.729633+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:13:37.809902+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:14:38.921820+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: Downloaded in 272.522s
2026-03-30T19:15:39.191830+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Retrying (%r) after connection broken by '%r': %s
2026-03-30T19:16:39.534563+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:17:39.834854+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:18:40.142529+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:19:40.752164+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: Applying quantization...
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:19:55 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:20:29 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
2026-03-30T19:20:40.892369+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:20:59 INFO device.py L1468:  'peak_ram': 16.13GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:10 INFO device.py L1468:  'peak_ram': 16.73GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:16 INFO device.py L1468:  'peak_ram': 19.15GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:26 INFO device.py L1468:  'peak_ram': 19.63GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:32 INFO device.py L1468:  'peak_ram': 22.05GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:38 INFO device.py L1468:  'peak_ram': 22.05GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:21:41.049014+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:46 INFO device.py L1468:  'peak_ram': 22.42GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:51 INFO device.py L1468:  'peak_ram': 22.42GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:21:58 INFO device.py L1468:  'peak_ram': 23.05GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:02 INFO device.py L1468:  'peak_ram': 23.05GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:07 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:14 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:23 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:22:41.621025+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:35 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:22:51 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:23:06 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:23:22 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:23:41.709121+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:23:37 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:23:52 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:24:10 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:24:26 INFO device.py L1468:  'peak_ram': 23.25GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:24:42.141163+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:24:43 INFO device.py L1468:  'peak_ram': 23.99GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:24:56 INFO device.py L1468:  'peak_ram': 23.99GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:25:09 INFO device.py L1468:  'peak_ram': 24.44GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:25:23 INFO device.py L1468:  'peak_ram': 24.44GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:25:34 INFO device.py L1468:  'peak_ram': 25.34GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:25:42.421240+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:25:46 INFO device.py L1468:  'peak_ram': 25.34GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:26:01 INFO device.py L1468:  'peak_ram': 26.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:26:13 INFO device.py L1468:  'peak_ram': 26.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:26:28 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:26:40 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:26:42.691317+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:26:53 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:27:08 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:27:20 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:27:32 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:27:43.188467+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:27:48 INFO device.py L1468:  'peak_ram': 27.14GB, 'peak_vram': 1.59GB[0m
Failed to get request counts for guanaco-submitter. Falling back to default
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:27:59 INFO device.py L1468:  'peak_ram': 27.52GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:28:15 INFO device.py L1468:  'peak_ram': 27.52GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:28:27 INFO device.py L1468:  'peak_ram': 28.28GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:28:43.311742+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:28:40 INFO device.py L1468:  'peak_ram': 28.28GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:28:54 INFO device.py L1468:  'peak_ram': 29.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:29:06 INFO device.py L1468:  'peak_ram': 29.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:29:21 INFO device.py L1468:  'peak_ram': 30.15GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:29:33 INFO device.py L1468:  'peak_ram': 30.15GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:29:43.629240+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:29:44 INFO device.py L1468:  'peak_ram': 30.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:29:58 INFO device.py L1468:  'peak_ram': 30.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:30:09 INFO device.py L1468:  'peak_ram': 31.65GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:30:20 INFO device.py L1468:  'peak_ram': 31.65GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:30:36 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:30:43.895635+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:30:53 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:31:12 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:31:30 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:31:44.179319+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:31:47 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:32:05 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:32:21 INFO device.py L1468:  'peak_ram': 32.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:32:38 INFO device.py L1468:  'peak_ram': 33.4GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:32:44.429732+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:32:52 INFO device.py L1468:  'peak_ram': 33.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:33:09 INFO device.py L1468:  'peak_ram': 33.83GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:33:26 INFO device.py L1468:  'peak_ram': 33.83GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:33:40 INFO device.py L1468:  'peak_ram': 34.53GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:33:44.710740+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:33:52 INFO device.py L1468:  'peak_ram': 34.53GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:34:07 INFO device.py L1468:  'peak_ram': 35.5GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:34:20 INFO device.py L1468:  'peak_ram': 35.5GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:34:45.663514+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:34:37 INFO device.py L1468:  'peak_ram': 36.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:34:52 INFO device.py L1468:  'peak_ram': 36.76GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:05 INFO device.py L1468:  'peak_ram': 37.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:19 INFO device.py L1468:  'peak_ram': 37.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:26 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:33 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:35:45.941696+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:38 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:45 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:35:59 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:36:11 INFO device.py L1468:  'peak_ram': 38.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:36:26 INFO device.py L1468:  'peak_ram': 38.4GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:36:46.059152+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:36:39 INFO device.py L1468:  'peak_ram': 39.58GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:36:48 INFO device.py L1468:  'peak_ram': 39.58GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:37:03 INFO device.py L1468:  'peak_ram': 40.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:37:16 INFO device.py L1468:  'peak_ram': 40.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:37:28 INFO device.py L1468:  'peak_ram': 41.18GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:37:46.165836+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:37:47 INFO device.py L1468:  'peak_ram': 41.18GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:38:02 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:38:17 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:38:34 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:38:46.313919+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:38:46 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:39:05 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:39:20 INFO device.py L1468:  'peak_ram': 42.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:39:35 INFO device.py L1468:  'peak_ram': 42.99GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:39:46.765578+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:39:53 INFO device.py L1468:  'peak_ram': 42.99GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:40:07 INFO device.py L1468:  'peak_ram': 43.27GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:40:24 INFO device.py L1468:  'peak_ram': 43.27GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:40:38 INFO device.py L1468:  'peak_ram': 43.27GB, 'peak_vram': 1.59GB[0m
2026-03-30T19:40:47.063284+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:40:46 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [33;1m2026-03-30 19:40:46 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: [38;20m2026-03-30 19:40:47 INFO device.py L1468:  'peak_ram': 43.27GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-chai-27711-v1-uploader: Checking if ChaiML/gspo-glm47-chai-rm-mega-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-chai-27711-v1-uploader: Creating repo ChaiML/gspo-glm47-chai-rm-mega-data-step500-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:40:47 (0:00:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+45 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 38 | get upload mode: 3 | pre-uploading: 0 | committing: 0 | waiting: 23
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:41:47.411658+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-chai-27711-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:41:48 (0:01:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 10/40 (41.8G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 30 | committing: 0 | waiting: 34
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:42:47.724593+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:42:48 (0:02:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:43:47.838720+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:43:48 (0:03:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:44:47.946126+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:44:48 (0:04:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
chaiml-gspo-glm47-chai-27711-v1-uploader: Failed to commit: Server error '504 Gateway Time-out' for url 'https://huggingface.co/api/models/ChaiML/gspo-glm47-chai-rm-mega-data-step500-W4A16/commit/main' (Amz CF ID: _gCxUOWAALNKi7btdq2f3CenMSuoN_2rCiSn2n3goJOsUGE8wziacA==)
chaiml-gspo-glm47-chai-27711-v1-uploader: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
chaiml-gspo-glm47-chai-27711-v1-uploader: Failed to commit 45 files at once. Will retry with less files in next batch.
2026-03-30T19:45:48.056904+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:45:48 (0:05:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T19:46:48.215481+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:46:48 (0:06:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
chaiml-gspo-glm47-chai-27711-v1-uploader: Failed to commit: Server error '504 Gateway Time-out' for url 'https://huggingface.co/api/models/ChaiML/gspo-glm47-chai-rm-mega-data-step500-W4A16/commit/main' (Amz CF ID: GXdlPpfTUbjvc9CfQr12_byy4_6Ja99ohWBcfydhCMExA-siTQSeDA==)
chaiml-gspo-glm47-chai-27711-v1-uploader: For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/504
chaiml-gspo-glm47-chai-27711-v1-uploader: Failed to commit 20 files at once. Will retry with less files in next batch.
2026-03-30T19:47:49.052206+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:47:48 (0:07:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:48:49.967441+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------- 2026-03-30 19:49:48 (0:09:00) ----------
chaiml-gspo-glm47-chai-27711-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 40/45 (170.6G/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-27711-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-chai-27711-v1-uploader: ---------------------------------------------------
2026-03-30T19:49:50.912357+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader:                              Processed model ChaiML/gspo-glm47-chai-rm-mega-data-step500 in 2415.138s
chaiml-gspo-glm47-chai-27711-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-27711-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-27711-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-27711-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-27711-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-27711-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-27711-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-chai-27711-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-27711-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-27711-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-27711-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-27711-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/config.json
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/tokenizer_config.json
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/generation_config.json
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/chat_template.jinja
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/quantization_config.json
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/tokenizer.json
2026-03-30T19:50:51.820098+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-27711-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-27711-v1/default/model-00003-of-00038.safetensors
Job chaiml-gspo-glm47-chai-27711-v1-uploader completed after 2522.17s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-27711-v1-uploader
Pipeline stage VLLMUploader completed in 2523.62s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 5.50s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-27711-v1
Waiting for inference service chaiml-gspo-glm47-chai-27711-v1 to be ready
2026-03-30T19:51:52.134714+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:52:52.311917+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:53:52.512975+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Failed to get response for submission chaiml-qwen-bobo-dpo-ju_56781_v7: ('http://chaiml-qwen-bobo-dpo-ju-56781-v7-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T19:54:52.721709+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T19:55:53.692664+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T19:56:55.076046+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:57:55.440252+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T19:58:56.017730+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T19:59:56.304160+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:00:56.910170+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:01:57.020888+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:02:57.130081+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:03:57.455320+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:04:58.300523+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-30T20:05:59.390267+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
2026-03-30T20:06:59.722739+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Inference service chaiml-gspo-glm47-chai-27711-v1 ready after 933.6664395332336s
Pipeline stage VLLMDeployer completed in 935.03s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.810734033584595s
Received healthy response to inference request in 2.0295493602752686s
Received healthy response to inference request in 8.977653741836548s
Received healthy response to inference request in 2.1343865394592285s
Received healthy response to inference request in 2.064331531524658s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 2.297107219696045s
Received healthy response to inference request in 2.133965015411377s
Received healthy response to inference request in 2.077721357345581s
Received healthy response to inference request in 2.014328956604004s
2026-03-30T20:08:00.326210+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
Received healthy response to inference request in 3.022456645965576s
Received healthy response to inference request in 3.3509693145751953s
Received healthy response to inference request in 2.1063060760498047s
Received healthy response to inference request in 2.048363447189331s
Received healthy response to inference request in 2.5084967613220215s
Received healthy response to inference request in 2.0385079383850098s
Received healthy response to inference request in 2.1458239555358887s
Received healthy response to inference request in 2.4906251430511475s
Received healthy response to inference request in 2.220912456512451s
Received healthy response to inference request in 2.298574209213257s
Received healthy response to inference request in 2.6883347034454346s
Received healthy response to inference request in 2.3952739238739014s
Received healthy response to inference request in 2.40138840675354s
Received healthy response to inference request in 2.065394639968872s
Received healthy response to inference request in 3.0824592113494873s
Received healthy response to inference request in 2.13242506980896s
Received healthy response to inference request in 2.0507590770721436s
Received healthy response to inference request in 3.5618629455566406s
Received healthy response to inference request in 2.106583833694458s
Received healthy response to inference request in 2.6530089378356934s
Received healthy response to inference request in 2.1164391040802s
2026-03-30T20:09:00.774719+00:00 monitor updated for chaiml-gspo-glm47-chai-_27711_v1
30 requests
0 failed requests
5th percentile: 2.033580720424652
10th percentile: 2.047377896308899
20th percentile: 2.065182018280029
30th percentile: 2.106500506401062
40th percentile: 2.1333490371704102
50th percentile: 2.18336820602417
60th percentile: 2.3372540950775145
70th percentile: 2.4959866285324095
80th percentile: 2.755159091949464
90th percentile: 3.37205867767334
95th percentile: 6.4487420439720005
99th percentile: 8.929247026443482
mean time: 2.800824785232544
Pipeline stage StressChecker completed in 99.80s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.85s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_27711_v1 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-chai-_27711_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-gspo-glm47-chai-_27711_v1 status is now torndown due to DeploymentManager action