chaiml-gspo-glm47-combi_24742

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_24742_v1
model_name: chaiml-gspo-glm47-combi_24742_v1
model_group: ChaiML/gspo-glm47-combin
status: torndown
timestamp: 2026-04-02T22:41:30+00:00
num_battles: 11539
num_wins: 6730
celo_rating: 1368.64
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm82-mega-data-step1100
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_24742_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm82-mega-data-step1100
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-30
win_ratio: 0.5832394488257214
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '</s>', '####', '<|user|>', '<|assistant|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-24742-v1-uploader
Waiting for job on chaiml-gspo-glm47-combi-24742-v1-uploader to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-combi-24742-v1-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-24742-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-24742-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-mega-data-step1100...
2026-03-30T20:22:53.160729+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:23:53.908362+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Retrying (%r) after connection broken by '%r': %s
Failed to get response for submission chaiml-gspo-glm47-chai-_27711_v1: HTTPConnectionPool(host='chaiml-gspo-glm47-chai-27711-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
2026-03-30T20:24:53.998396+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T20:25:54.687686+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:26:55.224270+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: Downloaded in 259.657s
2026-03-30T20:27:55.422081+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:28:55.590314+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:29:55.763358+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:30:56.976024+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-combi-24742-v1-uploader: Applying quantization...
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:31:40 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T20:31:57.108414+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:32:13 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:32:32 INFO device.py L1468:  'peak_ram': 11.8GB, 'peak_vram': 1.44GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:32:44 INFO device.py L1468:  'peak_ram': 16.28GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:32:57.408163+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:02 INFO device.py L1468:  'peak_ram': 16.82GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:16 INFO device.py L1468:  'peak_ram': 19.23GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:29 INFO device.py L1468:  'peak_ram': 19.67GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:36 INFO device.py L1468:  'peak_ram': 22.09GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:42 INFO device.py L1468:  'peak_ram': 22.09GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:33:57.502527+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:50 INFO device.py L1468:  'peak_ram': 22.67GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:33:56 INFO device.py L1468:  'peak_ram': 22.67GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:34:04 INFO device.py L1468:  'peak_ram': 23.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:34:09 INFO device.py L1468:  'peak_ram': 23.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:34:18 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:34:31 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:34:57.611983+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:34:53 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:07 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:13 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:20 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:24 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:28 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:34 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:39 INFO device.py L1468:  'peak_ram': 23.55GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:45 INFO device.py L1468:  'peak_ram': 24.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:49 INFO device.py L1468:  'peak_ram': 24.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:35:53 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:35:57.706458+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:36:00 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:36:04 INFO device.py L1468:  'peak_ram': 25.75GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:36:16 INFO device.py L1468:  'peak_ram': 25.75GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:36:33 INFO device.py L1468:  'peak_ram': 26.6GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:36:47 INFO device.py L1468:  'peak_ram': 26.6GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:36:57.798053+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:37:04 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:37:30 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:37:47 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:37:58.103874+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:38:03 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:38:17 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:38:33 INFO device.py L1468:  'peak_ram': 27.63GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:38:47 INFO device.py L1468:  'peak_ram': 27.85GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:38:58.894984+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:39:02 INFO device.py L1468:  'peak_ram': 27.85GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:39:18 INFO device.py L1468:  'peak_ram': 28.79GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:39:32 INFO device.py L1468:  'peak_ram': 28.79GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:39:49 INFO device.py L1468:  'peak_ram': 29.86GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:39:59.813829+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:40:03 INFO device.py L1468:  'peak_ram': 29.86GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:40:20 INFO device.py L1468:  'peak_ram': 30.74GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:40:33 INFO device.py L1468:  'peak_ram': 30.74GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:40:49 INFO device.py L1468:  'peak_ram': 31.12GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:41:00.636937+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Failed to get request counts for guanaco-submitter. Falling back to default
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:41:05 INFO device.py L1468:  'peak_ram': 31.12GB, 'peak_vram': 1.59GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:41:19 INFO device.py L1468:  'peak_ram': 32.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:41:31 INFO device.py L1468:  'peak_ram': 32.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:41:46 INFO device.py L1468:  'peak_ram': 33.11GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:42:01.012671+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:41:57 INFO device.py L1468:  'peak_ram': 33.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:42:12 INFO device.py L1468:  'peak_ram': 33.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:42:24 INFO device.py L1468:  'peak_ram': 33.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:42:34 INFO device.py L1468:  'peak_ram': 33.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:42:50 INFO device.py L1468:  'peak_ram': 33.17GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:43:01.219638+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:43:03 INFO device.py L1468:  'peak_ram': 33.17GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:43:17 INFO device.py L1468:  'peak_ram': 34.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:43:28 INFO device.py L1468:  'peak_ram': 34.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:43:40 INFO device.py L1468:  'peak_ram': 34.58GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:43:54 INFO device.py L1468:  'peak_ram': 34.58GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:44:01.319066+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:44:05 INFO device.py L1468:  'peak_ram': 35.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:44:16 INFO device.py L1468:  'peak_ram': 35.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:44:30 INFO device.py L1468:  'peak_ram': 36.41GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:44:42 INFO device.py L1468:  'peak_ram': 36.41GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:44:55 INFO device.py L1468:  'peak_ram': 37.51GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:45:01.413225+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:45:06 INFO device.py L1468:  'peak_ram': 37.51GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:45:18 INFO device.py L1468:  'peak_ram': 38.1GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:45:32 INFO device.py L1468:  'peak_ram': 38.1GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:45:42 INFO device.py L1468:  'peak_ram': 38.96GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:45:57 INFO device.py L1468:  'peak_ram': 38.96GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:46:01.695970+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:46:08 INFO device.py L1468:  'peak_ram': 38.96GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:46:20 INFO device.py L1468:  'peak_ram': 38.96GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:46:37 INFO device.py L1468:  'peak_ram': 38.96GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:47:01.982242+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:46:55 INFO device.py L1468:  'peak_ram': 39.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:47:12 INFO device.py L1468:  'peak_ram': 39.14GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:47:33 INFO device.py L1468:  'peak_ram': 40.41GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:47:50 INFO device.py L1468:  'peak_ram': 40.41GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:48:03.662563+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:48:11 INFO device.py L1468:  'peak_ram': 41.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:48:28 INFO device.py L1468:  'peak_ram': 41.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:48:46 INFO device.py L1468:  'peak_ram': 42.15GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:49:04.064495+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:49:06 INFO device.py L1468:  'peak_ram': 42.15GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:49:15 INFO device.py L1468:  'peak_ram': 43.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:49:24 INFO device.py L1468:  'peak_ram': 43.21GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:49:37 INFO device.py L1468:  'peak_ram': 44.12GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:49:49 INFO device.py L1468:  'peak_ram': 44.12GB, 'peak_vram': 1.59GB[0m
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-30T20:50:04.410899+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:50:07 INFO device.py L1468:  'peak_ram': 45.47GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:50:20 INFO device.py L1468:  'peak_ram': 45.47GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:50:33 INFO device.py L1468:  'peak_ram': 46.31GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:50:52 INFO device.py L1468:  'peak_ram': 46.31GB, 'peak_vram': 1.59GB[0m
2026-03-30T20:51:04.689337+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:51:04 INFO device.py L1468:  'peak_ram': 46.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:51:22 INFO device.py L1468:  'peak_ram': 46.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:51:32 INFO device.py L1468:  'peak_ram': 46.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:51:41 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [33;1m2026-03-30 20:51:41 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: [38;20m2026-03-30 20:51:42 INFO device.py L1468:  'peak_ram': 46.59GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-combi-24742-v1-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-24742-v1-uploader: Creating repo ChaiML/gspo-glm47-combine-rm82-mega-data-step1100-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------- 2026-03-30 20:51:42 (0:00:00) ----------
chaiml-gspo-glm47-combi-24742-v1-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/1 (0.0/197.4G) (+41 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v1-uploader: Workers: hashing: 38 | get upload mode: 1 | pre-uploading: 1 | committing: 0 | waiting: 24
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------------------------------------------------
2026-03-30T20:52:04.999837+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------- 2026-03-30 20:52:42 (0:01:00) ----------
chaiml-gspo-glm47-combi-24742-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 11/40 (41.8G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 29 | committing: 0 | waiting: 35
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------------------------------------------------
2026-03-30T20:53:06.004968+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------- 2026-03-30 20:53:43 (0:02:00) ----------
chaiml-gspo-glm47-combi-24742-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 30/40 (143.7G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-24742-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 54
chaiml-gspo-glm47-combi-24742-v1-uploader: ---------------------------------------------------
2026-03-30T20:54:06.323124+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm82-mega-data-step1100 in 1933.167s
chaiml-gspo-glm47-combi-24742-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-24742-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-24742-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-24742-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-24742-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-24742-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-24742-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-24742-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/quantization_config.json
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/generation_config.json
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/tokenizer_config.json
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/chat_template.jinja
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/config.json
2026-03-30T20:55:06.680033+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-24742-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-24742-v1/default/model-00012-of-00038.safetensors
Job chaiml-gspo-glm47-combi-24742-v1-uploader completed after 2042.8s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-24742-v1-uploader
Pipeline stage VLLMUploader completed in 2044.99s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 1.18s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 5.45s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-24742-v1
Waiting for inference service chaiml-gspo-glm47-combi-24742-v1 to be ready
2026-03-30T20:56:06.961443+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:57:07.435429+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:58:08.046073+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T20:59:09.072957+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:00:09.395810+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:01:09.511418+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:02:09.626590+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:03:10.642132+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:04:11.872431+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:05:12.196761+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:06:12.601292+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:07:13.677632+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:08:14.030589+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:09:14.382735+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:10:14.701986+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Failed to get response for submission chaiml-gspo-glm47-chai-r_6996_v1: ('http://chaiml-gspo-glm47-chai-r-6996-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'request timeout')
2026-03-30T21:11:15.011838+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
2026-03-30T21:12:15.836527+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Inference service chaiml-gspo-glm47-combi-24742-v1 ready after 973.0211684703827s
Pipeline stage VLLMDeployer completed in 975.69s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.726537466049194s
Received healthy response to inference request in 2.8979427814483643s
Received healthy response to inference request in 2.0528719425201416s
Received healthy response to inference request in 2.84077787399292s
Received healthy response to inference request in 2.1490561962127686s
Received healthy response to inference request in 2.492488384246826s
Received healthy response to inference request in 2.027252197265625s
Received healthy response to inference request in 2.0422308444976807s
Received healthy response to inference request in 2.810486078262329s
Received healthy response to inference request in 2.6282718181610107s
Received healthy response to inference request in 2.1409974098205566s
Received healthy response to inference request in 3.1428394317626953s
Received healthy response to inference request in 3.079232931137085s
Received healthy response to inference request in 2.3480935096740723s
2026-03-30T21:13:16.363922+00:00 monitor updated for chaiml-gspo-glm47-combi_24742_v1
Received healthy response to inference request in 2.779360294342041s
Received healthy response to inference request in 2.065258741378784s
Received healthy response to inference request in 2.9804365634918213s
Received healthy response to inference request in 2.395082950592041s
Received healthy response to inference request in 2.801126718521118s
Received healthy response to inference request in 2.604985237121582s
Received healthy response to inference request in 2.0152268409729004s
Received healthy response to inference request in 3.019589900970459s
Received healthy response to inference request in 2.287397861480713s
Received healthy response to inference request in 2.0505728721618652s
Received healthy response to inference request in 2.4734723567962646s
Received healthy response to inference request in 2.330235242843628s
Received healthy response to inference request in 2.0573999881744385s
Received healthy response to inference request in 2.3035802841186523s
Received healthy response to inference request in 2.7685821056365967s
Received healthy response to inference request in 2.2202351093292236s
30 requests
0 failed requests
5th percentile: 2.03399258852005
10th percentile: 2.0497386693954467
20th percentile: 2.063686990737915
30th percentile: 2.198881435394287
40th percentile: 2.319573259353638
50th percentile: 2.434277653694153
60th percentile: 2.6142998695373536
70th percentile: 2.785890221595764
80th percentile: 2.852210855484009
90th percentile: 3.0255542039871215
95th percentile: 3.1142165064811707
99th percentile: 7.1072650361061145
mean time: 2.6843873977661135
Pipeline stage StressChecker completed in 102.34s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.37s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_24742_v1 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_24742_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-gspo-glm47-combi_24742_v1 status is now torndown due to DeploymentManager action