chaiml-gspo-glm47-combi_16773

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-combi_16773_v2
model_name: chaiml-gspo-glm47-combi_16773_v2
model_group: ChaiML/gspo-glm47-combin
status: torndown
timestamp: 2026-04-03T23:41:15+00:00
num_battles: 10362
num_wins: 6089
celo_rating: 1354.87
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-combine-rm721-mega-data-step1000
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-combi_16773_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-combine-rm721-mega-data-step1000
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-31
win_ratio: 0.5876278710673615
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '<|user|>', '####', '<|im_end|>', '<|assistant|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-16773-v2-uploader
Waiting for job on chaiml-gspo-glm47-combi-16773-v2-uploader to finish
chaiml-gspo-glm47-combi-16773-v2-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-16773-v2-uploader: Checking if ChaiML/gspo-glm47-combine-rm721-mega-data-step1000-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-16773-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm47-combi-16773-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm721-mega-data-step1000-W4A16...
2026-03-31T19:04:31.586134+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:05:31.864352+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
chaiml-gspo-glm47-combi-16773-v2-uploader: Downloaded in 88.534s
chaiml-gspo-glm47-combi-16773-v2-uploader: Processed model ChaiML/gspo-glm47-combine-rm721-mega-data-step1000 in 91.324s
chaiml-gspo-glm47-combi-16773-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-16773-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-16773-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-16773-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-16773-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-16773-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-16773-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/config.json
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/.gitattributes
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/chat_template.jinja
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/generation_config.json
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/quantization_config.json
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/tokenizer_config.json
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v2/default/model-00035-of-00038.safetensors
2026-03-31T19:06:33.332605+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Job chaiml-gspo-glm47-combi-16773-v2-uploader completed after 196.96s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-16773-v2-uploader
Pipeline stage VLLMUploader completed in 197.53s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.67s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-16773-v2
Waiting for inference service chaiml-gspo-glm47-combi-16773-v2 to be ready
2026-03-31T19:07:33.442843+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:08:33.586172+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:09:33.722854+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:10:33.819626+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Failed to get request counts for guanaco-submitter. Falling back to default
2026-03-31T19:11:34.008616+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:12:34.255685+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:13:34.480694+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:14:35.168393+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:15:35.476685+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:16:35.673688+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-31T19:17:35.767545+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
2026-03-31T19:18:35.866960+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Inference service chaiml-gspo-glm47-combi-16773-v2 ready after 742.885938167572s
Pipeline stage VLLMDeployer completed in 744.44s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.257064580917358s
Received healthy response to inference request in 2.1436681747436523s
Received healthy response to inference request in 2.030360221862793s
Received healthy response to inference request in 2.103459358215332s
2026-03-31T19:19:35.962509+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Received healthy response to inference request in 2.320014715194702s
Received healthy response to inference request in 2.4229140281677246s
Received healthy response to inference request in 2.091975212097168s
Received healthy response to inference request in 1.9950926303863525s
Received healthy response to inference request in 2.22174334526062s
Received healthy response to inference request in 1.9627172946929932s
Received healthy response to inference request in 2.0466878414154053s
Received healthy response to inference request in 2.04248046875s
Received healthy response to inference request in 2.0239408016204834s
Received healthy response to inference request in 2.0759975910186768s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-31T19:20:36.063840+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-31T19:21:36.305361+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-31T19:22:36.557174+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-31T19:23:37.333941+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 9.061258792877197s
Received healthy response to inference request in 2.223965883255005s
Received healthy response to inference request in 2.834580183029175s
30 requests
13 failed requests
5th percentile: 2.0080743074417113
10th percentile: 2.029718279838562
20th percentile: 2.0701356410980223
30th percentile: 2.131605529785156
40th percentile: 2.2815951824188234
50th percentile: 5.947919487953186
60th percentile: 20.135033130645752
70th percentile: 20.14516065120697
80th percentile: 20.154928112030028
90th percentile: 20.19813702106476
95th percentile: 20.2697758436203
99th percentile: 20.423999104499817
mean time: 10.4462459564209
%s, retrying in %s seconds...
2026-03-31T19:24:37.523352+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Received healthy response to inference request in 8.983070850372314s
Received healthy response to inference request in 2.1949901580810547s
Received healthy response to inference request in 1.9897704124450684s
Received healthy response to inference request in 2.0228826999664307s
Received healthy response to inference request in 2.181058645248413s
Received healthy response to inference request in 3.0330328941345215s
Received healthy response to inference request in 2.0928285121917725s
Received healthy response to inference request in 2.211665153503418s
Received healthy response to inference request in 1.9719994068145752s
Received healthy response to inference request in 2.459226608276367s
Received healthy response to inference request in 2.0568432807922363s
Received healthy response to inference request in 2.415371894836426s
Received healthy response to inference request in 2.131622314453125s
Received healthy response to inference request in 2.0846714973449707s
Received healthy response to inference request in 1.9905381202697754s
Received healthy response to inference request in 2.0911104679107666s
Received healthy response to inference request in 2.545097589492798s
Received healthy response to inference request in 2.287297487258911s
Received healthy response to inference request in 1.9672422409057617s
Received healthy response to inference request in 2.0080885887145996s
Received healthy response to inference request in 2.2533814907073975s
Received healthy response to inference request in 2.256010055541992s
Received healthy response to inference request in 2.1360154151916504s
2026-03-31T19:25:37.628175+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v2
Received healthy response to inference request in 1.9987952709197998s
Received healthy response to inference request in 2.015263319015503s
Received healthy response to inference request in 2.0350942611694336s
Received healthy response to inference request in 2.038966417312622s
Received healthy response to inference request in 2.2300968170166016s
Received healthy response to inference request in 2.0709736347198486s
Received healthy response to inference request in 2.1397104263305664s
30 requests
0 failed requests
5th percentile: 1.9799963593482972
10th percentile: 1.9904613494873047
20th percentile: 2.0138283729553224
30th percentile: 2.0378047704696653
40th percentile: 2.079192352294922
50th percentile: 2.1122254133224487
60th percentile: 2.156249713897705
70th percentile: 2.217194652557373
80th percentile: 2.262267541885376
90th percentile: 2.4678137063980103
95th percentile: 2.8134620070457443
99th percentile: 7.25755984306336
mean time: 2.396423864364624
Pipeline stage StressChecker completed in 393.73s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.93s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_16773_v2 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_16773_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-gspo-glm47-combi_16773_v2 status is now torndown due to DeploymentManager action