developer_uid: richhx
submission_id: chaiml-gspo-glm47-chai-_76408_v1
model_name: chaiml-gspo-glm47-chai-_76408_v1
model_group: ChaiML/gspo-glm47-chai-r
status: deployed
timestamp: 2026-03-27T20:14:48+00:00
num_battles: 12637
num_wins: 7279
celo_rating: 1346.2
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-chai-_76408_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5760069636780881
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '</s>', '<|assistant|>', '<|im_end|>', '####'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-chai-76408-v1-uploader
Waiting for job on chaiml-gspo-glm47-chai-76408-v1-uploader to finish
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: {repo_id} is already quantized
chaiml-gspo-glm47-chai-76408-v1-uploader: Using quantization_mode: none
chaiml-gspo-glm47-chai-76408-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8...
2026-03-27T20:00:03.635206+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:00:14.276244+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-95963-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-95963-v1/default/model-00003-of-00038.safetensors
Job chaiml-gspo-glm47-chai-95963-v1-uploader completed after 348.31s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-95963-v1-uploader
Pipeline stage VLLMUploader completed in 529.54s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.46s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.58s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-95963-v1
Waiting for inference service chaiml-gspo-glm47-chai-95963-v1 to be ready
2026-03-27T20:01:03.982704+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:01:14.794647+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:02:04.366523+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:02:15.138496+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-76408-v1-uploader: 'The read operation timed out' thrown while requesting HEAD https://huggingface.co/ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8/resolve/c41657dbe1763084c3fbe5f16b192ef824d3c91e/model-00040-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: Retrying in 1s [Retry 1/5].
chaiml-gspo-glm47-chai-76408-v1-uploader: Downloaded in 163.503s
2026-03-27T20:03:04.884348+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:03:15.477270+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:04:05.225697+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
2026-03-27T20:04:15.861276+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:05:06.234519+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
chaiml-gspo-glm47-chai-76408-v1-uploader: Processed model ChaiML/gspo-glm47-chai-rm-kimi-data-step500-FP8 in 315.454s
chaiml-gspo-glm47-chai-76408-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-76408-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-76408-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
2026-03-27T20:05:16.264370+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
chaiml-gspo-glm47-chai-76408-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-76408-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-76408-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-76408-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-76408-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/generation_config.json
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/config.json
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/.gitattributes
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/chat_template.jinja
Inference service chaiml-gspo-glm47-chai-95963-v1 ready after 273.25780272483826s
Pipeline stage VLLMDeployer completed in 275.32s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.03371262550354s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00072-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00072-of-00072.safetensors
Received healthy response to inference request in 8.59642767906189s
Received healthy response to inference request in 2.17704701423645s
Received healthy response to inference request in 8.703476905822754s
Received healthy response to inference request in 2.176699638366699s
2026-03-27T20:06:06.653565+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
Received healthy response to inference request in 9.317871809005737s
2026-03-27T20:06:17.013261+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 9.36390733718872s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00007-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00007-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00009-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00009-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00045-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00045-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00005-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00005-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00036-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00036-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00038-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00038-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00039-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00039-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00018-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00018-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00003-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00003-of-00072.safetensors
Received healthy response to inference request in 2.1320817470550537s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00012-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00012-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00013-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00013-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00017-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00017-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00067-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00067-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00030-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00030-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00041-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00041-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00040-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00040-of-00072.safetensors
Received healthy response to inference request in 2.131870746612549s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00024-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00024-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00042-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00042-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00021-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00021-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00026-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00026-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00044-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00044-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00010-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00010-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00047-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00047-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00037-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00037-of-00072.safetensors
Received healthy response to inference request in 2.1966140270233154s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00046-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00046-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00014-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00014-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00034-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00034-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00048-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00048-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00015-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00015-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00029-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00029-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00052-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00052-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00032-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00032-of-00072.safetensors
Received healthy response to inference request in 2.097015857696533s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00004-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00004-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00016-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00016-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00027-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00027-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00054-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00054-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00069-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00069-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00020-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00020-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00022-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00022-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00031-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00031-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00025-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00025-of-00072.safetensors
Received healthy response to inference request in 2.1254215240478516s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00001-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00001-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00062-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00062-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00006-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00006-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00068-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00068-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00050-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00050-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00043-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00043-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00066-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00066-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00033-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00033-of-00072.safetensors
Received healthy response to inference request in 2.2825069427490234s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00019-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00019-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00035-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00035-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00063-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00063-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00011-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00011-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00070-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00070-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00064-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00064-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00065-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00065-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00060-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00060-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00023-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00023-of-00072.safetensors
Received healthy response to inference request in 2.3192391395568848s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00028-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00028-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00059-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00059-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00008-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00008-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00055-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00055-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00061-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00061-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00049-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00049-of-00072.safetensors
Received healthy response to inference request in 2.245492696762085s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00058-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00058-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00057-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00057-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00051-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00051-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00056-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00056-of-00072.safetensors
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00053-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00053-of-00072.safetensors
Received healthy response to inference request in 2.1251649856567383s
chaiml-gspo-glm47-chai-76408-v1-uploader: cp /dev/shm/model_output/model-00002-of-00072.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-76408-v1/default/model-00002-of-00072.safetensors
Received healthy response to inference request in 2.2733314037323s
Received healthy response to inference request in 2.2194712162017822s
Received healthy response to inference request in 2.3096272945404053s
Received healthy response to inference request in 2.416706085205078s
Job chaiml-gspo-glm47-chai-76408-v1-uploader completed after 463.1s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-76408-v1-uploader
Pipeline stage VLLMUploader completed in 465.01s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Received healthy response to inference request in 2.150949239730835s
Pipeline stage VLLMUploaderAMD completed in 0.59s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.57s
Received healthy response to inference request in 2.365492105484009s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-76408-v1
Waiting for inference service chaiml-gspo-glm47-chai-76408-v1 to be ready
Received healthy response to inference request in 2.1242623329162598s
2026-03-27T20:07:07.000569+00:00 monitor updated for chaiml-gspo-glm47-chai-_95963_v1
Received healthy response to inference request in 2.179020404815674s
Received healthy response to inference request in 2.346162796020508s
Received healthy response to inference request in 2.181541919708252s
2026-03-27T20:07:17.358329+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 2.6161746978759766s
Received healthy response to inference request in 2.1475462913513184s
Received healthy response to inference request in 2.2532219886779785s
Received healthy response to inference request in 2.2360966205596924s
30 requests
0 failed requests
5th percentile: 2.124668526649475
10th percentile: 2.1253958702087403
20th percentile: 2.1444533824920655
30th percentile: 2.176942801475525
40th percentile: 2.19058518409729
50th percentile: 2.2407946586608887
60th percentile: 2.2770016193389893
70th percentile: 2.3273162364959714
80th percentile: 2.4565998077392583
90th percentile: 8.736500477790832
95th percentile: 9.190000176429749
99th percentile: 9.350557034015656
mean time: 3.3614718357721967
Pipeline stage StressChecker completed in 122.99s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.29s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_95963_v1 status is now deployed due to DeploymentManager action
2026-03-27T20:08:17.793017+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:09:18.090635+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:10:18.631222+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:11:18.957594+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
2026-03-27T20:12:19.246594+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Inference service chaiml-gspo-glm47-chai-76408-v1 ready after 354.39419078826904s
Pipeline stage VLLMDeployer completed in 357.35s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.640963315963745s
2026-03-27T20:13:19.501828+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 9.38604736328125s
Received healthy response to inference request in 9.45284390449524s
Received healthy response to inference request in 2.3761961460113525s
Received healthy response to inference request in 2.5067546367645264s
Received healthy response to inference request in 9.471636295318604s
Received healthy response to inference request in 2.1963870525360107s
Received healthy response to inference request in 2.117433547973633s
Received healthy response to inference request in 2.150157928466797s
Received healthy response to inference request in 2.1868598461151123s
Received healthy response to inference request in 2.59916353225708s
Received healthy response to inference request in 2.1907131671905518s
Received healthy response to inference request in 2.1483676433563232s
Received healthy response to inference request in 2.8313252925872803s
Received healthy response to inference request in 2.1140081882476807s
Received healthy response to inference request in 2.107861042022705s
Received healthy response to inference request in 2.106801986694336s
Received healthy response to inference request in 2.1476938724517822s
Received healthy response to inference request in 2.083630084991455s
Received healthy response to inference request in 2.1762421131134033s
2026-03-27T20:14:19.760142+00:00 monitor updated for chaiml-gspo-glm47-chai-_76408_v1
Received healthy response to inference request in 2.2498250007629395s
Received healthy response to inference request in 2.139725923538208s
Received healthy response to inference request in 2.1106159687042236s
Received healthy response to inference request in 2.2753522396087646s
Received healthy response to inference request in 2.3356246948242188s
Received healthy response to inference request in 2.1750857830047607s
Received healthy response to inference request in 2.276712656021118s
Received healthy response to inference request in 2.136345863342285s
Received healthy response to inference request in 2.185283660888672s
Received healthy response to inference request in 2.1333463191986084s
30 requests
0 failed requests
5th percentile: 2.107278561592102
10th percentile: 2.1103404760360718
20th percentile: 2.1301637649536134
30th percentile: 2.14530348777771
40th percentile: 2.165114641189575
50th percentile: 2.186071753501892
60th percentile: 2.2177622318267822
70th percentile: 2.2943862676620483
80th percentile: 2.5252364158630374
90th percentile: 9.392727017402649
95th percentile: 9.463179719448089
99th percentile: 9.591858479976654
mean time: 3.200300168991089
Pipeline stage StressChecker completed in 103.46s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.40s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_76408_v1 status is now deployed due to DeploymentManager action