developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-chai-_46797_v2
model_name: chaiml-gspo-glm47-chai-_46797_v2
model_group: ChaiML/gspo-glm47-chai-r
status: inactive
timestamp: 2026-03-27T10:18:41+00:00
num_battles: 11642
num_wins: 6693
celo_rating: 1344.81
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-chai-rm-kimi-data-step500
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-chai-_46797_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-chai-rm-kimi-data-step500
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5749012197216973
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['####', '</s>', '<|assistant|>', '<|user|>', '<|im_end|>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-chai-46797-v2-uploader
Waiting for job on chaiml-gspo-glm47-chai-46797-v2-uploader to finish
2026-03-27T08:02:21.682515+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-combi-87059-v2-uploader: 2026-03-27 08:02:23 INFO device.py L1468:  'peak_ram': 42.57GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-combi-87059-v2-uploader: 2026-03-27 08:02:32 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/
chaiml-gspo-glm47-combi-87059-v2-uploader: 2026-03-27 08:02:32 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict
chaiml-gspo-glm47-combi-87059-v2-uploader: 2026-03-27 08:02:33 INFO device.py L1468:  'peak_ram': 42.57GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-combi-87059-v2-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-87059-v2-uploader: Creating repo ChaiML/gspo-glm47-combine-rm82-kimi-data-step500-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-combi-87059-v2-uploader: ---------- 2026-03-27 08:02:34 (0:00:00) ----------
chaiml-gspo-glm47-combi-87059-v2-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+41 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-87059-v2-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-gspo-glm47-combi-87059-v2-uploader: ---------------------------------------------------
chaiml-gspo-glm47-chai-46797-v2-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-chai-46797-v2-uploader: Checking if ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-chai-46797-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-chai-rm-kimi-data-step500...
2026-03-27T08:03:18.959438+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:03:21.968513+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-combi-87059-v2-uploader: 







chaiml-gspo-glm47-combi-87059-v2-uploader: ---------- 2026-03-27 08:03:34 (0:01:00) ----------
chaiml-gspo-glm47-combi-87059-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 11/40 (41.8G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-87059-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 29 | committing: 0 | waiting: 35
chaiml-gspo-glm47-combi-87059-v2-uploader: ---------------------------------------------------
2026-03-27T08:04:19.255453+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:04:22.492363+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-combi-87059-v2-uploader:                              







chaiml-gspo-glm47-combi-87059-v2-uploader: ---------- 2026-03-27 08:04:34 (0:02:00) ----------
chaiml-gspo-glm47-combi-87059-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 35/40 (170.6G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-combi-87059-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 5 | committing: 0 | waiting: 59
chaiml-gspo-glm47-combi-87059-v2-uploader: ---------------------------------------------------
2026-03-27T08:05:19.562583+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:05:22.792646+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-combi-87059-v2-uploader:                              Processed model ChaiML/gspo-glm47-combine-rm82-kimi-data-step500 in 1946.739s
chaiml-gspo-glm47-combi-87059-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-87059-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-87059-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-87059-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-87059-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-87059-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-87059-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/chat_template.jinja
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/generation_config.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/tokenizer_config.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/config.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/tokenizer.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/quantization_config.json
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00021-of-00038.safetensors
2026-03-27T08:06:19.856322+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v2/default/model-00033-of-00038.safetensors
2026-03-27T08:06:23.090318+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
Job chaiml-gspo-glm47-combi-87059-v2-uploader completed after 2056.46s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-87059-v2-uploader
Pipeline stage VLLMUploader completed in 2057.96s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.12s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-87059-v2
Waiting for inference service chaiml-gspo-glm47-combi-87059-v2 to be ready
2026-03-27T08:07:20.426814+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:07:23.417491+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: Downloaded in 247.894s
2026-03-27T08:08:20.760674+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:08:23.762107+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
2026-03-27T08:09:21.089222+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:09:24.089096+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
2026-03-27T08:10:21.420306+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:10:24.447502+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
Inference service chaiml-gspo-glm47-combi-87059-v2 ready after 283.0655655860901s
Pipeline stage VLLMDeployer completed in 284.86s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-27T08:11:21.750679+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:11:24.774517+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
Received healthy response to inference request in 8.886432647705078s
Received healthy response to inference request in 8.88759446144104s
Received healthy response to inference request in 8.78194785118103s
Received healthy response to inference request in 8.943094491958618s
Received healthy response to inference request in 2.11165452003479s
Received healthy response to inference request in 2.162609577178955s
Received healthy response to inference request in 2.1387414932250977s
chaiml-gspo-glm47-chai-46797-v2-uploader: Applying quantization...
Received healthy response to inference request in 8.828080654144287s
2026-03-27T08:12:22.336125+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:12:25.380883+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:12:04 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE
Received healthy response to inference request in 2.1159474849700928s
Received healthy response to inference request in 2.1539924144744873s
Received healthy response to inference request in 2.1094753742218018s
Received healthy response to inference request in 2.106980323791504s
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:12:36 INFO base.py L486: using torch.bfloat16 for quantization tuning
Received healthy response to inference request in 2.2113053798675537s
Received healthy response to inference request in 2.2233991622924805s
Received healthy response to inference request in 2.130335569381714s
Received healthy response to inference request in 2.1228585243225098s
Received healthy response to inference request in 2.1133008003234863s
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:12:47 INFO device.py L1468:  'peak_ram': 11.94GB, 'peak_vram': 1.44GB
Received healthy response to inference request in 2.2122013568878174s
Received healthy response to inference request in 2.2248446941375732s
Received healthy response to inference request in 2.2551991939544678s
Received healthy response to inference request in 2.2109341621398926s
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:12:52 INFO device.py L1468:  'peak_ram': 16.4GB, 'peak_vram': 1.59GB
Received healthy response to inference request in 2.4221878051757812s
Received healthy response to inference request in 2.2163898944854736s
Received healthy response to inference request in 2.235858678817749s
Received healthy response to inference request in 2.2354657649993896s
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:02 INFO device.py L1468:  'peak_ram': 17.0GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:07 INFO device.py L1468:  'peak_ram': 19.42GB, 'peak_vram': 1.59GB
Received healthy response to inference request in 2.189955472946167s
Received healthy response to inference request in 2.624922752380371s
Received healthy response to inference request in 2.1128175258636475s
Received healthy response to inference request in 2.3282339572906494s
Received healthy response to inference request in 2.18342924118042s
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:16 INFO device.py L1468:  'peak_ram': 19.83GB, 'peak_vram': 1.59GB
30 requests
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:21 INFO device.py L1468:  'peak_ram': 22.25GB, 'peak_vram': 1.59GB
0 failed requests
5th percentile: 2.1104559898376465
10th percentile: 2.1127012252807615
20th percentile: 2.1214763164520263
30th percentile: 2.1494171380996705
40th percentile: 2.1873449802398683
50th percentile: 2.2117533683776855
60th percentile: 2.2239773750305174
70th percentile: 2.2416608333587646
80th percentile: 2.4627347946166998
90th percentile: 8.833915853500367
95th percentile: 8.887071645259857
99th percentile: 8.92699948310852
mean time: 3.316006374359131
Pipeline stage StressChecker completed in 123.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.85s
2026-03-27T08:13:28.040207+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Shutdown handler de-registered
chaiml-gspo-glm47-combi_87059_v2 status is now deployed due to DeploymentManager action
2026-03-27T08:13:28.182901+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:27 INFO device.py L1468:  'peak_ram': 22.25GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:34 INFO device.py L1468:  'peak_ram': 22.7GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:40 INFO device.py L1468:  'peak_ram': 22.7GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:48 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:13:59 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:07 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:13 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:18 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
2026-03-27T08:14:28.443578+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:27 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:32 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:40 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:45 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:51 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:14:59 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:15:07 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
2026-03-27T08:15:28.654962+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:15:22 INFO device.py L1468:  'peak_ram': 24.37GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:15:34 INFO device.py L1468:  'peak_ram': 24.37GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:15:45 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:16:00 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:16:13 INFO device.py L1468:  'peak_ram': 25.71GB, 'peak_vram': 1.59GB
2026-03-27T08:16:28.917297+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:16:27 INFO device.py L1468:  'peak_ram': 25.71GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:16:54 INFO device.py L1468:  'peak_ram': 26.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:17:09 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:17:23 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
2026-03-27T08:17:29.134346+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:17:36 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:17:53 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:18:03 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:18:15 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
2026-03-27T08:18:29.344960+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:18:29 INFO device.py L1468:  'peak_ram': 27.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:18:40 INFO device.py L1468:  'peak_ram': 27.39GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:18:54 INFO device.py L1468:  'peak_ram': 27.39GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:19:07 INFO device.py L1468:  'peak_ram': 28.17GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:19:19 INFO device.py L1468:  'peak_ram': 28.17GB, 'peak_vram': 1.59GB
2026-03-27T08:19:29.569661+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:19:34 INFO device.py L1468:  'peak_ram': 29.3GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:19:45 INFO device.py L1468:  'peak_ram': 29.3GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:19:59 INFO device.py L1468:  'peak_ram': 30.15GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:20:13 INFO device.py L1468:  'peak_ram': 30.15GB, 'peak_vram': 1.59GB
2026-03-27T08:20:29.774206+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:20:26 INFO device.py L1468:  'peak_ram': 30.58GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:20:58 INFO device.py L1468:  'peak_ram': 31.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:21:08 INFO device.py L1468:  'peak_ram': 31.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:21:25 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
2026-03-27T08:21:30.044145+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:21:40 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:21:57 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:22:10 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:22:25 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
2026-03-27T08:22:30.269063+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:22:42 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:22:57 INFO device.py L1468:  'peak_ram': 32.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:23:17 INFO device.py L1468:  'peak_ram': 33.32GB, 'peak_vram': 1.59GB
2026-03-27T08:23:30.530269+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:23:32 INFO device.py L1468:  'peak_ram': 33.32GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:23:46 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:24:04 INFO device.py L1468:  'peak_ram': 33.75GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:24:14 INFO device.py L1468:  'peak_ram': 34.42GB, 'peak_vram': 1.59GB
2026-03-27T08:24:30.749981+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:24:30 INFO device.py L1468:  'peak_ram': 34.42GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:24:48 INFO device.py L1468:  'peak_ram': 35.32GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:25:03 INFO device.py L1468:  'peak_ram': 35.32GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:25:17 INFO device.py L1468:  'peak_ram': 36.36GB, 'peak_vram': 1.59GB
2026-03-27T08:25:31.042929+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:25:44 INFO device.py L1468:  'peak_ram': 36.88GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:26:03 INFO device.py L1468:  'peak_ram': 36.88GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:26:19 INFO device.py L1468:  'peak_ram': 37.69GB, 'peak_vram': 1.59GB
2026-03-27T08:26:35.360388+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:26:32 INFO device.py L1468:  'peak_ram': 37.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:26:42 INFO device.py L1468:  'peak_ram': 37.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:26:52 INFO device.py L1468:  'peak_ram': 37.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:27:07 INFO device.py L1468:  'peak_ram': 37.69GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:27:19 INFO device.py L1468:  'peak_ram': 37.88GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:27:30 INFO device.py L1468:  'peak_ram': 37.88GB, 'peak_vram': 1.59GB
2026-03-27T08:27:35.580286+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:27:44 INFO device.py L1468:  'peak_ram': 39.0GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:27:55 INFO device.py L1468:  'peak_ram': 39.0GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:28:10 INFO device.py L1468:  'peak_ram': 39.92GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:28:21 INFO device.py L1468:  'peak_ram': 39.92GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:28:32 INFO device.py L1468:  'peak_ram': 40.44GB, 'peak_vram': 1.59GB
2026-03-27T08:28:35.812840+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:28:47 INFO device.py L1468:  'peak_ram': 40.44GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:28:59 INFO device.py L1468:  'peak_ram': 41.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:29:11 INFO device.py L1468:  'peak_ram': 41.52GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:29:21 INFO device.py L1468:  'peak_ram': 42.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:29:26 INFO device.py L1468:  'peak_ram': 42.46GB, 'peak_vram': 1.59GB
2026-03-27T08:29:36.041863+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:29:37 INFO device.py L1468:  'peak_ram': 43.59GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:29:47 INFO device.py L1468:  'peak_ram': 43.59GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:11 INFO device.py L1468:  'peak_ram': 44.49GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:15 INFO device.py L1468:  'peak_ram': 44.78GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:23 INFO device.py L1468:  'peak_ram': 44.78GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:27 INFO device.py L1468:  'peak_ram': 44.78GB, 'peak_vram': 1.59GB
2026-03-27T08:30:36.477306+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:37 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:37 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict
chaiml-gspo-glm47-chai-46797-v2-uploader: 2026-03-27 08:30:38 INFO device.py L1468:  'peak_ram': 44.78GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-chai-46797-v2-uploader: Checking if ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-chai-46797-v2-uploader: Creating repo ChaiML/gspo-glm47-chai-rm-kimi-data-step500-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-chai-46797-v2-uploader: ---------- 2026-03-27 08:30:38 (0:00:00) ----------
chaiml-gspo-glm47-chai-46797-v2-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/1 (0.0/197.4G) (+43 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-46797-v2-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-gspo-glm47-chai-46797-v2-uploader: ---------------------------------------------------
2026-03-27T08:31:36.704259+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: 







chaiml-gspo-glm47-chai-46797-v2-uploader: ---------- 2026-03-27 08:31:38 (0:01:00) ----------
chaiml-gspo-glm47-chai-46797-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 11/40 (41.8G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-46797-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 29 | committing: 0 | waiting: 35
chaiml-gspo-glm47-chai-46797-v2-uploader: ---------------------------------------------------
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-cas72-44480-v2-uploader
Waiting for job on chaiml-gspo-glm47-cas72-44480-v2-uploader to finish
2026-03-27T08:32:36.967872+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader:                              







chaiml-gspo-glm47-chai-46797-v2-uploader: ---------- 2026-03-27 08:32:38 (0:02:00) ----------
chaiml-gspo-glm47-chai-46797-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 39/40 (192.0G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-46797-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 63
chaiml-gspo-glm47-chai-46797-v2-uploader: ---------------------------------------------------
chaiml-gspo-glm47-cas72-44480-v2-uploader: Checking if ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-cas72-44480-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500...
2026-03-27T08:33:21.788820+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:33:37.291939+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader:                              Processed model ChaiML/gspo-glm47-chai-rm-kimi-data-step500 in 1829.749s
chaiml-gspo-glm47-chai-46797-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-46797-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-chai-46797-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-46797-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-46797-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-46797-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/chat_template.jinja
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/quantization_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/generation_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/tokenizer_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/tokenizer.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00037-of-00038.safetensors
2026-03-27T08:34:22.109526+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00023-of-00038.safetensors
2026-03-27T08:34:37.638139+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Job chaiml-gspo-glm47-chai-46797-v2-uploader completed after 1945.09s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-46797-v2-uploader
Pipeline stage VLLMUploader completed in 1946.71s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.48s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.08s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-46797-v2
Waiting for inference service chaiml-gspo-glm47-chai-46797-v2 to be ready
2026-03-27T08:35:22.441630+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:35:37.963003+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:36:22.868420+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:36:38.311934+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:37:23.269917+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: Downloaded in 247.189s
2026-03-27T08:37:38.644343+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:38:23.690007+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:38:38.985718+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:39:24.097290+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Inference service chaiml-gspo-glm47-chai-46797-v2 ready after 273.74899220466614s
Pipeline stage VLLMDeployer completed in 275.71s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.624801635742188s
Received healthy response to inference request in 2.2115416526794434s
2026-03-27T08:39:39.321139+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Received healthy response to inference request in 9.377281427383423s
Received healthy response to inference request in 2.194185495376587s
Received healthy response to inference request in 2.1215548515319824s
Received healthy response to inference request in 8.59005093574524s
Received healthy response to inference request in 8.840625286102295s
Received healthy response to inference request in 2.1734626293182373s
Received healthy response to inference request in 2.209822416305542s
Received healthy response to inference request in 2.2038381099700928s
Received healthy response to inference request in 2.1268436908721924s
2026-03-27T08:40:24.810769+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Received healthy response to inference request in 9.476300239562988s
Received healthy response to inference request in 2.1524040699005127s
Received healthy response to inference request in 2.141620635986328s
Received healthy response to inference request in 2.0960922241210938s
Received healthy response to inference request in 2.29460072517395s
2026-03-27T08:40:39.707367+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Received healthy response to inference request in 2.8463664054870605s
Received healthy response to inference request in 2.7904272079467773s
Received healthy response to inference request in 2.1949307918548584s
Received healthy response to inference request in 2.3021576404571533s
Received healthy response to inference request in 2.2721035480499268s
Received healthy response to inference request in 2.2221686840057373s
Received healthy response to inference request in 2.082716464996338s
Received healthy response to inference request in 2.184131622314453s
Received healthy response to inference request in 2.3050320148468018s
Received healthy response to inference request in 2.1360812187194824s
Received healthy response to inference request in 2.1146388053894043s
Received healthy response to inference request in 2.1970412731170654s
Received healthy response to inference request in 2.220491886138916s
2026-03-27T08:41:25.157797+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Received healthy response to inference request in 2.190171003341675s
30 requests
0 failed requests
5th percentile: 2.1044381856918335
10th percentile: 2.1208632469177244
20th percentile: 2.140512752532959
30th percentile: 2.1809309244155886
40th percentile: 2.1946326732635497
50th percentile: 2.2068302631378174
60th percentile: 2.2211626052856444
70th percentile: 2.2968677997589113
80th percentile: 2.8016150474548343
90th percentile: 8.646384000778198
95th percentile: 9.135786163806914
99th percentile: 9.447584784030914
mean time: 3.363116153081258
Pipeline stage StressChecker completed in 125.90s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_46797_v2 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-chai-_46797_v2 status is now inactive due to auto deactivation removed underperforming models