developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-cas72_44480_v2
model_name: chaiml-gspo-glm47-cas72_44480_v2
model_group: ChaiML/gspo-glm47-cas72b
status: inactive
timestamp: 2026-03-27T10:47:32+00:00
num_battles: 11526
num_wins: 6736
celo_rating: 1354.28
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-cas72_44480_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-27
win_ratio: 0.5844178379316328
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '####', '<|assistant|>', '<|user|>', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>\n{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>\n{bot_name}: {message}', 'user_template': '<|user|>\n{message}', 'response_template': '<|assistant|>\n<think></think>\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-cas72-44480-v2-uploader
Waiting for job on chaiml-gspo-glm47-cas72-44480-v2-uploader to finish
2026-03-27T08:32:36.967872+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader:                              







chaiml-gspo-glm47-chai-46797-v2-uploader: ---------- 2026-03-27 08:32:38 (0:02:00) ----------
chaiml-gspo-glm47-chai-46797-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 39/40 (192.0G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-chai-46797-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 63
chaiml-gspo-glm47-chai-46797-v2-uploader: ---------------------------------------------------
chaiml-gspo-glm47-cas72-44480-v2-uploader: Checking if ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-cas72-44480-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500...
2026-03-27T08:33:21.788820+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:33:37.291939+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
chaiml-gspo-glm47-chai-46797-v2-uploader:                              Processed model ChaiML/gspo-glm47-chai-rm-kimi-data-step500 in 1829.749s
chaiml-gspo-glm47-chai-46797-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-chai-46797-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-chai-46797-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-chai-46797-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-chai-46797-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-chai-46797-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-chai-46797-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-chai-46797-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/chat_template.jinja
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/quantization_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/generation_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/tokenizer_config.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/tokenizer.json
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00037-of-00038.safetensors
2026-03-27T08:34:22.109526+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-chai-46797-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-chai-46797-v2/default/model-00023-of-00038.safetensors
2026-03-27T08:34:37.638139+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Job chaiml-gspo-glm47-chai-46797-v2-uploader completed after 1945.09s with status: succeeded
Stopping job with name chaiml-gspo-glm47-chai-46797-v2-uploader
Pipeline stage VLLMUploader completed in 1946.71s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.48s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.08s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-chai-46797-v2
Waiting for inference service chaiml-gspo-glm47-chai-46797-v2 to be ready
2026-03-27T08:35:22.441630+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:35:37.963003+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:36:22.868420+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:36:38.311934+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:37:23.269917+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: Downloaded in 247.189s
2026-03-27T08:37:38.644343+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:38:23.690007+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T08:38:38.985718+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
2026-03-27T08:39:24.097290+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Inference service chaiml-gspo-glm47-chai-46797-v2 ready after 273.74899220466614s
Pipeline stage VLLMDeployer completed in 275.71s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.624801635742188s
Received healthy response to inference request in 2.2115416526794434s
2026-03-27T08:39:39.321139+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Received healthy response to inference request in 9.377281427383423s
Received healthy response to inference request in 2.194185495376587s
Received healthy response to inference request in 2.1215548515319824s
Received healthy response to inference request in 8.59005093574524s
Received healthy response to inference request in 8.840625286102295s
Received healthy response to inference request in 2.1734626293182373s
Received healthy response to inference request in 2.209822416305542s
Received healthy response to inference request in 2.2038381099700928s
Received healthy response to inference request in 2.1268436908721924s
2026-03-27T08:40:24.810769+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Received healthy response to inference request in 9.476300239562988s
Received healthy response to inference request in 2.1524040699005127s
Received healthy response to inference request in 2.141620635986328s
Received healthy response to inference request in 2.0960922241210938s
Received healthy response to inference request in 2.29460072517395s
2026-03-27T08:40:39.707367+00:00 monitor updated for chaiml-gspo-glm47-chai-_46797_v2
Received healthy response to inference request in 2.8463664054870605s
Received healthy response to inference request in 2.7904272079467773s
Received healthy response to inference request in 2.1949307918548584s
Received healthy response to inference request in 2.3021576404571533s
Received healthy response to inference request in 2.2721035480499268s
Received healthy response to inference request in 2.2221686840057373s
Received healthy response to inference request in 2.082716464996338s
Received healthy response to inference request in 2.184131622314453s
Received healthy response to inference request in 2.3050320148468018s
Received healthy response to inference request in 2.1360812187194824s
Received healthy response to inference request in 2.1146388053894043s
Received healthy response to inference request in 2.1970412731170654s
Received healthy response to inference request in 2.220491886138916s
2026-03-27T08:41:25.157797+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Received healthy response to inference request in 2.190171003341675s
30 requests
0 failed requests
5th percentile: 2.1044381856918335
10th percentile: 2.1208632469177244
20th percentile: 2.140512752532959
30th percentile: 2.1809309244155886
40th percentile: 2.1946326732635497
50th percentile: 2.2068302631378174
60th percentile: 2.2211626052856444
70th percentile: 2.2968677997589113
80th percentile: 2.8016150474548343
90th percentile: 8.646384000778198
95th percentile: 9.135786163806914
99th percentile: 9.447584784030914
mean time: 3.363116153081258
Pipeline stage StressChecker completed in 125.90s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-gspo-glm47-chai-_46797_v2 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-cas72-44480-v2-uploader: Applying quantization...
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:41:51 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE
2026-03-27T08:42:25.565758+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:42:29 INFO base.py L486: using torch.bfloat16 for quantization tuning
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:42:48 INFO device.py L1468:  'peak_ram': 11.91GB, 'peak_vram': 1.44GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:43:01 INFO device.py L1468:  'peak_ram': 16.36GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:43:18 INFO device.py L1468:  'peak_ram': 16.94GB, 'peak_vram': 1.59GB
2026-03-27T08:43:25.761955+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:43:29 INFO device.py L1468:  'peak_ram': 19.36GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:43:44 INFO device.py L1468:  'peak_ram': 19.81GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:43:56 INFO device.py L1468:  'peak_ram': 22.23GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:06 INFO device.py L1468:  'peak_ram': 22.23GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:20 INFO device.py L1468:  'peak_ram': 22.77GB, 'peak_vram': 1.59GB
2026-03-27T08:44:25.979660+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:28 INFO device.py L1468:  'peak_ram': 22.77GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:36 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:40 INFO device.py L1468:  'peak_ram': 23.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:44 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:51 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:44:59 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:06 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:13 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:17 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:23 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
2026-03-27T08:45:26.222449+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:30 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:41 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:45:55 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:46:09 INFO device.py L1468:  'peak_ram': 23.66GB, 'peak_vram': 1.59GB
2026-03-27T08:46:26.433200+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:46:25 INFO device.py L1468:  'peak_ram': 24.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:46:41 INFO device.py L1468:  'peak_ram': 24.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:46:56 INFO device.py L1468:  'peak_ram': 24.87GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:47:13 INFO device.py L1468:  'peak_ram': 24.87GB, 'peak_vram': 1.59GB
2026-03-27T08:47:26.789965+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:47:24 INFO device.py L1468:  'peak_ram': 25.86GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:47:36 INFO device.py L1468:  'peak_ram': 25.86GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:47:50 INFO device.py L1468:  'peak_ram': 26.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:48:01 INFO device.py L1468:  'peak_ram': 26.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:48:15 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
2026-03-27T08:48:27.005939+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:48:26 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:48:37 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:48:51 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:49:02 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:49:12 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
2026-03-27T08:49:27.223106+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:49:27 INFO device.py L1468:  'peak_ram': 27.4GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:49:37 INFO device.py L1468:  'peak_ram': 27.61GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:50:02 INFO device.py L1468:  'peak_ram': 28.27GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:50:13 INFO device.py L1468:  'peak_ram': 28.27GB, 'peak_vram': 1.59GB
2026-03-27T08:50:27.436756+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:50:28 INFO device.py L1468:  'peak_ram': 29.34GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:50:38 INFO device.py L1468:  'peak_ram': 29.34GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:50:53 INFO device.py L1468:  'peak_ram': 30.22GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:51:18 INFO device.py L1468:  'peak_ram': 30.6GB, 'peak_vram': 1.59GB
2026-03-27T08:51:29.923242+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:51:33 INFO device.py L1468:  'peak_ram': 30.6GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:51:44 INFO device.py L1468:  'peak_ram': 31.57GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:51:55 INFO device.py L1468:  'peak_ram': 31.57GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:52:08 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:52:18 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
2026-03-27T08:52:30.145647+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:52:32 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:52:42 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:52:50 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:53:03 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:53:14 INFO device.py L1468:  'peak_ram': 32.46GB, 'peak_vram': 1.59GB
2026-03-27T08:53:30.372021+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:53:28 INFO device.py L1468:  'peak_ram': 33.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:53:38 INFO device.py L1468:  'peak_ram': 33.31GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:53:49 INFO device.py L1468:  'peak_ram': 33.68GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:54:03 INFO device.py L1468:  'peak_ram': 33.68GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:54:14 INFO device.py L1468:  'peak_ram': 34.34GB, 'peak_vram': 1.59GB
2026-03-27T08:54:30.645851+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:54:26 INFO device.py L1468:  'peak_ram': 34.34GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:54:40 INFO device.py L1468:  'peak_ram': 35.37GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:54:52 INFO device.py L1468:  'peak_ram': 35.37GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:55:06 INFO device.py L1468:  'peak_ram': 36.5GB, 'peak_vram': 1.59GB
2026-03-27T08:55:30.903207+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:55:33 INFO device.py L1468:  'peak_ram': 37.0GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:55:47 INFO device.py L1468:  'peak_ram': 37.0GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:55:59 INFO device.py L1468:  'peak_ram': 37.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:56:12 INFO device.py L1468:  'peak_ram': 37.82GB, 'peak_vram': 1.59GB
2026-03-27T08:56:31.814075+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:56:26 INFO device.py L1468:  'peak_ram': 37.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:56:37 INFO device.py L1468:  'peak_ram': 37.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:56:52 INFO device.py L1468:  'peak_ram': 37.82GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:57:04 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:57:15 INFO device.py L1468:  'peak_ram': 38.11GB, 'peak_vram': 1.59GB
2026-03-27T08:57:32.072684+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:57:30 INFO device.py L1468:  'peak_ram': 39.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:57:41 INFO device.py L1468:  'peak_ram': 39.41GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:57:56 INFO device.py L1468:  'peak_ram': 40.58GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:58:06 INFO device.py L1468:  'peak_ram': 40.58GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:58:18 INFO device.py L1468:  'peak_ram': 41.13GB, 'peak_vram': 1.59GB
2026-03-27T08:58:32.328416+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:58:34 INFO device.py L1468:  'peak_ram': 41.13GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:58:46 INFO device.py L1468:  'peak_ram': 41.87GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:58:57 INFO device.py L1468:  'peak_ram': 41.87GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:59:10 INFO device.py L1468:  'peak_ram': 42.97GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:59:21 INFO device.py L1468:  'peak_ram': 42.97GB, 'peak_vram': 1.59GB
2026-03-27T08:59:32.591155+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:59:35 INFO device.py L1468:  'peak_ram': 44.22GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:59:46 INFO device.py L1468:  'peak_ram': 44.22GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 08:59:58 INFO device.py L1468:  'peak_ram': 45.08GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:00:13 INFO device.py L1468:  'peak_ram': 45.08GB, 'peak_vram': 1.59GB
2026-03-27T09:00:32.817620+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:00:26 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:00:41 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:00:55 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:01:03 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:01:03 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict
chaiml-gspo-glm47-cas72-44480-v2-uploader: 2026-03-27 09:01:03 INFO device.py L1468:  'peak_ram': 45.35GB, 'peak_vram': 1.59GB
chaiml-gspo-glm47-cas72-44480-v2-uploader: Checking if ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-cas72-44480-v2-uploader: Creating repo ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------- 2026-03-27 09:01:04 (0:00:00) ----------
chaiml-gspo-glm47-cas72-44480-v2-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+45 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-cas72-44480-v2-uploader: Workers: hashing: 38 | get upload mode: 2 | pre-uploading: 0 | committing: 0 | waiting: 24
chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------------------------------------------------
2026-03-27T09:01:33.174782+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader: 







chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------- 2026-03-27 09:02:04 (0:01:00) ----------
chaiml-gspo-glm47-cas72-44480-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 14/40 (57.9G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-cas72-44480-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 26 | committing: 0 | waiting: 38
chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------------------------------------------------
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-glm-47-bobo-v1-s-81799-v4-uploader
Waiting for job on chaiml-glm-47-bobo-v1-s-81799-v4-uploader to finish
2026-03-27T09:02:33.445677+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader:                              







chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------- 2026-03-27 09:03:04 (0:02:00) ----------
chaiml-gspo-glm47-cas72-44480-v2-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 40/40 (197.4G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-cas72-44480-v2-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 63
chaiml-gspo-glm47-cas72-44480-v2-uploader: ---------------------------------------------------
chaiml-glm-47-bobo-v1-s-81799-v4-uploader: Checking if ChaiML/glm_47_bobo_v1-step2498-merged-W4A16 already exists in ChaiML
chaiml-glm-47-bobo-v1-s-81799-v4-uploader: Downloading snapshot of ChaiML/glm_47_bobo_v1-step2498-merged...
2026-03-27T09:03:24.720954+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
2026-03-27T09:03:33.764481+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
chaiml-gspo-glm47-cas72-44480-v2-uploader:                              Processed model ChaiML/gspo-glm47-cas72b-combine-rm82-kimi-data-step500 in 1829.414s
chaiml-gspo-glm47-cas72-44480-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-cas72-44480-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-cas72-44480-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-cas72-44480-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-cas72-44480-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/config.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/generation_config.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/tokenizer_config.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/quantization_config.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/chat_template.jinja
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/tokenizer.json
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00013-of-00038.safetensors
2026-03-27T09:04:25.046901+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-cas72-44480-v2-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-cas72-44480-v2/default/model-00029-of-00038.safetensors
2026-03-27T09:04:34.112572+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Job chaiml-gspo-glm47-cas72-44480-v2-uploader completed after 1937.91s with status: succeeded
Stopping job with name chaiml-gspo-glm47-cas72-44480-v2-uploader
Pipeline stage VLLMUploader completed in 1939.74s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.47s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.12s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-cas72-44480-v2
Waiting for inference service chaiml-gspo-glm47-cas72-44480-v2 to be ready
2026-03-27T09:05:25.666087+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
2026-03-27T09:05:34.498722+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T09:06:25.985508+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
2026-03-27T09:06:34.862914+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T09:07:26.343753+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
chaiml-glm-47-bobo-v1-s-81799-v4-uploader: Downloaded in 250.210s
2026-03-27T09:07:35.241072+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T09:08:26.716014+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
2026-03-27T09:08:35.622514+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Inference service chaiml-gspo-glm47-cas72-44480-v2 ready after 272.91880679130554s
Pipeline stage VLLMDeployer completed in 274.71s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-27T09:09:27.187995+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
Received healthy response to inference request in 8.637159824371338s
2026-03-27T09:09:36.000117+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
Received healthy response to inference request in 9.028732776641846s
Received healthy response to inference request in 2.433098554611206s
Received healthy response to inference request in 2.089993476867676s
Received healthy response to inference request in 9.254623889923096s
Received healthy response to inference request in 2.350205898284912s
Received healthy response to inference request in 2.3044228553771973s
Received healthy response to inference request in 8.839781999588013s
Received healthy response to inference request in 2.1314573287963867s
Received healthy response to inference request in 2.3130152225494385s
Received healthy response to inference request in 2.1179702281951904s
Received healthy response to inference request in 2.1941018104553223s
Received healthy response to inference request in 2.717045783996582s
Received healthy response to inference request in 2.1322731971740723s
Received healthy response to inference request in 2.0935580730438232s
Received healthy response to inference request in 2.1006979942321777s
2026-03-27T09:10:36.392302+00:00 monitor updated for chaiml-gspo-glm47-cas72_44480_v2
2026-03-27T09:10:27.595065+00:00 monitor updated for chaiml-glm-47-bobo-v1-s_81799_v4
Received healthy response to inference request in 2.1885313987731934s
Received healthy response to inference request in 2.285914182662964s
Received healthy response to inference request in 2.607898235321045s
Received healthy response to inference request in 2.1671738624572754s
Received healthy response to inference request in 2.2119898796081543s
Received healthy response to inference request in 2.214454412460327s
Received healthy response to inference request in 2.429957151412964s
Received healthy response to inference request in 2.2102766036987305s
Received healthy response to inference request in 2.207002878189087s
Received healthy response to inference request in 2.302797555923462s
Received healthy response to inference request in 2.1994123458862305s
Received healthy response to inference request in 2.0917775630950928s
Received healthy response to inference request in 2.1471688747406006s
Received healthy response to inference request in 2.1244988441467285s
30 requests
0 failed requests
5th percentile: 2.0925787925720214
10th percentile: 2.0999840021133425
20th percentile: 2.1300656318664553
30th percentile: 2.161172366142273
40th percentile: 2.1972881317138673
50th percentile: 2.2111332416534424
60th percentile: 2.292667531967163
70th percentile: 2.3241724252700804
80th percentile: 2.468058490753174
90th percentile: 8.657422041893005
95th percentile: 8.94370492696762
99th percentile: 9.189115467071533
mean time: 3.137566423416138
Pipeline stage StressChecker completed in 118.42s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.89s
Shutdown handler de-registered
chaiml-gspo-glm47-cas72_44480_v2 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-cas72_44480_v2 status is now inactive due to auto deactivation removed underperforming models