Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-16773-v3-uploader
Waiting for job on chaiml-gspo-glm47-combi-16773-v3-uploader to finish
chaiml-gspo-glm47-combi-16773-v3-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-16773-v3-uploader: Checking if ChaiML/gspo-glm47-combine-rm721-mega-data-step1000-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-16773-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm47-combi-16773-v3-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm721-mega-data-step1000-W4A16...
2026-04-02T23:23:45.729359+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
2026-04-02T23:24:45.815688+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
chaiml-gspo-glm47-combi-16773-v3-uploader: Downloaded in 83.202s
chaiml-gspo-glm47-combi-16773-v3-uploader: Processed model ChaiML/gspo-glm47-combine-rm721-mega-data-step1000 in 85.870s
chaiml-gspo-glm47-combi-16773-v3-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-16773-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-16773-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-16773-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-16773-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-16773-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-16773-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-16773-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/config.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/generation_config.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/.gitattributes
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/tokenizer_config.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/chat_template.jinja
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/quantization_config.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/tokenizer.json
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-16773-v3-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-16773-v3/default/model-00036-of-00038.safetensors
2026-04-02T23:25:46.081107+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
Job chaiml-gspo-glm47-combi-16773-v3-uploader completed after 184.85s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-16773-v3-uploader
Pipeline stage VLLMUploader completed in 185.80s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.24s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-16773-v3
Waiting for inference service chaiml-gspo-glm47-combi-16773-v3 to be ready
2026-04-02T23:26:46.211000+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
2026-04-02T23:27:46.314282+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
2026-04-02T23:28:46.603700+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
2026-04-02T23:29:46.901981+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
2026-04-02T23:30:47.083924+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
Inference service chaiml-gspo-glm47-combi-16773-v3 ready after 310.84182929992676s
Pipeline stage VLLMDeployer completed in 311.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.536298513412476s
Received healthy response to inference request in 8.994609594345093s
Received healthy response to inference request in 8.539863348007202s
Received healthy response to inference request in 8.779015064239502s
Received healthy response to inference request in 2.104154109954834s
2026-04-02T23:31:47.510662+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
Received healthy response to inference request in 12.014469623565674s
Received healthy response to inference request in 2.0919785499572754s
Received healthy response to inference request in 2.000647783279419s
Received healthy response to inference request in 2.012080669403076s
Received healthy response to inference request in 2.0260181427001953s
Received healthy response to inference request in 2.047297477722168s
Received healthy response to inference request in 2.002960443496704s
Received healthy response to inference request in 2.022059679031372s
Received healthy response to inference request in 2.082610607147217s
Received healthy response to inference request in 2.1247589588165283s
Received healthy response to inference request in 2.099055528640747s
Received healthy response to inference request in 2.222813844680786s
Received healthy response to inference request in 2.0519931316375732s
Received healthy response to inference request in 1.943185806274414s
Received healthy response to inference request in 1.9963409900665283s
Received healthy response to inference request in 2.044396162033081s
Received healthy response to inference request in 2.376878023147583s
Received healthy response to inference request in 2.103773832321167s
Received healthy response to inference request in 2.4609766006469727s
Received healthy response to inference request in 2.0339131355285645s
Received healthy response to inference request in 2.014094352722168s
Received healthy response to inference request in 2.088749647140503s
Received healthy response to inference request in 2.0842185020446777s
Received healthy response to inference request in 2.1432509422302246s
2026-04-02T23:32:47.948671+00:00 monitor updated for chaiml-gspo-glm47-combi_16773_v3
Received healthy response to inference request in 2.221508741378784s
30 requests
0 failed requests
5th percentile: 1.998279047012329
10th percentile: 2.0027291774749756
20th percentile: 2.020466613769531
30th percentile: 2.0412512540817263
40th percentile: 2.0703636169433595
50th percentile: 2.090364098548889
60th percentile: 2.103925943374634
70th percentile: 2.1667282819747924
80th percentile: 2.3936977386474614
90th percentile: 8.80057451725006
95th percentile: 9.842538499832148
99th percentile: 11.585800001621248
mean time: 3.375465726852417
Pipeline stage StressChecker completed in 104.06s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.93s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_16773_v3 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_16773_v3 status is now inactive due to auto deactivation removed underperforming models