Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-combi-87059-v3-uploader
Waiting for job on chaiml-gspo-glm47-combi-87059-v3-uploader to finish
chaiml-gspo-glm47-combi-87059-v3-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-combi-87059-v3-uploader: Checking if ChaiML/gspo-glm47-combine-rm82-kimi-data-step500-W4A16 already exists in ChaiML
chaiml-gspo-glm47-combi-87059-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-gspo-glm47-combi-87059-v3-uploader: Downloading snapshot of ChaiML/gspo-glm47-combine-rm82-kimi-data-step500-W4A16...
2026-03-27T14:49:10.196717+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
2026-03-27T14:50:10.391933+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
chaiml-gspo-glm47-combi-87059-v3-uploader: Downloaded in 75.588s
chaiml-gspo-glm47-combi-87059-v3-uploader: Processed model ChaiML/gspo-glm47-combine-rm82-kimi-data-step500 in 78.583s
chaiml-gspo-glm47-combi-87059-v3-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-combi-87059-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-combi-87059-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-combi-87059-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-combi-87059-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-combi-87059-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-combi-87059-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-combi-87059-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/generation_config.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/config.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/chat_template.jinja
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/quantization_config.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/tokenizer_config.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/.gitattributes
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model.safetensors.index.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/tokenizer.json
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00038-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00033-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00033-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00008-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00008-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00031-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00017-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00017-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00006-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00006-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00027-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00027-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-combi-87059-v3-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-combi-87059-v3/default/model-00029-of-00038.safetensors
2026-03-27T14:51:10.599466+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
Job chaiml-gspo-glm47-combi-87059-v3-uploader completed after 184.95s with status: succeeded
Stopping job with name chaiml-gspo-glm47-combi-87059-v3-uploader
Pipeline stage VLLMUploader completed in 186.44s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.35s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.07s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-combi-87059-v3
Waiting for inference service chaiml-gspo-glm47-combi-87059-v3 to be ready
2026-03-27T14:52:10.831302+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
2026-03-27T14:53:11.113902+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
2026-03-27T14:54:11.482324+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
2026-03-27T14:55:11.701933+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
Inference service chaiml-gspo-glm47-combi-87059-v3 ready after 283.7856628894806s
Pipeline stage VLLMDeployer completed in 285.02s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-27T14:56:11.915839+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
Received healthy response to inference request in 9.338301658630371s
Received healthy response to inference request in 2.439791440963745s
Received healthy response to inference request in 8.794272422790527s
Received healthy response to inference request in 2.4011788368225098s
Received healthy response to inference request in 2.2216272354125977s
Received healthy response to inference request in 9.445435762405396s
Received healthy response to inference request in 2.210151195526123s
Received healthy response to inference request in 2.2261569499969482s
Received healthy response to inference request in 2.11649751663208s
Received healthy response to inference request in 8.826252460479736s
Received healthy response to inference request in 2.0579397678375244s
Received healthy response to inference request in 2.2808027267456055s
Received healthy response to inference request in 2.1619555950164795s
Received healthy response to inference request in 2.3353703022003174s
Received healthy response to inference request in 2.0629055500030518s
Received healthy response to inference request in 2.1866886615753174s
Received healthy response to inference request in 2.355041980743408s
2026-03-27T14:57:12.270059+00:00 monitor updated for chaiml-gspo-glm47-combi_87059_v3
Received healthy response to inference request in 2.1945512294769287s
Received healthy response to inference request in 2.4422993659973145s
Received healthy response to inference request in 2.134225368499756s
Received healthy response to inference request in 2.238474130630493s
Received healthy response to inference request in 2.2546887397766113s
Received healthy response to inference request in 2.1992757320404053s
Received healthy response to inference request in 2.2783360481262207s
Received healthy response to inference request in 2.2854959964752197s
Received healthy response to inference request in 2.131878137588501s
Received healthy response to inference request in 2.141369342803955s
Received healthy response to inference request in 2.1910853385925293s
Received healthy response to inference request in 2.1843857765197754s
Received healthy response to inference request in 2.2643229961395264s
30 requests
0 failed requests
5th percentile: 2.0870219349861143
10th percentile: 2.130340075492859
20th percentile: 2.1578383445739746
30th percentile: 2.1897663354873655
40th percentile: 2.205801010131836
50th percentile: 2.2323155403137207
60th percentile: 2.269928216934204
70th percentile: 2.3004582881927487
80th percentile: 2.4089013576507567
90th percentile: 8.797470426559448
95th percentile: 9.107879519462584
99th percentile: 9.414366872310639
mean time: 3.1466919422149657
Pipeline stage StressChecker completed in 100.89s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.22s
Shutdown handler de-registered
chaiml-gspo-glm47-combi_87059_v3 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-combi_87059_v3 status is now inactive due to auto deactivation removed underperforming models