Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-kimi-25686-v2-uploader
Waiting for job on chaiml-gspo-glm47-kimi-25686-v2-uploader to finish
chaiml-gspo-glm47-kimi-25686-v2-uploader: Using quantization_mode: fp8
chaiml-gspo-glm47-kimi-25686-v2-uploader: Checking if ChaiML/gspo-glm47-kimi-rm-kimi-mega-pony-euqal-apr02-step400-FP8 already exists in ChaiML
chaiml-gspo-glm47-kimi-25686-v2-uploader: Model already exists. Downloading to /tmp/model_output...
chaiml-gspo-glm47-kimi-25686-v2-uploader: Downloading snapshot of ChaiML/gspo-glm47-kimi-rm-kimi-mega-pony-euqal-apr02-step400-FP8...
2026-04-08T15:04:06.566902+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:05:06.678920+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:06:06.823153+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:07:06.952265+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
chaiml-gspo-glm47-kimi-25686-v2-uploader: Downloaded in 204.368s
chaiml-gspo-glm47-kimi-25686-v2-uploader: Processed model ChaiML/gspo-glm47-kimi-rm-kimi-mega-pony-euqal-apr02-step400 in 207.382s
chaiml-gspo-glm47-kimi-25686-v2-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-25686-v2-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-kimi-25686-v2-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-25686-v2-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-25686-v2-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-25686-v2-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-25686-v2-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-kimi-25686-v2-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-kimi-25686-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-kimi-25686-v2-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-kimi-25686-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-kimi-25686-v2-uploader: uploading /tmp/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/chat_template.jinja
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/generation_config.json
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/config.json
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/.gitattributes
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/recipe.yaml
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/tokenizer_config.json
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model.safetensors.index.json
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/tokenizer.json
2026-04-08T15:08:07.061958+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:09:07.168188+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00015-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00015-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00014-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00014-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00013-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00013-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00012-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00012-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00009-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00009-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00010-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00010-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00011-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00011-of-00015.safetensors
2026-04-08T15:10:07.281243+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00004-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00004-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00005-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00005-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00001-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00001-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00007-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00007-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00002-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00002-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00003-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00003-of-00015.safetensors
chaiml-gspo-glm47-kimi-25686-v2-uploader: cp /tmp/model_output/model-00008-of-00015.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-25686-v2/default/model-00008-of-00015.safetensors
2026-04-08T15:11:07.393542+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
Job chaiml-gspo-glm47-kimi-25686-v2-uploader completed after 480.67s with status: succeeded
Stopping job with name chaiml-gspo-glm47-kimi-25686-v2-uploader
Pipeline stage VLLMUploader completed in 481.18s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.88s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-kimi-25686-v2
Waiting for inference service chaiml-gspo-glm47-kimi-25686-v2 to be ready
2026-04-08T15:12:07.492766+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:13:07.600720+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:14:07.705991+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:15:07.811886+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:16:07.917494+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:17:08.024220+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:18:08.132279+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
2026-04-08T15:19:08.242719+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
Inference service chaiml-gspo-glm47-kimi-25686-v2 ready after 520.9560344219208s
Pipeline stage VLLMDeployer completed in 521.41s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.193167209625244s
2026-04-08T15:20:08.413485+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
Received healthy response to inference request in 9.378132820129395s
Received healthy response to inference request in 1.978132724761963s
Received healthy response to inference request in 2.2726104259490967s
Received healthy response to inference request in 1.9904265403747559s
Received healthy response to inference request in 9.522236824035645s
Received healthy response to inference request in 2.084469795227051s
Received healthy response to inference request in 2.08292555809021s
Received healthy response to inference request in 2.157146453857422s
Received healthy response to inference request in 2.0788142681121826s
Received healthy response to inference request in 2.3404006958007812s
Received healthy response to inference request in 1.997661828994751s
Received healthy response to inference request in 2.7669100761413574s
Received healthy response to inference request in 1.9854559898376465s
Received healthy response to inference request in 1.973935842514038s
Received healthy response to inference request in 2.380492925643921s
Received healthy response to inference request in 2.016134023666382s
Received healthy response to inference request in 2.000373601913452s
Received healthy response to inference request in 1.9885482788085938s
Received healthy response to inference request in 2.2444796562194824s
Received healthy response to inference request in 2.2607314586639404s
Received healthy response to inference request in 2.3236441612243652s
Received healthy response to inference request in 2.0089991092681885s
Received healthy response to inference request in 1.9814021587371826s
2026-04-08T15:21:08.534730+00:00 monitor updated for chaiml-gspo-glm47-kimi-_25686_v2
Received healthy response to inference request in 2.160130739212036s
Received healthy response to inference request in 2.2094733715057373s
Received healthy response to inference request in 2.0098366737365723s
Received healthy response to inference request in 2.167459487915039s
Received healthy response to inference request in 2.396700620651245s
Received healthy response to inference request in 2.087578773498535s
30 requests
0 failed requests
5th percentile: 1.9796039700508117
10th percentile: 1.9850506067276001
20th percentile: 1.9962147712707519
30th percentile: 2.009585404396057
40th percentile: 2.081281042098999
50th percentile: 2.1223626136779785
60th percentile: 2.1842650413513183
70th percentile: 2.264295148849487
80th percentile: 2.3484191417694094
90th percentile: 3.4095357894897553
95th percentile: 9.294898295402527
99th percentile: 9.480446662902832
mean time: 2.867947069803874
Pipeline stage StressChecker completed in 89.28s
Shutdown handler de-registered
chaiml-gspo-glm47-kimi-_25686_v2 status is now deployed due to DeploymentManager action
chaiml-gspo-glm47-kimi-_25686_v2 status is now inactive due to auto deactivation removed underperforming models