Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kaniwara-japan-fut-433-v5-uploader
Waiting for job on chaiml-kaniwara-japan-fut-433-v5-uploader to finish
chaiml-kaniwara-japan-fut-433-v5-uploader: Using quantization_mode: fp8
chaiml-kaniwara-japan-fut-433-v5-uploader: Checking if ChaiML/Kaniwara-Japan-Future-RPG260223150837_sft-FP8 already exists in ChaiML
chaiml-kaniwara-japan-fut-433-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kaniwara-japan-fut-433-v5-uploader: Downloading snapshot of ChaiML/Kaniwara-Japan-Future-RPG260223150837_sft-FP8...
chaiml-kaniwara-japan-fut-433-v5-uploader: Downloaded in 12.863s
chaiml-kaniwara-japan-fut-433-v5-uploader: Processed model ChaiML/Kaniwara-Japan-Future-RPG260223150837_sft in 16.591s
chaiml-kaniwara-japan-fut-433-v5-uploader: creating bucket guanaco-vllm-models
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kaniwara-japan-fut-433-v5-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kaniwara-japan-fut-433-v5-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kaniwara-japan-fut-433-v5-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kaniwara-japan-fut-433-v5-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kaniwara-japan-fut-433-v5-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kaniwara-japan-fut-433-v5-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kaniwara-japan-fut-433-v5-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kaniwara-japan-fut-433-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kaniwara-japan-fut-433-v5-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kaniwara-japan-fut-433-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kaniwara-japan-fut-433-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/model.safetensors.index.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/config.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/recipe.yaml
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/.gitattributes
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/tokenizer_config.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/generation_config.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/special_tokens_map.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/tokenizer.json
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/model-00006-of-00006.safetensors s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/model-00006-of-00006.safetensors
chaiml-kaniwara-japan-fut-433-v5-uploader: cp /dev/shm/model_output/model-00005-of-00006.safetensors s3://guanaco-vllm-models/chaiml-kaniwara-japan-fut-433-v5/default/model-00005-of-00006.safetensors
Job chaiml-kaniwara-japan-fut-433-v5-uploader completed after 76.51s with status: succeeded
Stopping job with name chaiml-kaniwara-japan-fut-433-v5-uploader
Pipeline stage VLLMUploader completed in 78.31s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.00s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kaniwara-japan-fut-433-v5
Waiting for inference service chaiml-kaniwara-japan-fut-433-v5 to be ready
Inference service chaiml-kaniwara-japan-fut-433-v5 ready after 162.79648756980896s
Pipeline stage VLLMDeployer completed in 164.22s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.30063533782959s
Received healthy response to inference request in 3.0037479400634766s
Received healthy response to inference request in 3.085245370864868s
Received healthy response to inference request in 2.9414772987365723s
Received healthy response to inference request in 2.892991065979004s
Received healthy response to inference request in 3.0539848804473877s
Received healthy response to inference request in 2.9610683917999268s
Received healthy response to inference request in 3.0787484645843506s
Received healthy response to inference request in 2.883024215698242s
Received healthy response to inference request in 2.932652235031128s
Received healthy response to inference request in 3.014549970626831s
Received healthy response to inference request in 3.2349801063537598s
Received healthy response to inference request in 2.811004877090454s
Received healthy response to inference request in 2.9117953777313232s
Received healthy response to inference request in 2.8411266803741455s
Received healthy response to inference request in 2.950253486633301s
Received healthy response to inference request in 2.8527371883392334s
Received healthy response to inference request in 2.8362371921539307s
Received healthy response to inference request in 2.7695462703704834s
Received healthy response to inference request in 2.8803579807281494s
Received healthy response to inference request in 2.9372222423553467s
Received healthy response to inference request in 2.877025604248047s
Received healthy response to inference request in 2.761472463607788s
Received healthy response to inference request in 3.6812517642974854s
Received healthy response to inference request in 2.7914223670959473s
Received healthy response to inference request in 2.7554452419281006s
Received healthy response to inference request in 2.957242012023926s
Received healthy response to inference request in 3.008432388305664s
Received healthy response to inference request in 2.8415496349334717s
Received healthy response to inference request in 3.217308282852173s
30 requests
0 failed requests
5th percentile: 2.765105676651001
10th percentile: 2.789234757423401
20th percentile: 2.8401487827301026
30th percentile: 2.8697390794754027
40th percentile: 2.8890043258666993
50th percentile: 2.9349372386932373
60th percentile: 2.953048896789551
70th percentile: 3.005153274536133
80th percentile: 3.0589375972747805
90th percentile: 3.2190754652023315
95th percentile: 3.271090483665466
99th percentile: 3.570873000621796
mean time: 2.96881787776947
Pipeline stage StressChecker completed in 94.06s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.25s
Shutdown handler de-registered
chaiml-kaniwara-japan-fut_433_v5 status is now deployed due to DeploymentManager action
chaiml-kaniwara-japan-fut_433_v5 status is now inactive due to system request