Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v36-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v36-uploader to finish
chaiml-kimid-v9-opusdv-23365-v36-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v36-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v36-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v36-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
2026-04-02T17:56:07.972196+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
2026-04-02T17:57:08.060990+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-kimid-v9-opusdv-23365-v36-uploader: Downloaded in 105.278s
chaiml-kimid-v9-opusdv-23365-v36-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 108.129s
chaiml-kimid-v9-opusdv-23365-v36-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v36-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v36-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v36-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v36-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v36-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v36-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v36-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v36-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v36-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v36-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v36-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/config.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model.safetensors.index.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00013-of-00027.safetensors
2026-04-02T17:58:08.139668+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v36-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v36/default/model-00024-of-00027.safetensors
Connection pool is full, discarding connection: %s. Connection pool size: %s
Job chaiml-kimid-v9-opusdv-23365-v36-uploader completed after 216.75s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v36-uploader
Pipeline stage VLLMUploader completed in 217.20s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.85s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v36
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v36 to be ready
2026-04-02T17:59:08.236150+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
2026-04-02T18:00:08.352830+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
2026-04-02T18:01:08.466587+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
Inference service chaiml-kimid-v9-opusdv-23365-v36 ready after 181.30536103248596s
Pipeline stage VLLMDeployer completed in 181.73s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.512707471847534s
Received healthy response to inference request in 1.4799420833587646s
Received healthy response to inference request in 2.1834018230438232s
Received healthy response to inference request in 1.7169060707092285s
Received healthy response to inference request in 1.5585172176361084s
Received healthy response to inference request in 1.6014971733093262s
Received healthy response to inference request in 1.7295775413513184s
Received healthy response to inference request in 1.495145559310913s
Received healthy response to inference request in 1.4705944061279297s
Received healthy response to inference request in 2.0136020183563232s
2026-04-02T18:02:08.578245+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v36
Received healthy response to inference request in 1.4548931121826172s
Received healthy response to inference request in 1.9281485080718994s
Received healthy response to inference request in 1.4499762058258057s
Received healthy response to inference request in 1.7446539402008057s
Received healthy response to inference request in 1.6223340034484863s
Received healthy response to inference request in 1.5328919887542725s
Received healthy response to inference request in 1.491767168045044s
Received healthy response to inference request in 1.6040868759155273s
Received healthy response to inference request in 1.4762728214263916s
Received healthy response to inference request in 1.5602376461029053s
Received healthy response to inference request in 1.5998599529266357s
Received healthy response to inference request in 1.4777908325195312s
Received healthy response to inference request in 1.5283710956573486s
Received healthy response to inference request in 1.6405246257781982s
Received healthy response to inference request in 1.6743524074554443s
Received healthy response to inference request in 1.8396461009979248s
Received healthy response to inference request in 1.4535610675811768s
Received healthy response to inference request in 1.7914152145385742s
Received healthy response to inference request in 1.6244502067565918s
Received healthy response to inference request in 1.7760188579559326s
30 requests
0 failed requests
5th percentile: 1.454160487651825
10th percentile: 1.4690242767333985
20th percentile: 1.479511833190918
30th percentile: 1.5184034347534179
40th percentile: 1.5595494747161864
50th percentile: 1.6027920246124268
60th percentile: 1.6308799743652342
70th percentile: 1.7207075119018553
80th percentile: 1.779098129272461
90th percentile: 1.936693859100342
95th percentile: 2.1069919109344477
99th percentile: 3.1272088336944592
mean time: 1.7011047999064128
Pipeline stage StressChecker completed in 54.18s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.29s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v36 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v36 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_23365_v36 status is now torndown due to DeploymentManager action