Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-23365-v34-uploader
Waiting for job on chaiml-kimid-v9-opusdv-23365-v34-uploader to finish
chaiml-kimid-v9-opusdv-23365-v34-uploader: Using quantization_mode: w4a16
chaiml-kimid-v9-opusdv-23365-v34-uploader: Checking if ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v9-opusdv-23365-v34-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v9-opusdv-23365-v34-uploader: Downloading snapshot of ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-W4A16...
2026-04-02T07:22:50.032396+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
2026-04-02T07:23:50.135421+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
chaiml-kimid-v9-opusdv-23365-v34-uploader: Downloaded in 97.420s
chaiml-kimid-v9-opusdv-23365-v34-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01 in 100.141s
chaiml-kimid-v9-opusdv-23365-v34-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v34-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v9-opusdv-23365-v34-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v34-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v34-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v34-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v9-opusdv-23365-v34-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v9-opusdv-23365-v34-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v9-opusdv-23365-v34-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v9-opusdv-23365-v34-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v9-opusdv-23365-v34-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-23365-v34-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/added_tokens.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/tokenizer_config.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/special_tokens_map.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/config.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/generation_config.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/merges.txt
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/chat_template.jinja
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/quantization_config.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/.gitattributes
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/vocab.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/tokenizer.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model.safetensors.index.json
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00027-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-23365-v34-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-23365-v34/default/model-00016-of-00027.safetensors
2026-04-02T07:24:50.222659+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
Job chaiml-kimid-v9-opusdv-23365-v34-uploader completed after 184.55s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-23365-v34-uploader
Pipeline stage VLLMUploader completed in 185.00s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.06s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-23365-v34
Waiting for inference service chaiml-kimid-v9-opusdv-23365-v34 to be ready
2026-04-02T07:25:50.316840+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
2026-04-02T07:26:50.414716+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
2026-04-02T07:27:50.507841+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
Inference service chaiml-kimid-v9-opusdv-23365-v34 ready after 220.4830765724182s
Pipeline stage VLLMDeployer completed in 221.01s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.4680559635162354s
Received healthy response to inference request in 3.502305507659912s
Received healthy response to inference request in 1.4474084377288818s
Received healthy response to inference request in 1.4363901615142822s
Received healthy response to inference request in 1.5140175819396973s
2026-04-02T07:28:50.601199+00:00 monitor updated for chaiml-kimid-v9-opusdv_23365_v34
Received healthy response to inference request in 1.504737138748169s
Received healthy response to inference request in 3.5167853832244873s
Received healthy response to inference request in 1.4776206016540527s
Received healthy response to inference request in 1.5910098552703857s
Received healthy response to inference request in 1.4602603912353516s
Received healthy response to inference request in 1.4421782493591309s
Received healthy response to inference request in 1.6588311195373535s
Received healthy response to inference request in 1.6039397716522217s
Received healthy response to inference request in 1.5153424739837646s
Received healthy response to inference request in 1.4798474311828613s
Received healthy response to inference request in 1.4884727001190186s
Received healthy response to inference request in 1.5106875896453857s
Received healthy response to inference request in 1.5291218757629395s
Received healthy response to inference request in 1.4956562519073486s
Received healthy response to inference request in 1.5003502368927002s
Received healthy response to inference request in 1.8221435546875s
Received healthy response to inference request in 1.5847830772399902s
Received healthy response to inference request in 2.020014524459839s
Received healthy response to inference request in 1.54923415184021s
Received healthy response to inference request in 1.5377304553985596s
Received healthy response to inference request in 1.4718403816223145s
Received healthy response to inference request in 1.4774799346923828s
Received healthy response to inference request in 1.4944369792938232s
Received healthy response to inference request in 1.441270351409912s
Received healthy response to inference request in 1.6247458457946777s
30 requests
0 failed requests
5th percentile: 1.4416789054870605
10th percentile: 1.4468854188919067
20th percentile: 1.4763520240783692
30th percentile: 1.4858851194381715
40th percentile: 1.4984726428985595
50th percentile: 1.5123525857925415
60th percentile: 1.5325653076171875
70th percentile: 1.5866511106491088
80th percentile: 1.631562900543213
90th percentile: 2.1648186683654806
95th percentile: 3.4868932127952577
99th percentile: 3.5125862193107604
mean time: 1.7388899326324463
Pipeline stage StressChecker completed in 55.45s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.66s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_23365_v34 status is now deployed due to DeploymentManager action
chaiml-kimid-v9-opusdv_23365_v34 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v9-opusdv_23365_v34 status is now torndown due to DeploymentManager action