Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-77693-v15-uploader
Waiting for job on chaiml-kimid-v8b-kimid-77693-v15-uploader to finish
chaiml-kimid-v8b-kimid-77693-v15-uploader: Using quantization_mode: w4a16
chaiml-kimid-v8b-kimid-77693-v15-uploader: Checking if ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16 already exists in ChaiML
chaiml-kimid-v8b-kimid-77693-v15-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-kimid-v8b-kimid-77693-v15-uploader: Downloading snapshot of ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-W4A16...
2026-04-03T05:23:00.151973+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
chaiml-kimid-v8b-kimid-77693-v15-uploader: Downloaded in 53.950s
chaiml-kimid-v8b-kimid-77693-v15-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01 in 56.583s
chaiml-kimid-v8b-kimid-77693-v15-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v15-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v8b-kimid-77693-v15-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v15-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v15-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v15-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v8b-kimid-77693-v15-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v8b-kimid-77693-v15-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v8b-kimid-77693-v15-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v8b-kimid-77693-v15-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v8b-kimid-77693-v15-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-77693-v15-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/added_tokens.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/generation_config.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/.gitattributes
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/special_tokens_map.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/config.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/quantization_config.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/merges.txt
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/chat_template.jinja
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/tokenizer_config.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/vocab.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model.safetensors.index.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/tokenizer.json
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00027-of-00027.safetensors
2026-04-03T05:24:00.236566+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00024-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00019-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00025-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00026-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00006-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00002-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00023-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00017-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00009-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00010-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00015-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00011-of-00027.safetensors
chaiml-kimid-v8b-kimid-77693-v15-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-77693-v15/default/model-00005-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-77693-v15-uploader completed after 137.03s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-77693-v15-uploader
Pipeline stage VLLMUploader completed in 137.53s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.10s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.78s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-77693-v15
Waiting for inference service chaiml-kimid-v8b-kimid-77693-v15 to be ready
2026-04-03T05:25:00.601840+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
2026-04-03T05:26:00.854085+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
2026-04-03T05:27:01.041797+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
Inference service chaiml-kimid-v8b-kimid-77693-v15 ready after 211.76526021957397s
Pipeline stage VLLMDeployer completed in 212.40s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.546962261199951s
2026-04-03T05:28:01.174187+00:00 monitor updated for chaiml-kimid-v8b-kimid_77693_v15
Received healthy response to inference request in 3.558075189590454s
Received healthy response to inference request in 1.4665193557739258s
Received healthy response to inference request in 1.4375309944152832s
Received healthy response to inference request in 1.4580399990081787s
Received healthy response to inference request in 1.9691493511199951s
Received healthy response to inference request in 1.5836868286132812s
Received healthy response to inference request in 1.5404119491577148s
Received healthy response to inference request in 1.5786316394805908s
Received healthy response to inference request in 1.6467561721801758s
Received healthy response to inference request in 1.4520695209503174s
Received healthy response to inference request in 1.534865140914917s
Received healthy response to inference request in 1.5267059803009033s
Received healthy response to inference request in 1.5713846683502197s
Received healthy response to inference request in 2.516864538192749s
Received healthy response to inference request in 1.5112874507904053s
Received healthy response to inference request in 1.4696698188781738s
Received healthy response to inference request in 1.4859497547149658s
Received healthy response to inference request in 1.5314819812774658s
Received healthy response to inference request in 1.4778034687042236s
Received healthy response to inference request in 1.5569586753845215s
Received healthy response to inference request in 1.645305871963501s
Received healthy response to inference request in 1.5067253112792969s
Received healthy response to inference request in 1.9985756874084473s
Received healthy response to inference request in 1.5706844329833984s
Received healthy response to inference request in 1.510754108428955s
Received healthy response to inference request in 1.468275547027588s
Received healthy response to inference request in 1.4840970039367676s
Received healthy response to inference request in 1.8543670177459717s
Received healthy response to inference request in 1.6197142601013184s
30 requests
0 failed requests
5th percentile: 1.454756236076355
10th percentile: 1.465671420097351
20th percentile: 1.4761767387390137
30th percentile: 1.5004926443099975
40th percentile: 1.520538568496704
50th percentile: 1.537638545036316
60th percentile: 1.570964527130127
70th percentile: 1.5944950580596924
80th percentile: 1.6882783412933355
90th percentile: 2.0504045724868782
95th percentile: 3.083418285846707
99th percentile: 3.5548524403572084
mean time: 1.735976799329122
Pipeline stage StressChecker completed in 55.33s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_77693_v15 status is now deployed due to DeploymentManager action
chaiml-kimid-v8b-kimid_77693_v15 status is now inactive due to auto deactivation removed underperforming models
chaiml-kimid-v8b-kimid_77693_v15 status is now torndown due to DeploymentManager action