Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v11-kimidv-25711-v5-uploader
Waiting for job on chaiml-kimid-v11-kimidv-25711-v5-uploader to finish
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-25711-v5-uploader: Using quantization_mode: w4a16
chaiml-kimid-v11-kimidv-25711-v5-uploader: Checking if ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-W4A16 already exists in ChaiML
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-kimid-v11-kimidv-25711-v5-uploader: Model already exists. Downloading to /dev/shm/model_output...
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-kimid-v11-kimidv-25711-v5-uploader: Downloading snapshot of ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01-W4A16...
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-kimid-v11-kimidv-25711-v5-uploader:
Fetching 39 files: 0%| | 0/39 [00:00<?, ?it/s]
Fetching 39 files: 3%|▎ | 1/39 [00:00<00:13, 2.78it/s]
Fetching 39 files: 18%|█▊ | 7/39 [00:13<01:02, 1.96s/it]
Fetching 39 files: 21%|██ | 8/39 [00:14<00:57, 1.84s/it]
Fetching 39 files: 26%|██▌ | 10/39 [00:16<00:43, 1.52s/it]
Fetching 39 files: 38%|███▊ | 15/39 [00:25<00:40, 1.69s/it]
Fetching 39 files: 41%|████ | 16/39 [00:27<00:39, 1.71s/it]
Fetching 39 files: 49%|████▊ | 19/39 [00:30<00:29, 1.49s/it]
Fetching 39 files: 59%|█████▉ | 23/39 [00:38<00:27, 1.70s/it]
Fetching 39 files: 62%|██████▏ | 24/39 [00:41<00:27, 1.86s/it]
Fetching 39 files: 69%|██████▉ | 27/39 [00:42<00:15, 1.26s/it]
Fetching 39 files: 72%|███████▏ | 28/39 [00:43<00:13, 1.24s/it]
Fetching 39 files: 74%|███████▍ | 29/39 [00:43<00:11, 1.15s/it]
Fetching 39 files: 79%|███████▉ | 31/39 [00:46<00:08, 1.12s/it]
Fetching 39 files: 82%|████████▏ | 32/39 [00:47<00:08, 1.18s/it]
Fetching 39 files: 100%|██████████| 39/39 [00:47<00:00, 1.22s/it]
chaiml-kimid-v11-kimidv-25711-v5-uploader: Downloaded in 47.506s
chaiml-kimid-v11-kimidv-25711-v5-uploader: Processed model ChaiML/kimid-v11-kimidv4ann-lr1e5ep2r64g8b01 in 48.043s
chaiml-kimid-v11-kimidv-25711-v5-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v11-kimidv-25711-v5-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-kimid-v11-kimidv-25711-v5-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v11-kimidv-25711-v5-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v11-kimidv-25711-v5-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v11-kimidv-25711-v5-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-kimid-v11-kimidv-25711-v5-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-kimid-v11-kimidv-25711-v5-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-kimid-v11-kimidv-25711-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-kimid-v11-kimidv-25711-v5-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-kimid-v11-kimidv-25711-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-kimidv-25711-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/chat_template.jinja
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/added_tokens.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/.gitattributes
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/generation_config.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/tokenizer_config.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/quantization_config.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/special_tokens_map.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/config.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/vocab.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/merges.txt
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model.safetensors.index.json
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/tokenizer.json
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission blend_maber_2026-02-10: HTTPConnectionPool(host='chaiml-bol-kimid-v8a-lr-42637-v9-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00007-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00004-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00010-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00016-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00021-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00005-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00020-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00008-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00017-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00003-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00022-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00012-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00026-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00002-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00018-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00015-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00023-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00006-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00025-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00009-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00001-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00024-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00014-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00011-of-00027.safetensors
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00019-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-kimidv-25711-v5-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-kimidv-25711-v5/model-00013-of-00027.safetensors
Job chaiml-kimid-v11-kimidv-25711-v5-uploader completed after 988.3s with status: succeeded
Stopping job with name chaiml-kimid-v11-kimidv-25711-v5-uploader
Pipeline stage VLLMUploader completed in 988.71s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v11-kimidv-25711-v5
Waiting for inference service chaiml-kimid-v11-kimidv-25711-v5 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-kimid-v11-kimidv-25711-v5 ready after 1062.4295346736908s
Pipeline stage VLLMDeployer completed in 1062.88s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8112003803253174s
Received healthy response to inference request in 1.7764544486999512s
Received healthy response to inference request in 1.8382346630096436s
Received healthy response to inference request in 1.9043314456939697s
Received healthy response to inference request in 1.8460819721221924s
Received healthy response to inference request in 1.8120596408843994s
Received healthy response to inference request in 1.7381906509399414s
Received healthy response to inference request in 1.7583796977996826s
Received healthy response to inference request in 1.890840768814087s
Received healthy response to inference request in 1.8201313018798828s
Received healthy response to inference request in 2.109456777572632s
Received healthy response to inference request in 1.829390525817871s
Received healthy response to inference request in 1.7570631504058838s
Received healthy response to inference request in 1.786402940750122s
Received healthy response to inference request in 1.9506816864013672s
Received healthy response to inference request in 2.164952516555786s
Received healthy response to inference request in 1.734914779663086s
Received healthy response to inference request in 1.8256595134735107s
Received healthy response to inference request in 1.7925012111663818s
Received healthy response to inference request in 1.7445476055145264s
Received healthy response to inference request in 1.7026581764221191s
Received healthy response to inference request in 1.8356189727783203s
Received healthy response to inference request in 2.2587873935699463s
Received healthy response to inference request in 2.2005045413970947s
Received healthy response to inference request in 2.333958625793457s
Received healthy response to inference request in 2.210761308670044s
Received healthy response to inference request in 1.9837403297424316s
Received healthy response to inference request in 1.73823881149292s
Received healthy response to inference request in 2.1330153942108154s
Received healthy response to inference request in 2.017430543899536s
30 requests
0 failed requests
5th percentile: 1.736388921737671
10th percentile: 1.738233995437622
20th percentile: 1.7581163883209228
30th percentile: 1.7906717300415038
40th percentile: 1.8169026374816895
50th percentile: 1.8325047492980957
60th percentile: 1.8639854907989502
70th percentile: 1.9605992794036864
80th percentile: 2.1141685009002686
90th percentile: 2.2015302181243896
95th percentile: 2.23717565536499
99th percentile: 2.3121589684486388
mean time: 1.9102063258488973
Pipeline stage StressChecker completed in 60.69s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
Shutdown handler de-registered
chaiml-kimid-v11-kimidv_25711_v5 status is now deployed due to DeploymentManager action
chaiml-kimid-v11-kimidv_25711_v5 status is now inactive due to auto deactivation removed underperforming models