Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-2fe5-c13f-linear-w01-v42-uploader
Waiting for job on chaiml-2fe5-c13f-linear-w01-v42-uploader to finish
HTTP Request: %s %s "%s %d %s"
chaiml-2fe5-c13f-linear-w01-v42-uploader: Using quantization_mode: none
chaiml-2fe5-c13f-linear-w01-v42-uploader: Downloading snapshot of ChaiML/2fe5-c13f-linear-w01...
chaiml-2fe5-c13f-linear-w01-v42-uploader:
Fetching 14 files: 0%| | 0/14 [00:00<?, ?it/s]
Fetching 14 files: 7%|▋ | 1/14 [00:00<00:03, 4.30it/s]
Fetching 14 files: 43%|████▎ | 6/14 [00:12<00:16, 2.10s/it]
Fetching 14 files: 50%|█████ | 7/14 [00:12<00:11, 1.70s/it]
Fetching 14 files: 57%|█████▋ | 8/14 [00:12<00:08, 1.35s/it]
Fetching 14 files: 64%|██████▍ | 9/14 [00:12<00:05, 1.14s/it]
Fetching 14 files: 100%|██████████| 14/14 [00:12<00:00, 1.10it/s]
chaiml-2fe5-c13f-linear-w01-v42-uploader: Downloaded in 12.879s
HTTP Request: %s %s "%s %d %s"
chaiml-2fe5-c13f-linear-w01-v42-uploader: Processed model ChaiML/2fe5-c13f-linear-w01 in 21.994s
chaiml-2fe5-c13f-linear-w01-v42-uploader: creating bucket guanaco-vllm-models
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-2fe5-c13f-linear-w01-v42-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-2fe5-c13f-linear-w01-v42-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-2fe5-c13f-linear-w01-v42-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-2fe5-c13f-linear-w01-v42-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-2fe5-c13f-linear-w01-v42-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-2fe5-c13f-linear-w01-v42-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-2fe5-c13f-linear-w01-v42-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-2fe5-c13f-linear-w01-v42-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-2fe5-c13f-linear-w01-v42-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-2fe5-c13f-linear-w01-v42-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-2fe5-c13f-linear-w01-v42-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/config.json
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/README.md
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/mergekit_config.yaml s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/mergekit_config.yaml
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/mergekit_config.yml s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/mergekit_config.yml
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/special_tokens_map.json
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/.gitattributes
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model.safetensors.index.json
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/tokenizer_config.json
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/tokenizer.json
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model-00005-of-00005.safetensors s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model-00005-of-00005.safetensors
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model-00002-of-00005.safetensors s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model-00002-of-00005.safetensors
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model-00001-of-00005.safetensors s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model-00001-of-00005.safetensors
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model-00003-of-00005.safetensors s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model-00003-of-00005.safetensors
chaiml-2fe5-c13f-linear-w01-v42-uploader: cp /dev/shm/model_output/model-00004-of-00005.safetensors s3://guanaco-vllm-models/chaiml-2fe5-c13f-linear-w01-v42/model-00004-of-00005.safetensors
Job chaiml-2fe5-c13f-linear-w01-v42-uploader completed after 230.13s with status: succeeded
Stopping job with name chaiml-2fe5-c13f-linear-w01-v42-uploader
Pipeline stage VLLMUploader completed in 231.72s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-2fe5-c13f-linear-w01-v42
Waiting for inference service chaiml-2fe5-c13f-linear-w01-v42 to be ready
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-2fe5-c13f-linear-w01-v42 ready after 1118.241976261139s
Pipeline stage VLLMDeployer completed in 1118.75s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.5812110900878906s
Received healthy response to inference request in 2.358673095703125s
Received healthy response to inference request in 2.6842751502990723s
Received healthy response to inference request in 2.4372708797454834s
Received healthy response to inference request in 2.4715497493743896s
Received healthy response to inference request in 2.419428825378418s
Received healthy response to inference request in 3.1548023223876953s
Received healthy response to inference request in 2.8943469524383545s
Received healthy response to inference request in 2.9137938022613525s
Received healthy response to inference request in 2.727607011795044s
Received healthy response to inference request in 3.1255874633789062s
Received healthy response to inference request in 2.425938129425049s
Received healthy response to inference request in 2.4425244331359863s
Received healthy response to inference request in 2.6963722705841064s
Received healthy response to inference request in 2.5103254318237305s
Received healthy response to inference request in 2.396024703979492s
Received healthy response to inference request in 2.482109785079956s
Received healthy response to inference request in 2.59700345993042s
Received healthy response to inference request in 3.240429401397705s
chaiml-2fe5-c13f-linear-w01_v42 status is now inactive due to auto deactivation removed underperforming models
chaiml-2fe5-c13f-linear-w01_v42 status is now torndown due to DeploymentManager action