Running pipeline stage MKMLizer
Starting job with name cycy233-l3-bpf-v0-c4-v1-mkmlizer
Waiting for job on cycy233-l3-bpf-v0-c4-v1-mkmlizer to finish
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ _____ __ __ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ /___/ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Version: 0.10.1 ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ https://mk1.ai ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ The license key for the current software has been verified as ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ belonging to: ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Chai Research Corp. ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Traceback (most recent call last):
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
cycy233-l3-bpf-v0-c4-v1-mkmlizer: response.raise_for_status()
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
cycy233-l3-bpf-v0-c4-v1-mkmlizer: raise HTTPError(http_error_msg, response=self)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://cdn-lfs-us-1.huggingface.co/repos/5e/38/dfae0babd08e28fc824d0/6937e78efcfbd87f9484219fa1412c78efc718021034bee4e4bd0ad54312afc3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27pytorch_model-00002-of-00004.bin%3B+filename%3D%22pytorch_model-00002-of-00004.bin%22%3B&response-content-type=application%2Foctet-stream&Expires=1725155760&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyNTE1NTc2MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzVlLzM4LzVlMzg2NzM5MjcxMGI5MTE0NDRiZWYyMTc0NDM0OGJjMzM3ZDgzN2NlZTBkZmFlMGJhYmQwOGUyOGZjODI0ZDAvNjkzN2U3OGVmY2ZiZDg3Zjk0ODQyMTlmYTE0MTJjNzhlZmM3MTgwMjEwMzRiZWU0ZTRiZDBhZDU0MzEyYWZjMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=eliMWraJeruO8va7x5m15NuvwS80VdyVOLzJQb9Fyc3oBRVBxI3pUzu52ORqfKMz9IKc2txIildBoPnw9yJVgvWw0CTx~zkE7M0v73lHe9U~pW8KDpAPUWkl4xDklSUYrA6HK9HKNwOgjgD0FdbjT12HyMg7X8T4NIx5iOiwBHvKFJo8weLMcjwTx3eRykTWRnKZTFEK~i~Iz1p3JUH3cqwCnrZTtU3MN8OD0aRYfb6a-KRRBvzDDdaRnuDYLrdgLiw4tKLVhjy9vBYXUDnCoKIABRJX1wzAHAa2hILhvBOZL-2qwwqx~35jL8SWmNFUsI5E1jvKBQuDVAm431XgpQ__&Key-Pair-Id=K24J24Z295AEI9
cycy233-l3-bpf-v0-c4-v1-mkmlizer: The above exception was the direct cause of the following exception:
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Traceback (most recent call last):
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 151, in <module>
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cli()
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return self.main(*args, **kwargs)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
cycy233-l3-bpf-v0-c4-v1-mkmlizer: rv = self.invoke(ctx)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return _process_result(sub_ctx.command.invoke(sub_ctx))
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return ctx.invoke(self.callback, **ctx.params)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return __callback(*args, **kwargs)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 38, in quantize
cycy233-l3-bpf-v0-c4-v1-mkmlizer: temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 65, in download_to_shared_memory
cycy233-l3-bpf-v0-c4-v1-mkmlizer: snapshot_download(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return fn(*args, **kwargs)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 297, in snapshot_download
cycy233-l3-bpf-v0-c4-v1-mkmlizer: _inner_hf_hub_download(file)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 273, in _inner_hf_hub_download
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return hf_hub_download(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return f(*args, **kwargs)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return fn(*args, **kwargs)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1220, in hf_hub_download
cycy233-l3-bpf-v0-c4-v1-mkmlizer: return _hf_hub_download_to_local_dir(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1515, in _hf_hub_download_to_local_dir
cycy233-l3-bpf-v0-c4-v1-mkmlizer: _download_to_tmp_and_move(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1915, in _download_to_tmp_and_move
cycy233-l3-bpf-v0-c4-v1-mkmlizer: http_get(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 463, in http_get
cycy233-l3-bpf-v0-c4-v1-mkmlizer: r = _request_wrapper(
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 400, in _request_wrapper
cycy233-l3-bpf-v0-c4-v1-mkmlizer: hf_raise_for_status(response)
cycy233-l3-bpf-v0-c4-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
cycy233-l3-bpf-v0-c4-v1-mkmlizer: raise HfHubHTTPError(str(e), response=response) from e
cycy233-l3-bpf-v0-c4-v1-mkmlizer: huggingface_hub.utils._errors.HfHubHTTPError: 503 Server Error: Service Unavailable for url: https://cdn-lfs-us-1.huggingface.co/repos/5e/38/dfae0babd08e28fc824d0/6937e78efcfbd87f9484219fa1412c78efc718021034bee4e4bd0ad54312afc3?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27pytorch_model-00002-of-00004.bin%3B+filename%3D%22pytorch_model-00002-of-00004.bin%22%3B&response-content-type=application%2Foctet-stream&Expires=1725155760&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyNTE1NTc2MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzVlLzM4LzVlMzg2NzM5MjcxMGI5MTE0NDRiZWYyMTc0NDM0OGJjMzM3ZDgzN2NlZTBkZmFlMGJhYmQwOGUyOGZjODI0ZDAvNjkzN2U3OGVmY2ZiZDg3Zjk0ODQyMTlmYTE0MTJjNzhlZmM3MTgwMjEwMzRiZWU0ZTRiZDBhZDU0MzEyYWZjMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSomcmVzcG9uc2UtY29udGVudC10eXBlPSoifV19&Signature=eliMWraJeruO8va7x5m15NuvwS80VdyVOLzJQb9Fyc3oBRVBxI3pUzu52ORqfKMz9IKc2txIildBoPnw9yJVgvWw0CTx~zkE7M0v73lHe9U~pW8KDpAPUWkl4xDklSUYrA6HK9HKNwOgjgD0FdbjT12HyMg7X8T4NIx5iOiwBHvKFJo8weLMcjwTx3eRykTWRnKZTFEK~i~Iz1p3JUH3cqwCnrZTtU3MN8OD0aRYfb6a-KRRBvzDDdaRnuDYLrdgLiw4tKLVhjy9vBYXUDnCoKIABRJX1wzAHAa2hILhvBOZL-2qwwqx~35jL8SWmNFUsI5E1jvKBQuDVAm431XgpQ__&Key-Pair-Id=K24J24Z295AEI9
Job cycy233-l3-bpf-v0-c4-v1-mkmlizer completed after 46.04s with status: failed
Stopping job with name cycy233-l3-bpf-v0-c4-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name cycy233-l3-bpf-v0-c4-v1-mkmlizer
Waiting for job on cycy233-l3-bpf-v0-c4-v1-mkmlizer to finish
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ _____ __ __ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ /___/ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Version: 0.10.1 ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ https://mk1.ai ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ The license key for the current software has been verified as ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ belonging to: ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Chai Research Corp. ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ║ ║
cycy233-l3-bpf-v0-c4-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-elo-alignment-run-3_v36: ('http://chaiml-elo-alignment-run-3-v36-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:56208->127.0.0.1:8080: read: connection reset by peer\n')
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Downloaded to shared memory in 37.663s
cycy233-l3-bpf-v0-c4-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpudcd1v0s, device:0
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cycy233-l3-bpf-v0-c4-v1-mkmlizer: quantized model in 25.669s
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Processed model cycy233/L3-bpf-v0-c4 in 63.333s
cycy233-l3-bpf-v0-c4-v1-mkmlizer: creating bucket guanaco-mkml-models
cycy233-l3-bpf-v0-c4-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cycy233-l3-bpf-v0-c4-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1/config.json
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1/special_tokens_map.json
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1/tokenizer_config.json
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1/tokenizer.json
cycy233-l3-bpf-v0-c4-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cycy233-l3-bpf-v0-c4-v1/flywheel_model.0.safetensors
cycy233-l3-bpf-v0-c4-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 4/291 [00:00<00:07, 38.44it/s]
Loading 0: 4%|▍ | 13/291 [00:00<00:04, 64.42it/s]
Loading 0: 8%|▊ | 22/291 [00:00<00:03, 71.68it/s]
Loading 0: 12%|█▏ | 34/291 [00:00<00:03, 76.32it/s]
Loading 0: 15%|█▍ | 43/291 [00:00<00:03, 78.73it/s]
Loading 0: 18%|█▊ | 52/291 [00:00<00:03, 77.65it/s]
Loading 0: 21%|██ | 61/291 [00:00<00:02, 80.81it/s]
Loading 0: 26%|██▌ | 76/291 [00:00<00:02, 91.79it/s]
Loading 0: 30%|██▉ | 86/291 [00:02<00:08, 23.56it/s]
Loading 0: 32%|███▏ | 94/291 [00:02<00:06, 28.42it/s]
Loading 0: 36%|███▋ | 106/291 [00:02<00:04, 37.21it/s]
Loading 0: 42%|████▏ | 121/291 [00:02<00:03, 49.70it/s]
Loading 0: 46%|████▌ | 133/291 [00:02<00:02, 57.68it/s]
Loading 0: 49%|████▉ | 142/291 [00:02<00:02, 61.85it/s]
Loading 0: 52%|█████▏ | 151/291 [00:02<00:02, 62.92it/s]
Loading 0: 57%|█████▋ | 166/291 [00:03<00:01, 75.08it/s]
Loading 0: 61%|██████ | 178/291 [00:03<00:01, 78.92it/s]
Loading 0: 64%|██████▍ | 187/291 [00:04<00:04, 25.43it/s]
Loading 0: 67%|██████▋ | 196/291 [00:04<00:03, 31.13it/s]
Loading 0: 70%|███████ | 205/291 [00:04<00:02, 36.89it/s]
Loading 0: 76%|███████▌ | 220/291 [00:04<00:01, 49.23it/s]
Loading 0: 79%|███████▊ | 229/291 [00:04<00:01, 54.05it/s]
Loading 0: 83%|████████▎ | 241/291 [00:04<00:00, 61.14it/s]
Loading 0: 86%|████████▌ | 250/291 [00:05<00:00, 62.10it/s]
Loading 0: 89%|████████▉ | 259/291 [00:05<00:00, 66.00it/s]
Loading 0: 94%|█████████▍| 274/291 [00:05<00:00, 78.20it/s]
Loading 0: 98%|█████████▊| 286/291 [00:05<00:00, 80.66it/s]
Job cycy233-l3-bpf-v0-c4-v1-mkmlizer completed after 86.47s with status: succeeded
Stopping job with name cycy233-l3-bpf-v0-c4-v1-mkmlizer
Pipeline stage MKMLizer completed in 133.83s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.13s
Running pipeline stage ISVCDeployer
Creating inference service cycy233-l3-bpf-v0-c4-v1
Waiting for inference service cycy233-l3-bpf-v0-c4-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service cycy233-l3-bpf-v0-c4-v1 ready after 171.07403373718262s
Pipeline stage ISVCDeployer completed in 171.87s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.101196765899658s
Received healthy response to inference request in 1.4870145320892334s
Received healthy response to inference request in 1.9804589748382568s
Received healthy response to inference request in 1.5445888042449951s
Received healthy response to inference request in 1.8865177631378174s
5 requests
0 failed requests
5th percentile: 1.4985293865203857
10th percentile: 1.510044240951538
20th percentile: 1.5330739498138428
30th percentile: 1.6129745960235595
40th percentile: 1.7497461795806886
50th percentile: 1.8865177631378174
60th percentile: 1.9240942478179932
70th percentile: 1.961670732498169
80th percentile: 2.004606533050537
90th percentile: 2.0529016494750976
95th percentile: 2.077049207687378
99th percentile: 2.0963672542572023
mean time: 1.7999553680419922
Pipeline stage StressChecker completed in 10.34s
cycy233-l3-bpf-v0-c4_v1 status is now deployed due to DeploymentManager action
cycy233-l3-bpf-v0-c4_v1 status is now inactive due to auto deactivation removed underperforming models
cycy233-l3-bpf-v0-c4_v1 status is now torndown due to DeploymentManager action
cycy233-l3-bpf-v0-c4_v1 status is now torndown due to DeploymentManager action