bosonai-higgs-llama-3-70b

submission_id: bosonai-higgs-llama-3-70b_v3
developer_uid: frankdu
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
model_name: bosonai-higgs-llama-3-70b_v3
model_repo: bosonai/Higgs-Llama-3-70B
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
timestamp: 2024-06-11T23:37:14+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
Waiting for job on bosonai-higgs-llama-3-70b-v3-mkmlizer to finish
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║     _____            __           __                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║       /___/                                                         ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  belonging to:                                                      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bosonai-higgs-llama-3-70b-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
bosonai-higgs-llama-3-70b-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
Failed to get response for submission nitral-ai-hathor-l3-8b-v-02_v8: ('http://nitral-ai-hathor-l3-8b-v-02-v8-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:60790->127.0.0.1:8080: read: connection reset by peer\n')
Retrying (%r) after connection broken by '%r': %s
Job bosonai-higgs-llama-3-70b-v3-mkmlizer completed after 419.15s with status: failed
Stopping job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
Waiting for job on bosonai-higgs-llama-3-70b-v3-mkmlizer to finish
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║     _____            __           __                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║       /___/                                                         ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  belonging to:                                                      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bosonai-higgs-llama-3-70b-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
bosonai-higgs-llama-3-70b-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 509, in http_get
bosonai-higgs-llama-3-70b-v3-mkmlizer:     hf_transfer.download(
bosonai-higgs-llama-3-70b-v3-mkmlizer: Exception: Error while downloading: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("cdn-lfs-us-1.huggingface.co")), port: None, path: "/repos/2a/91/20ccaf4ee9e2e6ba9d76a/0cdf80eedb750a10fe64dacc19ed96e19fe735b2e3dbd735c8571dcd3e95cff5", query: Some("response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00014-of-00062.safetensors%3B+filename%3D%22model-00014-of-00062.safetensors%22%3B&Expires=1718408954&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxODQwODk1NH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzJhLzkxLzJhOTFlZDI2ODAxNmIyMTliZjA3NTE2ZjIzZDVhNWJjZmIyZjE1NGY0ZmQyMGNjYWY0ZWU5ZTJlNmJhOWQ3NmEvMGNkZjgwZWVkYjc1MGExMGZlNjRkYWNjMTllZDk2ZTE5ZmU3MzViMmUzZGJkNzM1Yzg1NzFkY2QzZTk1Y2ZmNT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=khnICfHI%7EfqK9PoInd6shsHJjsXzX-nunnVUapkhjk0iNYSxYPt1n7Emiq0pzemhEij7PzLkrhEHny1yw-if14e%7EmkbQmIRnWdPI-TZzxU2OAHClykfhTnm9Z-9fPziHbacIL6EnuP7a-x7UYRemONk0rAcialOhmKIFCyxSC6-5UEEG8oEg%7EEqni9wFGGwO8VFddZM56O76i7FFkB1hL9Qoq1JKcsTSJpk%7EJPii%7E0JEASAwnvO6UemYQwsgSsW1s86PXyz5DAfAumsvFF9aQHYac-646Xj1U%7EPXhGOcM1OJILp0HxEg0Cpjq807i9Y-BIjvuSLfYem1c4B3jJ9Xzg__&Key-Pair-Id=K2FPYV99P2N66Q"), fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Temporary failure in name resolution" })) }
bosonai-higgs-llama-3-70b-v3-mkmlizer: The above exception was the direct cause of the following exception:
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
bosonai-higgs-llama-3-70b-v3-mkmlizer:     cli()
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return self.main(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
bosonai-higgs-llama-3-70b-v3-mkmlizer:     rv = self.invoke(ctx)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return __callback(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 38, in quantize
bosonai-higgs-llama-3-70b-v3-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 65, in download_to_shared_memory
bosonai-higgs-llama-3-70b-v3-mkmlizer:     snapshot_download(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return fn(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 314, in snapshot_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     _inner_hf_hub_download(file)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 290, in _inner_hf_hub_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return hf_hub_download(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return fn(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1492, in hf_hub_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     http_get(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 520, in http_get
bosonai-higgs-llama-3-70b-v3-mkmlizer:     raise RuntimeError(
bosonai-higgs-llama-3-70b-v3-mkmlizer: RuntimeError: An error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.
Job bosonai-higgs-llama-3-70b-v3-mkmlizer completed after 309.41s with status: failed
Stopping job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
Waiting for job on bosonai-higgs-llama-3-70b-v3-mkmlizer to finish
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║     _____            __           __                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║       /___/                                                         ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  belonging to:                                                      ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ║                                                                     ║
bosonai-higgs-llama-3-70b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
bosonai-higgs-llama-3-70b-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
bosonai-higgs-llama-3-70b-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     conn = connection.create_connection(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
bosonai-higgs-llama-3-70b-v3-mkmlizer:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/socket.py", line 955, in getaddrinfo
bosonai-higgs-llama-3-70b-v3-mkmlizer:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
bosonai-higgs-llama-3-70b-v3-mkmlizer: socket.gaierror: [Errno -3] Temporary failure in name resolution
bosonai-higgs-llama-3-70b-v3-mkmlizer: During handling of the above exception, another exception occurred:
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
bosonai-higgs-llama-3-70b-v3-mkmlizer:     httplib_response = self._make_request(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
bosonai-higgs-llama-3-70b-v3-mkmlizer:     self._validate_conn(conn)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     conn.connect()
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
bosonai-higgs-llama-3-70b-v3-mkmlizer:     self.sock = conn = self._new_conn()
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     raise NewConnectionError(
bosonai-higgs-llama-3-70b-v3-mkmlizer: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f091ddc67a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
bosonai-higgs-llama-3-70b-v3-mkmlizer: During handling of the above exception, another exception occurred:
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
bosonai-higgs-llama-3-70b-v3-mkmlizer:     resp = conn.urlopen(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
bosonai-higgs-llama-3-70b-v3-mkmlizer:     retries = retries.increment(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
bosonai-higgs-llama-3-70b-v3-mkmlizer:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
bosonai-higgs-llama-3-70b-v3-mkmlizer: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Max retries exceeded with url: /repos/2a/91/20ccaf4ee9e2e6ba9d76a/de7e035902dd0caf588644c1bd8d17d6c9d38d4f6299756619dc894141f64c33?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00005-of-00062.safetensors%3B+filename%3D%22model-00005-of-00062.safetensors%22%3B&Expires=1718409035&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxODQwOTAzNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzJhLzkxLzJhOTFlZDI2ODAxNmIyMTliZjA3NTE2ZjIzZDVhNWJjZmIyZjE1NGY0ZmQyMGNjYWY0ZWU5ZTJlNmJhOWQ3NmEvZGU3ZTAzNTkwMmRkMGNhZjU4ODY0NGMxYmQ4ZDE3ZDZjOWQzOGQ0ZjYyOTk3NTY2MTlkYzg5NDE0MWY2NGMzMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=nWlA5cmV1~uR8-CEt6KDrVPZU81j71IixVfaGNrnvMLsYpog6eyuA~kp77HCt87gzSjnZxfiUGury3pGa~CLwR4QFm-q2LPNHhSAN-O8zbJ6~o3l7ttL9f~pQw~K51Yn-1X6uq1xJ8FUosTd0Ym9YrDl9cQMy97UR3Klks-0TdTRFrLRQueOGj5qkQO3FdL0BoKV-p5wf727Puk1xlKRIStd0GDCInm2PTRdH1s0fc1qc-ZBr8iASsjhy7Cx6nkonNaFrQQJFWDAfOkH75D81jjxdhYW~RYuyGi5In21yIYVKCl~8kai7awOdBdXv2XVdceuGeGwvxE29fkxZT30kA__&Key-Pair-Id=K2FPYV99P2N66Q (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f091ddc67a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
bosonai-higgs-llama-3-70b-v3-mkmlizer: During handling of the above exception, another exception occurred:
bosonai-higgs-llama-3-70b-v3-mkmlizer: Traceback (most recent call last):
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
bosonai-higgs-llama-3-70b-v3-mkmlizer:     cli()
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return self.main(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
bosonai-higgs-llama-3-70b-v3-mkmlizer:     rv = self.invoke(ctx)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return __callback(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 38, in quantize
bosonai-higgs-llama-3-70b-v3-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 65, in download_to_shared_memory
bosonai-higgs-llama-3-70b-v3-mkmlizer:     snapshot_download(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return fn(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 314, in snapshot_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     _inner_hf_hub_download(file)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 290, in _inner_hf_hub_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return hf_hub_download(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return fn(*args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1492, in hf_hub_download
bosonai-higgs-llama-3-70b-v3-mkmlizer:     http_get(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 456, in http_get
bosonai-higgs-llama-3-70b-v3-mkmlizer:     r = _request_wrapper(
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 392, in _request_wrapper
bosonai-higgs-llama-3-70b-v3-mkmlizer:     response = get_session().request(method=method, url=url, **params)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
bosonai-higgs-llama-3-70b-v3-mkmlizer:     resp = self.send(prep, **send_kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
bosonai-higgs-llama-3-70b-v3-mkmlizer:     r = adapter.send(request, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 68, in send
bosonai-higgs-llama-3-70b-v3-mkmlizer:     return super().send(request, *args, **kwargs)
bosonai-higgs-llama-3-70b-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
bosonai-higgs-llama-3-70b-v3-mkmlizer:     raise ConnectionError(e, request=request)
bosonai-higgs-llama-3-70b-v3-mkmlizer: requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='cdn-lfs-us-1.huggingface.co', port=443): Max retries exceeded with url: /repos/2a/91/20ccaf4ee9e2e6ba9d76a/de7e035902dd0caf588644c1bd8d17d6c9d38d4f6299756619dc894141f64c33?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00005-of-00062.safetensors%3B+filename%3D%22model-00005-of-00062.safetensors%22%3B&Expires=1718409035&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcxODQwOTAzNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzJhLzkxLzJhOTFlZDI2ODAxNmIyMTliZjA3NTE2ZjIzZDVhNWJjZmIyZjE1NGY0ZmQyMGNjYWY0ZWU5ZTJlNmJhOWQ3NmEvZGU3ZTAzNTkwMmRkMGNhZjU4ODY0NGMxYmQ4ZDE3ZDZjOWQzOGQ0ZjYyOTk3NTY2MTlkYzg5NDE0MWY2NGMzMz9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=nWlA5cmV1~uR8-CEt6KDrVPZU81j71IixVfaGNrnvMLsYpog6eyuA~kp77HCt87gzSjnZxfiUGury3pGa~CLwR4QFm-q2LPNHhSAN-O8zbJ6~o3l7ttL9f~pQw~K51Yn-1X6uq1xJ8FUosTd0Ym9YrDl9cQMy97UR3Klks-0TdTRFrLRQueOGj5qkQO3FdL0BoKV-p5wf727Puk1xlKRIStd0GDCInm2PTRdH1s0fc1qc-ZBr8iASsjhy7Cx6nkonNaFrQQJFWDAfOkH75D81jjxdhYW~RYuyGi5In21yIYVKCl~8kai7awOdBdXv2XVdceuGeGwvxE29fkxZT30kA__&Key-Pair-Id=K2FPYV99P2N66Q (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f091ddc67a0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: 35a05c03-3e9f-4455-96f8-052a0ebc6c1b)')
Job bosonai-higgs-llama-3-70b-v3-mkmlizer completed after 85.91s with status: failed
Stopping job with name bosonai-higgs-llama-3-70b-v3-mkmlizer
MKMLizerError('')
bosonai-higgs-llama-3-70b_v3 status is now failed due to DeploymentManager action
admin requested tearing down of bosonai-higgs-llama-3-70b_v3
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.22s
Running pipeline stage MKMLModelDeleter
Skipping deletion as no model was successfully uploaded
Pipeline stage MKMLModelDeleter completed in 0.20s
bosonai-higgs-llama-3-70b_v3 status is now torndown due to DeploymentManager action