alsebay-aetheno-v0-2-beta

submission_id: alsebay-aetheno-v0-2-beta_v1
developer_uid: Alsebay
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.5, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
model_name: alsebay-testsmp-v0-1_v1
model_repo: Alsebay/Aetheno-V0.2-Beta
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
timestamp: 2024-06-26T12:40:33+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name alsebay-aetheno-v0-2-beta-v1-mkmlizer
Waiting for job on alsebay-aetheno-v0-2-beta-v1-mkmlizer to finish
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║     _____            __           __                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║       /___/                                                         ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Version: 0.8.14                                                    ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  belonging to:                                                      ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   warnings.warn(warning_message, FutureWarning)
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Traceback (most recent call last):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     conn = connection.create_connection(
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/socket.py", line 955, in getaddrinfo
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
alsebay-aetheno-v0-2-beta-v1-mkmlizer: socket.gaierror: [Errno -3] Temporary failure in name resolution
alsebay-aetheno-v0-2-beta-v1-mkmlizer: During handling of the above exception, another exception occurred:
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Traceback (most recent call last):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     httplib_response = self._make_request(
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     self._validate_conn(conn)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     conn.connect()
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     self.sock = conn = self._new_conn()
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     raise NewConnectionError(
alsebay-aetheno-v0-2-beta-v1-mkmlizer: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fb85e040d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
alsebay-aetheno-v0-2-beta-v1-mkmlizer: During handling of the above exception, another exception occurred:
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Traceback (most recent call last):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     resp = conn.urlopen(
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     retries = retries.increment(
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
alsebay-aetheno-v0-2-beta-v1-mkmlizer: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/Alsebay/Aetheno-V0.2-Beta/tree/main?recursive=True&expand=False (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb85e040d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
alsebay-aetheno-v0-2-beta-v1-mkmlizer: During handling of the above exception, another exception occurred:
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Traceback (most recent call last):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     cli()
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return self.main(*args, **kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     rv = self.invoke(ctx)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return __callback(*args, **kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 38, in quantize
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 60, in download_to_shared_memory
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     if repo_has_model_safetensors(repo_id, revision, token):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 83, in repo_has_model_safetensors
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     files = [f.path for f in list_files_info(repo_id, revision=revision, token=token)]
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 83, in <listcomp>
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     files = [f.path for f in list_files_info(repo_id, revision=revision, token=token)]
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2755, in list_files_info
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     for subpath_info in paginate(path=tree_url, headers=headers, params={"recursive": True, "expand": expand}):
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_pagination.py", line 36, in paginate
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     r = session.get(path, params=params, headers=headers)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return self.request("GET", url, **kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     resp = self.send(prep, **send_kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     r = adapter.send(request, **kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 68, in send
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     return super().send(request, *args, **kwargs)
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 519, in send
alsebay-aetheno-v0-2-beta-v1-mkmlizer:     raise ConnectionError(e, request=request)
alsebay-aetheno-v0-2-beta-v1-mkmlizer: requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/Alsebay/Aetheno-V0.2-Beta/tree/main?recursive=True&expand=False (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb85e040d00>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))"), '(Request ID: e370376c-0f7c-4c86-97a4-b7c55bf2e0a0)')
Job alsebay-aetheno-v0-2-beta-v1-mkmlizer completed after 23.79s with status: failed
Stopping job with name alsebay-aetheno-v0-2-beta-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name alsebay-aetheno-v0-2-beta-v1-mkmlizer
Waiting for job on alsebay-aetheno-v0-2-beta-v1-mkmlizer to finish
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║     _____            __           __                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║       /___/                                                         ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Version: 0.8.14                                                    ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  belonging to:                                                      ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ║                                                                     ║
alsebay-aetheno-v0-2-beta-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   warnings.warn(warning_message, FutureWarning)
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Downloaded to shared memory in 50.353s
alsebay-aetheno-v0-2-beta-v1-mkmlizer: quantizing model to /dev/shm/model_cache
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: 
Loading 0:   0%|          | 0/579 [00:00<?, ?it/s]
Loading 0:   0%|          | 2/579 [00:04<21:38,  2.25s/it]
Loading 0:   2%|▏         | 14/579 [00:04<02:17,  4.09it/s]
Loading 0:   4%|▍         | 24/579 [00:04<01:08,  8.08it/s]
Loading 0:   6%|▌         | 36/579 [00:04<00:37, 14.34it/s]
Loading 0:   8%|▊         | 49/579 [00:04<00:23, 22.91it/s]
Loading 0:  10%|█         | 60/579 [00:05<00:23, 22.36it/s]
Loading 0:  13%|█▎        | 77/579 [00:05<00:14, 35.04it/s]
Loading 0:  16%|█▌        | 94/579 [00:05<00:09, 49.27it/s]
Loading 0:  18%|█▊        | 106/579 [00:05<00:08, 58.63it/s]
Loading 0:  21%|██        | 122/579 [00:05<00:06, 73.49it/s]
Loading 0:  23%|██▎       | 135/579 [00:06<00:05, 82.12it/s]
Loading 0:  26%|██▌       | 148/579 [00:06<00:04, 89.97it/s]
Loading 0:  28%|██▊       | 161/579 [00:06<00:04, 90.04it/s]
Loading 0:  30%|██▉       | 173/579 [00:06<00:07, 53.76it/s]
Loading 0:  32%|███▏      | 184/579 [00:06<00:06, 61.68it/s]
Loading 0:  34%|███▎      | 194/579 [00:06<00:05, 68.26it/s]
Loading 0:  35%|███▌      | 204/579 [00:07<00:05, 73.05it/s]
Loading 0:  37%|███▋      | 215/579 [00:07<00:04, 80.96it/s]
Loading 0:  39%|███▉      | 225/579 [00:07<00:04, 84.31it/s]
Loading 0:  41%|████      | 235/579 [00:07<00:04, 85.74it/s]
Loading 0:  42%|████▏     | 245/579 [00:07<00:03, 87.81it/s]
Loading 0:  44%|████▍     | 256/579 [00:07<00:03, 88.62it/s]
Loading 0:  46%|████▌     | 266/579 [00:08<00:06, 47.63it/s]
Loading 0:  48%|████▊     | 280/579 [00:08<00:04, 62.76it/s]
Loading 0:  51%|█████     | 293/579 [00:08<00:03, 73.37it/s]
Loading 0:  52%|█████▏    | 303/579 [00:08<00:03, 78.33it/s]
Loading 0:  55%|█████▌    | 319/579 [00:08<00:02, 93.38it/s]
Loading 0:  57%|█████▋    | 330/579 [00:08<00:02, 92.51it/s]
Loading 0:  60%|█████▉    | 346/579 [00:08<00:02, 106.44it/s]
Loading 0:  62%|██████▏   | 358/579 [00:08<00:02, 109.12it/s]
Loading 0:  64%|██████▍   | 370/579 [00:09<00:03, 63.75it/s] 
Loading 0:  66%|██████▌   | 380/579 [00:09<00:02, 66.44it/s]
Loading 0:  68%|██████▊   | 391/579 [00:09<00:02, 71.65it/s]
Loading 0:  69%|██████▉   | 401/579 [00:09<00:02, 77.44it/s]
Loading 0:  71%|███████   | 411/579 [00:09<00:02, 81.07it/s]
Loading 0:  73%|███████▎  | 421/579 [00:09<00:01, 80.44it/s]
Loading 0:  75%|███████▌  | 437/579 [00:09<00:01, 96.80it/s]
Loading 0:  78%|███████▊  | 450/579 [00:10<00:01, 103.69it/s]
Loading 0:  80%|███████▉  | 463/579 [00:10<00:01, 110.59it/s]
Loading 0:  82%|████████▏ | 475/579 [00:10<00:01, 60.16it/s] 
Loading 0:  85%|████████▍ | 490/579 [00:10<00:01, 74.20it/s]
Loading 0:  87%|████████▋ | 501/579 [00:20<00:18,  4.27it/s]
Loading 0:  89%|████████▉ | 517/579 [00:20<00:09,  6.49it/s]
Loading 0:  91%|█████████ | 528/579 [00:20<00:05,  8.56it/s]
Loading 0:  94%|█████████▍| 544/579 [00:20<00:02, 12.70it/s]
Loading 0:  96%|█████████▌| 555/579 [00:20<00:01, 16.29it/s]
Loading 0:  99%|█████████▊| 571/579 [00:20<00:00, 23.64it/s]
                                                            
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
alsebay-aetheno-v0-2-beta-v1-mkmlizer: quantized model in 28.271s
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Processed model Alsebay/Aetheno-V0.2-Beta in 80.914s
alsebay-aetheno-v0-2-beta-v1-mkmlizer: creating bucket guanaco-mkml-models
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alsebay-aetheno-v0-2-beta-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/special_tokens_map.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/config.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/tokenizer_config.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/tokenizer.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/flywheel_model.1.safetensors
Retrying (%r) after connection broken by '%r': %s
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alsebay-aetheno-v0-2-beta-v1/flywheel_model.0.safetensors
alsebay-aetheno-v0-2-beta-v1-mkmlizer: loading reward model from ChaiML/reward_gpt2_medium_preference_24m_e2
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   warnings.warn(
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   warnings.warn(
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   warnings.warn(
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
alsebay-aetheno-v0-2-beta-v1-mkmlizer: /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
alsebay-aetheno-v0-2-beta-v1-mkmlizer:   return self.fget.__get__(instance, owner)()
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Saving duration: 0.293s
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Processed model ChaiML/reward_gpt2_medium_preference_24m_e2 in 12.788s
alsebay-aetheno-v0-2-beta-v1-mkmlizer: creating bucket guanaco-reward-models
alsebay-aetheno-v0-2-beta-v1-mkmlizer: Bucket 's3://guanaco-reward-models/' created
alsebay-aetheno-v0-2-beta-v1-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/tokenizer_config.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/special_tokens_map.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/merges.txt
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/config.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/vocab.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/tokenizer.json
alsebay-aetheno-v0-2-beta-v1-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/alsebay-aetheno-v0-2-beta-v1_reward/reward.tensors
Job alsebay-aetheno-v0-2-beta-v1-mkmlizer completed after 115.41s with status: succeeded
Stopping job with name alsebay-aetheno-v0-2-beta-v1-mkmlizer
Pipeline stage MKMLizer completed in 140.10s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.16s
Running pipeline stage ISVCDeployer
Creating inference service alsebay-aetheno-v0-2-beta-v1
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Waiting for inference service alsebay-aetheno-v0-2-beta-v1 to be ready
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v33: ('http://mistralai-mixtral-8x7b-3473-v33-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission mistralai-mixtral-8x7b-_3473_v33: ('http://mistralai-mixtral-8x7b-3473-v33-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:52938->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service alsebay-aetheno-v0-2-beta-v1
%s, retrying in %s seconds...
Creating inference service alsebay-aetheno-v0-2-beta-v1
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Waiting for inference service alsebay-aetheno-v0-2-beta-v1 to be ready
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service alsebay-aetheno-v0-2-beta-v1
%s, retrying in %s seconds...
Creating inference service alsebay-aetheno-v0-2-beta-v1
Ignoring service alsebay-aetheno-v0-2-beta-v1 already deployed
Waiting for inference service alsebay-aetheno-v0-2-beta-v1 to be ready
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_kirak_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service alsebay-aetheno-v0-2-beta-v1
DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Date\': \'Wed, 26 Jun 2024 13:04:58 GMT\', \'Content-Type\': \'application/json\', \'Content-Length\': \'302\', \'Connection\': \'keep-alive\', \'Cache-Control\': \'no-cache, private\', \'x-kubernetes-pf-flowschema-uid\': \'95e685fd-6059-4b39-a2ec-40803f81f66b\', \'x-kubernetes-pf-prioritylevel-uid\': \'559b8561-d805-4cc8-9f38-ae50b54f7b32\', \'CF-Cache-Status\': \'DYNAMIC\', \'Set-Cookie\': \'__cf_bm=lPmRTo5D_5aQFHyw4ZynQSquwiN2QCSTi3c03rs4Mwo-1719407098-1.0.1.1-q5Jfq8eP3cJg1Wz6VbWiqzWh5hKjMM0yY3tOpnPjUA_XdWrzRxq_2KOSGWwH0A0V.oYhM2ZuY09laIelF._a1Q; path=/; expires=Wed, 26-Jun-24 13:34:58 GMT; domain=.coreweave.com; HttpOnly; Secure; SameSite=None\', \'Server\': \'cloudflare\', \'CF-RAY\': \'899d67ff1dea1259-ORD\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kubeflow.org \\\\"alsebay-aetheno-v0-2-beta-v1\\\\" not found","reason":"NotFound","details":{"name":"alsebay-aetheno-v0-2-beta-v1","group":"serving.kubeflow.org","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n  File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n    resp = func(self, *args, **kwargs)\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 270, in request\n    api_response = self.client.call_api(\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n    return self.__call_api(resource_path, method,\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n    response_data = self.request(\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 415, in request\n    return self.rest_client.DELETE(url,\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 266, in DELETE\n    return self.request("DELETE", url,\n\n  File "/usr/local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request\n    raise ApiException(http_resp=r)\n')
alsebay-aetheno-v0-2-beta_v1 status is now failed due to DeploymentManager action
admin requested tearing down of alsebay-aetheno-v0-2-beta_v1
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.40s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key alsebay-aetheno-v0-2-beta-v1/config.json from bucket guanaco-mkml-models
Deleting key alsebay-aetheno-v0-2-beta-v1/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key alsebay-aetheno-v0-2-beta-v1/flywheel_model.1.safetensors from bucket guanaco-mkml-models
Deleting key alsebay-aetheno-v0-2-beta-v1/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key alsebay-aetheno-v0-2-beta-v1/tokenizer.json from bucket guanaco-mkml-models
Deleting key alsebay-aetheno-v0-2-beta-v1/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Failed to get response for submission blend_kidub_2024-06-26: ('http://mistralai-mixtral-8x7b-3473-v46-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/config.json from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/merges.txt from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/reward.tensors from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key alsebay-aetheno-v0-2-beta-v1_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 4.40s
alsebay-aetheno-v0-2-beta_v1 status is now torndown due to DeploymentManager action