zonemercy-vingt-deux-v11_5332

developer_uid: chai_backend_admin
submission_id: zonemercy-vingt-deux-v11_5332_v2
model_name: tempv1-4
model_group: zonemercy/Vingt-Deux-v11
status: torndown
timestamp: 2024-11-01T00:16:22+00:00
num_battles: 14425
num_wins: 7134
celo_rating: 1236.15
family_friendly_score: 0.5664
family_friendly_standard_error: 0.007008438342455472
submission_type: basic
model_repo: zonemercy/Vingt-Deux-v11k256128-22b01e5
model_architecture: MistralForCausalLM
model_num_parameters: 22247282688.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: tempv1-4
is_internal_developer: True
language_model: zonemercy/Vingt-Deux-v11k256128-22b01e5
model_size: 22B
ranking_group: single
us_pacific_date: 2024-10-31
win_ratio: 0.4945580589254766
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-vingt-deux-v11-5332-v2-mkmlizer
Waiting for job on zonemercy-vingt-deux-v11-5332-v2-mkmlizer to finish
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Traceback (most recent call last):
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     response = conn.getresponse()
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     httplib_response = super().getresponse()
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 1375, in getresponse
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     response.begin()
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 318, in begin
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     version, status, reason = self._read_status()
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 279, in _read_status
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/socket.py", line 705, in readinto
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return self._sock.recv_into(b)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/ssl.py", line 1307, in recv_into
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return self.read(nbytes, buffer)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/ssl.py", line 1163, in read
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return self._sslobj.read(len, buffer)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: TimeoutError: The read operation timed out
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: The above exception was the direct cause of the following exception:
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Traceback (most recent call last):
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     resp = conn.urlopen(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     retries = retries.increment(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     raise reraise(type(error), error, _stacktrace)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     raise value
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     response = self._make_request(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     raise ReadTimeoutError(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: During handling of the above exception, another exception occurred:
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Traceback (most recent call last):
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     cli()
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return self.main(*args, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     rv = self.invoke(ctx)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return __callback(*args, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 38, in quantize
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 65, in download_to_shared_memory
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     snapshot_download(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return fn(*args, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 292, in snapshot_download
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     _inner_hf_hub_download(file)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 268, in _inner_hf_hub_download
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return hf_hub_download(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return fn(*args, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1202, in hf_hub_download
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return _hf_hub_download_to_local_dir(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1487, in _hf_hub_download_to_local_dir
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     _download_to_tmp_and_move(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1884, in _download_to_tmp_and_move
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     http_get(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 459, in http_get
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     r = _request_wrapper(
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 395, in _request_wrapper
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     response = get_session().request(method=method, url=url, **params)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     resp = self.send(prep, **send_kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     r = adapter.send(request, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 66, in send
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     return super().send(request, *args, **kwargs)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 713, in send
zonemercy-vingt-deux-v11-5332-v2-mkmlizer:     raise ReadTimeout(e, request=request)
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 61dcad66-f110-4f88-8ae4-98b5e8c32c5c)')
Job zonemercy-vingt-deux-v11-5332-v2-mkmlizer completed after 99.46s with status: failed
Stopping job with name zonemercy-vingt-deux-v11-5332-v2-mkmlizer
%s, retrying in %s seconds...
Starting job with name zonemercy-vingt-deux-v11-5332-v2-mkmlizer
Waiting for job on zonemercy-vingt-deux-v11-5332-v2-mkmlizer to finish
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Downloaded to shared memory in 56.196s
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpsv14ba26, device:0
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: quantized model in 43.494s
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Processed model zonemercy/Vingt-Deux-v11k256128-22b01e5 in 99.690s
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/config.json
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/special_tokens_map.json
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/tokenizer_config.json
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/tokenizer.json
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/flywheel_model.1.safetensors
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v11-5332-v2/flywheel_model.0.safetensors
zonemercy-vingt-deux-v11-5332-v2-mkmlizer: 
Loading 0:   0%|          | 0/507 [00:00<?, ?it/s]
Loading 0:   1%|          | 4/507 [00:00<00:12, 39.67it/s]
Loading 0:   2%|▏         | 8/507 [00:00<00:16, 29.51it/s]
Loading 0:   3%|▎         | 13/507 [00:00<00:14, 33.86it/s]
Loading 0:   3%|▎         | 17/507 [00:00<00:13, 35.42it/s]
Loading 0:   4%|▍         | 22/507 [00:00<00:12, 37.77it/s]
Loading 0:   5%|▌         | 26/507 [00:00<00:12, 38.39it/s]
Loading 0:   6%|▌         | 31/507 [00:00<00:12, 39.52it/s]
Loading 0:   7%|▋         | 35/507 [00:00<00:12, 38.95it/s]
Loading 0:   8%|▊         | 40/507 [00:01<00:11, 39.95it/s]
Loading 0:   9%|▉         | 45/507 [00:01<00:10, 42.46it/s]
Loading 0:  10%|▉         | 50/507 [00:01<00:12, 35.59it/s]
Loading 0:  11%|█         | 54/507 [00:01<00:16, 27.22it/s]
Loading 0:  11%|█▏        | 58/507 [00:01<00:15, 28.51it/s]
Loading 0:  12%|█▏        | 63/507 [00:01<00:13, 31.98it/s]
Loading 0:  13%|█▎        | 67/507 [00:01<00:13, 32.59it/s]
Loading 0:  14%|█▍        | 73/507 [00:02<00:12, 34.15it/s]
Loading 0:  16%|█▌        | 80/507 [00:02<00:11, 37.00it/s]
Loading 0:  17%|█▋        | 87/507 [00:02<00:10, 41.86it/s]
Loading 0:  18%|█▊        | 92/507 [00:02<00:10, 40.79it/s]
Loading 0:  19%|█▉        | 97/507 [00:02<00:10, 40.20it/s]
Loading 0:  20%|██        | 102/507 [00:02<00:09, 40.90it/s]
Loading 0:  21%|██        | 107/507 [00:02<00:11, 34.68it/s]
Loading 0:  22%|██▏       | 113/507 [00:03<00:12, 30.44it/s]
Loading 0:  23%|██▎       | 117/507 [00:03<00:13, 29.45it/s]
Loading 0:  24%|██▍       | 122/507 [00:03<00:13, 28.92it/s]
Loading 0:  25%|██▌       | 127/507 [00:03<00:11, 32.92it/s]
Loading 0:  26%|██▌       | 131/507 [00:03<00:12, 31.02it/s]
Loading 0:  27%|██▋       | 138/507 [00:03<00:09, 38.49it/s]
Loading 0:  28%|██▊       | 143/507 [00:04<00:09, 38.67it/s]
Loading 0:  29%|██▉       | 148/507 [00:04<00:09, 39.42it/s]
Loading 0:  30%|███       | 153/507 [00:04<00:08, 40.88it/s]
Loading 0:  31%|███       | 158/507 [00:04<00:10, 34.09it/s]
Loading 0:  32%|███▏      | 164/507 [00:04<00:08, 39.42it/s]
Loading 0:  33%|███▎      | 169/507 [00:04<00:10, 31.56it/s]
Loading 0:  34%|███▍      | 173/507 [00:04<00:10, 32.32it/s]
Loading 0:  35%|███▍      | 177/507 [00:05<00:10, 31.20it/s]
Loading 0:  36%|███▌      | 183/507 [00:05<00:08, 36.30it/s]
Loading 0:  37%|███▋      | 187/507 [00:05<00:08, 35.86it/s]
Loading 0:  38%|███▊      | 192/507 [00:05<00:08, 38.67it/s]
Loading 0:  39%|███▉      | 197/507 [00:05<00:08, 38.51it/s]
Loading 0:  40%|███▉      | 202/507 [00:05<00:07, 39.23it/s]
Loading 0:  41%|████      | 207/507 [00:05<00:07, 40.18it/s]
Loading 0:  42%|████▏     | 212/507 [00:05<00:08, 33.68it/s]
Loading 0:  43%|████▎     | 218/507 [00:06<00:07, 38.53it/s]
Loading 0:  44%|████▍     | 223/507 [00:06<00:08, 34.94it/s]
Loading 0:  45%|████▍     | 227/507 [00:06<00:09, 30.07it/s]
Loading 0:  46%|████▌     | 231/507 [00:06<00:09, 29.40it/s]
Loading 0:  46%|████▋     | 235/507 [00:06<00:08, 31.56it/s]
Loading 0:  47%|████▋     | 239/507 [00:06<00:09, 29.75it/s]
Loading 0:  48%|████▊     | 244/507 [00:06<00:07, 33.30it/s]
Loading 0:  49%|████▉     | 248/507 [00:07<00:08, 29.76it/s]
Loading 0:  50%|█████     | 255/507 [00:07<00:06, 36.36it/s]
Loading 0:  51%|█████     | 259/507 [00:07<00:07, 35.21it/s]
Loading 0:  52%|█████▏    | 264/507 [00:07<00:06, 36.63it/s]
Loading 0:  53%|█████▎    | 268/507 [00:07<00:06, 34.66it/s]
Loading 0:  54%|█████▍    | 273/507 [00:07<00:06, 36.62it/s]
Loading 0:  55%|█████▍    | 277/507 [00:07<00:06, 34.68it/s]
Loading 0:  56%|█████▌    | 283/507 [00:08<00:06, 35.70it/s]
Loading 0:  57%|█████▋    | 287/507 [00:08<00:09, 22.95it/s]
Loading 0:  58%|█████▊    | 293/507 [00:08<00:08, 26.34it/s]
Loading 0:  59%|█████▉    | 299/507 [00:23<00:07, 26.34it/s]
Loading 0:  59%|█████▉    | 300/507 [00:23<02:43,  1.27it/s]
Loading 0:  60%|█████▉    | 302/507 [00:23<02:22,  1.44it/s]
Loading 0:  61%|██████    | 307/507 [00:23<01:36,  2.07it/s]
Loading 0:  61%|██████    | 310/507 [00:23<01:16,  2.57it/s]
Loading 0:  62%|██████▏   | 313/507 [00:23<00:59,  3.27it/s]
Loading 0:  63%|██████▎   | 318/507 [00:23<00:38,  4.88it/s]
Loading 0:  64%|██████▎   | 322/507 [00:23<00:28,  6.48it/s]
Loading 0:  64%|██████▍   | 327/507 [00:23<00:19,  9.15it/s]
Loading 0:  65%|██████▌   | 331/507 [00:24<00:15, 11.43it/s]
Loading 0:  66%|██████▌   | 335/507 [00:24<00:12, 14.05it/s]
Loading 0:  67%|██████▋   | 340/507 [00:24<00:11, 14.71it/s]
Loading 0:  68%|██████▊   | 343/507 [00:24<00:10, 16.20it/s]
Loading 0:  68%|██████▊   | 347/507 [00:24<00:08, 18.28it/s]
Loading 0:  70%|██████▉   | 354/507 [00:24<00:05, 25.89it/s]
Loading 0:  71%|███████   | 358/507 [00:25<00:05, 26.71it/s]
Loading 0:  72%|███████▏  | 363/507 [00:25<00:04, 30.67it/s]
Loading 0:  72%|███████▏  | 367/507 [00:25<00:04, 30.49it/s]
Loading 0:  73%|███████▎  | 372/507 [00:25<00:04, 33.62it/s]
Loading 0:  74%|███████▍  | 376/507 [00:25<00:03, 33.24it/s]
Loading 0:  75%|███████▌  | 381/507 [00:25<00:03, 36.56it/s]
Loading 0:  76%|███████▌  | 385/507 [00:25<00:03, 36.09it/s]
Loading 0:  77%|███████▋  | 389/507 [00:25<00:03, 36.39it/s]
Loading 0:  78%|███████▊  | 393/507 [00:25<00:03, 35.63it/s]
Loading 0:  78%|███████▊  | 397/507 [00:26<00:04, 25.31it/s]
Loading 0:  79%|███████▉  | 401/507 [00:26<00:04, 25.86it/s]
Loading 0:  80%|████████  | 408/507 [00:26<00:02, 33.31it/s]
Loading 0:  81%|████████▏ | 412/507 [00:26<00:02, 33.03it/s]
Loading 0:  82%|████████▏ | 417/507 [00:26<00:02, 35.39it/s]
Loading 0:  83%|████████▎ | 421/507 [00:26<00:02, 34.33it/s]
Loading 0:  84%|████████▍ | 426/507 [00:26<00:02, 36.39it/s]
Loading 0:  85%|████████▍ | 430/507 [00:27<00:02, 33.80it/s]
Loading 0:  86%|████████▌ | 435/507 [00:27<00:01, 36.46it/s]
Loading 0:  87%|████████▋ | 439/507 [00:27<00:01, 35.31it/s]
Loading 0:  88%|████████▊ | 444/507 [00:27<00:01, 37.98it/s]
Loading 0:  88%|████████▊ | 448/507 [00:27<00:01, 36.70it/s]
Loading 0:  89%|████████▉ | 453/507 [00:27<00:01, 39.87it/s]
Loading 0:  90%|█████████ | 458/507 [00:30<00:08,  6.05it/s]
Loading 0:  91%|█████████ | 461/507 [00:30<00:06,  7.27it/s]
Loading 0:  92%|█████████▏| 465/507 [00:30<00:04,  9.18it/s]
Loading 0:  93%|█████████▎| 472/507 [00:30<00:02, 14.02it/s]
Loading 0:  94%|█████████▍| 476/507 [00:30<00:01, 16.22it/s]
Loading 0:  95%|█████████▍| 481/507 [00:30<00:01, 20.23it/s]
Loading 0:  96%|█████████▌| 485/507 [00:30<00:00, 22.25it/s]
Loading 0:  97%|█████████▋| 490/507 [00:30<00:00, 26.06it/s]
Loading 0:  97%|█████████▋| 494/507 [00:31<00:00, 27.55it/s]
Loading 0:  98%|█████████▊| 499/507 [00:31<00:00, 31.05it/s]
Loading 0:  99%|█████████▉| 503/507 [00:31<00:00, 31.36it/s]
                                                            
Job zonemercy-vingt-deux-v11-5332-v2-mkmlizer completed after 125.94s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v11-5332-v2-mkmlizer
Pipeline stage MKMLizer completed in 227.22s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v11-5332-v2
Waiting for inference service zonemercy-vingt-deux-v11-5332-v2 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service zonemercy-vingt-deux-v11-5332-v2 ready after 130.9141674041748s
Pipeline stage MKMLDeployer completed in 131.54s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.080044984817505s
Received healthy response to inference request in 2.2467446327209473s
Received healthy response to inference request in 2.5133631229400635s
Received healthy response to inference request in 1.6739177703857422s
5 requests
1 failed requests
5th percentile: 1.7884831428527832
10th percentile: 1.9030485153198242
20th percentile: 2.1321792602539062
30th percentile: 2.3000683307647707
40th percentile: 2.406715726852417
50th percentile: 2.5133631229400635
60th percentile: 2.74003586769104
70th percentile: 2.9667086124420163
80th percentile: 6.501029396057132
90th percentile: 13.34299821853638
95th percentile: 16.763982629776
99th percentile: 19.5007701587677
mean time: 5.939807510375976
%s, retrying in %s seconds...
Received healthy response to inference request in 2.3892900943756104s
Received healthy response to inference request in 2.4101829528808594s
Received healthy response to inference request in 2.3113112449645996s
Received healthy response to inference request in 1.698263168334961s
Received healthy response to inference request in 2.110954761505127s
5 requests
0 failed requests
5th percentile: 1.780801486968994
10th percentile: 1.8633398056030273
20th percentile: 2.0284164428710936
30th percentile: 2.1510260581970213
40th percentile: 2.2311686515808105
50th percentile: 2.3113112449645996
60th percentile: 2.342502784729004
70th percentile: 2.373694324493408
80th percentile: 2.39346866607666
90th percentile: 2.4018258094787597
95th percentile: 2.4060043811798097
99th percentile: 2.4093472385406494
mean time: 2.1840004444122316
Pipeline stage StressChecker completed in 44.36s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.47s
Shutdown handler de-registered
zonemercy-vingt-deux-v11_5332_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3301.15s
Shutdown handler de-registered
zonemercy-vingt-deux-v11_5332_v2 status is now inactive due to auto deactivation removed underperforming models
zonemercy-vingt-deux-v11_5332_v2 status is now torndown due to DeploymentManager action