chaiml-snugstable1-5-1_59274

developer_uid: zonemercy
submission_id: chaiml-snugstable1-5-1_59274_v25
model_name: chaiml-snugstable1-5-1_59274_v25
model_group: ChaiML/SnugStable1.5-12b
status: torndown
timestamp: 2025-03-21T21:51:10+00:00
num_battles: 8386
num_wins: 4045
celo_rating: 1267.0
family_friendly_score: 0.4828
family_friendly_standard_error: 0.00706688276399149
submission_type: basic
model_repo: ChaiML/SnugStable1.5-12b0-l1e5r7
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.6088544543343235, 'latency_mean': 1.6423554122447968, 'latency_p50': 1.6442805528640747, 'latency_p90': 1.8104100465774535}, {'batch_size': 3, 'throughput': 1.1316257644832233, 'latency_mean': 2.6484584438800813, 'latency_p50': 2.6554324626922607, 'latency_p90': 2.8982133865356445}, {'batch_size': 5, 'throughput': 1.358748639276556, 'latency_mean': 3.6596515130996705, 'latency_p50': 3.6842657327651978, 'latency_p90': 4.074724960327148}, {'batch_size': 6, 'throughput': 1.4342525681108431, 'latency_mean': 4.166365143060684, 'latency_p50': 4.170834064483643, 'latency_p90': 4.703719115257263}, {'batch_size': 8, 'throughput': 1.5168593322041386, 'latency_mean': 5.237376655340195, 'latency_p50': 5.249892473220825, 'latency_p90': 5.973178791999817}, {'batch_size': 10, 'throughput': 1.5505200199935305, 'latency_mean': 6.403021677732467, 'latency_p50': 6.404962539672852, 'latency_p90': 7.170540475845336}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: chaiml-snugstable1-5-1_59274_v25
is_internal_developer: True
language_model: ChaiML/SnugStable1.5-12b0-l1e5r7
model_size: 13B
ranking_group: single
throughput_3p7s: 1.37
us_pacific_date: 2025-03-21
win_ratio: 0.4823515382780825
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['Bot:', '<|eot_id|>', '\n', 'You:', '</s>', 'User:', '####', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-snugstable1-5-1-59274-v25-mkmlizer
Waiting for job on chaiml-snugstable1-5-1-59274-v25-mkmlizer to finish
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║     _____            __           __                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║       /___/                                                         ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Version: 0.12.8                                                    ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  belonging to:                                                      ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Traceback (most recent call last):
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     response = conn.getresponse()
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     httplib_response = super().getresponse()
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 1375, in getresponse
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     response.begin()
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 318, in begin
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     version, status, reason = self._read_status()
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/http/client.py", line 279, in _read_status
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/socket.py", line 705, in readinto
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return self._sock.recv_into(b)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/ssl.py", line 1307, in recv_into
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return self.read(nbytes, buffer)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/ssl.py", line 1163, in read
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return self._sslobj.read(len, buffer)
chaiml-snugstable1-5-1-59274-v25-mkmlizer: TimeoutError: The read operation timed out
chaiml-snugstable1-5-1-59274-v25-mkmlizer: The above exception was the direct cause of the following exception:
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Traceback (most recent call last):
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     resp = conn.urlopen(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     retries = retries.increment(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     raise reraise(type(error), error, _stacktrace)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     raise value
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     response = self._make_request(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     raise ReadTimeoutError(
chaiml-snugstable1-5-1-59274-v25-mkmlizer: urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)
chaiml-snugstable1-5-1-59274-v25-mkmlizer: During handling of the above exception, another exception occurred:
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Traceback (most recent call last):
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/code/uploading/mkmlize.py", line 193, in <module>
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     cli()
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return self.main(*args, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     rv = self.invoke(ctx)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return __callback(*args, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/code/uploading/mkmlize.py", line 39, in quantize
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/code/uploading/mkmlize.py", line 86, in download_to_shared_memory
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     snapshot_download(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return fn(*args, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 294, in snapshot_download
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     _inner_hf_hub_download(file)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 270, in _inner_hf_hub_download
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return hf_hub_download(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return fn(*args, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 840, in hf_hub_download
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return _hf_hub_download_to_local_dir(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1136, in _hf_hub_download_to_local_dir
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     _download_to_tmp_and_move(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1543, in _download_to_tmp_and_move
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     http_get(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 369, in http_get
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     r = _request_wrapper(
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 301, in _request_wrapper
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     response = get_session().request(method=method, url=url, **params)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     resp = self.send(prep, **send_kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     r = adapter.send(request, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 93, in send
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     return super().send(request, *args, **kwargs)
chaiml-snugstable1-5-1-59274-v25-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 713, in send
chaiml-snugstable1-5-1-59274-v25-mkmlizer:     raise ReadTimeout(e, request=request)
chaiml-snugstable1-5-1-59274-v25-mkmlizer: requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 7c8d289a-fff5-4328-b626-57e5f6f49bde)')
Job chaiml-snugstable1-5-1-59274-v25-mkmlizer completed after 37.19s with status: failed
Stopping job with name chaiml-snugstable1-5-1-59274-v25-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-snugstable1-5-1-59274-v25-mkmlizer
Waiting for job on chaiml-snugstable1-5-1-59274-v25-mkmlizer to finish
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║     _____            __           __                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║       /___/                                                         ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Version: 0.12.8                                                    ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  belonging to:                                                      ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ║                                                                     ║
chaiml-snugstable1-5-1-59274-v25-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Downloaded to shared memory in 28.999s
chaiml-snugstable1-5-1-59274-v25-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpe0rho77v, device:0
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-snugstable1-5-1-59274-v25-mkmlizer: quantized model in 35.116s
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Processed model ChaiML/SnugStable1.5-12b0-l1e5r7 in 64.120s
chaiml-snugstable1-5-1-59274-v25-mkmlizer: creating bucket guanaco-mkml-models
chaiml-snugstable1-5-1-59274-v25-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-snugstable1-5-1-59274-v25-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-snugstable1-5-1-59274-v25
chaiml-snugstable1-5-1-59274-v25-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-snugstable1-5-1-59274-v25/tokenizer_config.json
chaiml-snugstable1-5-1-59274-v25-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-snugstable1-5-1-59274-v25/tokenizer.json
chaiml-snugstable1-5-1-59274-v25-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-snugstable1-5-1-59274-v25/flywheel_model.0.safetensors
chaiml-snugstable1-5-1-59274-v25-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 28.61it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 48.49it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.13it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 42.19it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 48.52it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 45.87it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 44.72it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 50.20it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 47.17it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 36.74it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 36.83it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 41.19it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 42.66it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:06, 43.00it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:05, 48.65it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:05, 47.49it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:05, 45.44it/s]
Loading 0:  30%|██▉       | 108/363 [00:02<00:04, 51.03it/s]
Loading 0:  31%|███▏      | 114/363 [00:02<00:05, 46.48it/s]
Loading 0:  33%|███▎      | 119/363 [00:02<00:05, 45.57it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:04, 50.55it/s]
Loading 0:  36%|███▋      | 132/363 [00:02<00:04, 48.44it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:04, 47.76it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 35.22it/s]
Loading 0:  40%|████      | 147/363 [00:03<00:06, 35.24it/s]
Loading 0:  42%|████▏     | 151/363 [00:03<00:05, 35.82it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 40.55it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:04, 41.12it/s]
Loading 0:  46%|████▋     | 168/363 [00:03<00:04, 41.74it/s]
Loading 0:  48%|████▊     | 175/363 [00:03<00:03, 47.65it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:03, 46.85it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:03, 45.33it/s]
Loading 0:  53%|█████▎    | 192/363 [00:04<00:03, 48.67it/s]
Loading 0:  54%|█████▍    | 197/363 [00:04<00:03, 48.26it/s]
Loading 0:  56%|█████▌    | 202/363 [00:04<00:03, 48.45it/s]
Loading 0:  57%|█████▋    | 208/363 [00:04<00:03, 47.33it/s]
Loading 0:  59%|█████▊    | 213/363 [00:04<00:03, 45.25it/s]
Loading 0:  61%|██████    | 220/363 [00:04<00:02, 51.26it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:04, 33.47it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 35.20it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 41.32it/s]
Loading 0:  67%|██████▋   | 244/363 [00:05<00:02, 42.38it/s]
Loading 0:  69%|██████▊   | 249/363 [00:05<00:02, 42.69it/s]
Loading 0:  71%|███████   | 256/363 [00:05<00:02, 46.92it/s]
Loading 0:  72%|███████▏  | 261/363 [00:05<00:02, 46.78it/s]
Loading 0:  73%|███████▎  | 266/363 [00:06<00:02, 41.19it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 48.22it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 45.38it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 44.94it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 48.72it/s]
Loading 0:  82%|████████▏ | 298/363 [00:06<00:01, 46.94it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:20,  2.86it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:15,  3.58it/s]
Loading 0:  86%|████████▌ | 312/363 [00:13<00:11,  4.57it/s]
Loading 0:  88%|████████▊ | 320/363 [00:13<00:05,  7.36it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:03,  9.85it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.43it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 17.27it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 21.04it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 24.40it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 30.74it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 32.78it/s]
                                                            
Job chaiml-snugstable1-5-1-59274-v25-mkmlizer completed after 132.98s with status: succeeded
Stopping job with name chaiml-snugstable1-5-1-59274-v25-mkmlizer
Pipeline stage MKMLizer completed in 171.30s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-snugstable1-5-1-59274-v25
Waiting for inference service chaiml-snugstable1-5-1-59274-v25 to be ready
Inference service chaiml-snugstable1-5-1-59274-v25 ready after 90.43313264846802s
Pipeline stage MKMLDeployer completed in 91.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.997215747833252s
Received healthy response to inference request in 1.5889499187469482s
Received healthy response to inference request in 1.6031832695007324s
Received healthy response to inference request in 1.5778172016143799s
Received healthy response to inference request in 1.7569911479949951s
5 requests
0 failed requests
5th percentile: 1.5800437450408935
10th percentile: 1.5822702884674071
20th percentile: 1.5867233753204346
30th percentile: 1.591796588897705
40th percentile: 1.5974899291992188
50th percentile: 1.6031832695007324
60th percentile: 1.6647064208984375
70th percentile: 1.7262295722961425
80th percentile: 1.8050360679626465
90th percentile: 1.9011259078979492
95th percentile: 1.9491708278656006
99th percentile: 1.9876067638397217
mean time: 1.7048314571380616
Pipeline stage StressChecker completed in 10.43s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.81s
Shutdown handler de-registered
chaiml-snugstable1-5-1_59274_v25 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 4477.28s
Shutdown handler de-registered
chaiml-snugstable1-5-1_59274_v25 status is now inactive due to auto deactivation removed underperforming models
chaiml-snugstable1-5-1_59274_v25 status is now torndown due to DeploymentManager action