chaiml-arataki-itto-roo_94227

developer_uid: chai_backend_admin
submission_id: chaiml-arataki-itto-roo_94227_v1
model_name: chaiml-arataki-itto-roo_94227_v1
model_group: ChaiML/Arataki-Itto-Room
status: torndown
timestamp: 2025-07-15T21:53:09+00:00
num_battles: 5594
num_wins: 2880
celo_rating: 1298.2
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/Arataki-Itto-Roomate250715143209_sft
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.5224375484071377, 'latency_mean': 1.913982208967209, 'latency_p50': 1.8962281942367554, 'latency_p90': 2.1446402311325072}, {'batch_size': 3, 'throughput': 1.0603521289217073, 'latency_mean': 2.811207271814346, 'latency_p50': 2.8176404237747192, 'latency_p90': 3.094263792037964}, {'batch_size': 5, 'throughput': 1.3493654447193473, 'latency_mean': 3.6860023605823518, 'latency_p50': 3.689338207244873, 'latency_p90': 4.120637106895447}, {'batch_size': 6, 'throughput': 1.457466691724659, 'latency_mean': 4.079384409189224, 'latency_p50': 4.037782311439514, 'latency_p90': 4.668158102035522}, {'batch_size': 8, 'throughput': 1.5863620393029736, 'latency_mean': 5.008737410306931, 'latency_p50': 5.002290725708008, 'latency_p90': 5.609381699562073}, {'batch_size': 10, 'throughput': 1.6750860263464906, 'latency_mean': 5.907668904066086, 'latency_p50': 5.894562482833862, 'latency_p90': 6.611048722267151}]
gpu_counts: {'NVIDIA A100-SXM4-80GB': 1}
display_name: chaiml-arataki-itto-roo_94227_v1
is_internal_developer: True
language_model: ChaiML/Arataki-Itto-Roomate250715143209_sft
model_size: 24B
ranking_group: single
throughput_3p7s: 1.36
us_pacific_date: 2025-07-15
win_ratio: 0.5148373257061137
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '####\n', '\n', 'You:', '####'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}</s>\n', 'user_template': 'You: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-arataki-itto-roo-94227-v1-mkmlizer
Waiting for job on chaiml-arataki-itto-roo-94227-v1-mkmlizer to finish
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Version: 0.29.15                                                   ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Traceback (most recent call last):
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 203, in _new_conn
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     sock = connection.create_connection(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/socket.py", line 955, in getaddrinfo
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
chaiml-arataki-itto-roo-94227-v1-mkmlizer: socket.gaierror: [Errno -3] Temporary failure in name resolution
chaiml-arataki-itto-roo-94227-v1-mkmlizer: The above exception was the direct cause of the following exception:
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Traceback (most recent call last):
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     response = self._make_request(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     raise new_e
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     self._validate_conn(conn)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     conn.connect()
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 611, in connect
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     self.sock = sock = self._new_conn()
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connection.py", line 210, in _new_conn
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     raise NameResolutionError(self.host, self, e) from e
chaiml-arataki-itto-roo-94227-v1-mkmlizer: urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPSConnection object at 0x7538ee3d2f50>: Failed to resolve 'cdn-lfs-us-1.hf.co' ([Errno -3] Temporary failure in name resolution)
chaiml-arataki-itto-roo-94227-v1-mkmlizer: The above exception was the direct cause of the following exception:
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Traceback (most recent call last):
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     resp = conn.urlopen(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     retries = retries.increment(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
chaiml-arataki-itto-roo-94227-v1-mkmlizer: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='cdn-lfs-us-1.hf.co', port=443): Max retries exceeded with url: /repos/9c/fd/4943cd7821cf9833b91f3/70fbb478919cbccba27de45e2cf4938189254dc39c59cba0369dcb20963e6488?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00010.safetensors%3B+filename%3D%22model-00001-of-00010.safetensors%22%3B&Expires=1752620127&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc1MjYyMDEyN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzljL2ZkLzljZmQ2YWQ5N2UzMjk5MzcyNGQxMjJlOTg0ZWEwM2VlNzNiNmZhMmQ1OTc0OTQzY2Q3ODIxY2Y5ODMzYjkxZjMvNzBmYmI0Nzg5MTljYmNjYmEyN2RlNDVlMmNmNDkzODE4OTI1NGRjMzljNTljYmEwMzY5ZGNiMjA5NjNlNjQ4OD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=FzAXk3CodT8OuImCmFuXoPT1ZdkRLz6xjsaxsWdBuXbJ0wGymbrnU-ngJ3gSwMkSmPo6C~TOzHcv1gfI2OQVTmDX0TrjdWgkLyZAov17uKXeNlMOMYbTd6xj5Dof~3ONjhfNG0NKmfkdqAuGEaeXXFfP5IQ8TbnfM9Blvrkcrshq-tiM9r3yvF6MAWloos3NKQO1uGBWDqqt6W2QQMIb47BXERXnfKqPSstOe5TKqO3JtXk7ociNiOdcC-tZA0iwTPpumPGzpylKJyKHVtVkO~o5eKhCVbSRCxE-7aIAjq9AGHyNnd8~GeB-WFsYOA05tFyQzTeVXS0GeEiPjeBvrQ__&Key-Pair-Id=K24J24Z295AEI9 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7538ee3d2f50>: Failed to resolve 'cdn-lfs-us-1.hf.co' ([Errno -3] Temporary failure in name resolution)"))
chaiml-arataki-itto-roo-94227-v1-mkmlizer: During handling of the above exception, another exception occurred:
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Traceback (most recent call last):
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 225, in <module>
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     cli()
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return self.main(*args, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     rv = self.invoke(ctx)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return __callback(*args, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 41, in quantize
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/code/uploading/mkmlize.py", line 91, in download_to_shared_memory
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     snapshot_download(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return fn(*args, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 294, in snapshot_download
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     _inner_hf_hub_download(file)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 270, in _inner_hf_hub_download
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return hf_hub_download(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return fn(*args, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 941, in hf_hub_download
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return _hf_hub_download_to_local_dir(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1241, in _hf_hub_download_to_local_dir
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     _download_to_tmp_and_move(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1675, in _download_to_tmp_and_move
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     http_get(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 376, in http_get
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     r = _request_wrapper(
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 308, in _request_wrapper
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     response = get_session().request(method=method, url=url, **params)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     resp = self.send(prep, **send_kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     r = adapter.send(request, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 96, in send
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     return super().send(request, *args, **kwargs)
chaiml-arataki-itto-roo-94227-v1-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
chaiml-arataki-itto-roo-94227-v1-mkmlizer:     raise ConnectionError(e, request=request)
chaiml-arataki-itto-roo-94227-v1-mkmlizer: requests.exceptions.ConnectionError: (MaxRetryError('HTTPSConnectionPool(host=\'cdn-lfs-us-1.hf.co\', port=443): Max retries exceeded with url: /repos/9c/fd/4943cd7821cf9833b91f3/70fbb478919cbccba27de45e2cf4938189254dc39c59cba0369dcb20963e6488?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27model-00001-of-00010.safetensors%3B+filename%3D%22model-00001-of-00010.safetensors%22%3B&Expires=1752620127&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc1MjYyMDEyN319LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zLzljL2ZkLzljZmQ2YWQ5N2UzMjk5MzcyNGQxMjJlOTg0ZWEwM2VlNzNiNmZhMmQ1OTc0OTQzY2Q3ODIxY2Y5ODMzYjkxZjMvNzBmYmI0Nzg5MTljYmNjYmEyN2RlNDVlMmNmNDkzODE4OTI1NGRjMzljNTljYmEwMzY5ZGNiMjA5NjNlNjQ4OD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=FzAXk3CodT8OuImCmFuXoPT1ZdkRLz6xjsaxsWdBuXbJ0wGymbrnU-ngJ3gSwMkSmPo6C~TOzHcv1gfI2OQVTmDX0TrjdWgkLyZAov17uKXeNlMOMYbTd6xj5Dof~3ONjhfNG0NKmfkdqAuGEaeXXFfP5IQ8TbnfM9Blvrkcrshq-tiM9r3yvF6MAWloos3NKQO1uGBWDqqt6W2QQMIb47BXERXnfKqPSstOe5TKqO3JtXk7ociNiOdcC-tZA0iwTPpumPGzpylKJyKHVtVkO~o5eKhCVbSRCxE-7aIAjq9AGHyNnd8~GeB-WFsYOA05tFyQzTeVXS0GeEiPjeBvrQ__&Key-Pair-Id=K24J24Z295AEI9 (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7538ee3d2f50>: Failed to resolve \'cdn-lfs-us-1.hf.co\' ([Errno -3] Temporary failure in name resolution)"))'), '(Request ID: defb1150-86d2-4639-8cbf-4d8a5df46dd4)')
Job chaiml-arataki-itto-roo-94227-v1-mkmlizer completed after 161.36s with status: failed
Stopping job with name chaiml-arataki-itto-roo-94227-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-arataki-itto-roo-94227-v1-mkmlizer
Waiting for job on chaiml-arataki-itto-roo-94227-v1-mkmlizer to finish
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Version: 0.29.15                                                   ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ║                                                                     ║
chaiml-arataki-itto-roo-94227-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Downloaded to shared memory in 84.809s
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Checking if ChaiML/Arataki-Itto-Roomate250715143209_sft already exists in ChaiML
chaiml-arataki-itto-roo-94227-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpvppr0y6a, device:0
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-arataki-itto-roo-94227-v1-mkmlizer: quantized model in 47.808s
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Processed model ChaiML/Arataki-Itto-Roomate250715143209_sft in 132.618s
chaiml-arataki-itto-roo-94227-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-arataki-itto-roo-94227-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-arataki-itto-roo-94227-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/config.json
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/special_tokens_map.json
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/tokenizer_config.json
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/tokenizer.json
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/flywheel_model.1.safetensors
chaiml-arataki-itto-roo-94227-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-arataki-itto-roo-94227-v1/nvidia/flywheel_model.0.safetensors
chaiml-arataki-itto-roo-94227-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:17, 20.95it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:09, 36.99it/s]
Loading 0:   5%|▍         | 17/363 [00:00<00:10, 33.30it/s]
Loading 0:   6%|▌         | 21/363 [00:00<00:09, 34.20it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:11, 29.96it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:09, 36.18it/s]
Loading 0:  10%|▉         | 36/363 [00:01<00:14, 21.96it/s]
Loading 0:  11%|█         | 40/363 [00:01<00:13, 24.56it/s]
Loading 0:  12%|█▏        | 44/363 [00:01<00:12, 25.37it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:11, 27.41it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:12, 25.78it/s]
Loading 0:  16%|█▌        | 57/363 [00:02<00:10, 28.75it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:11, 27.31it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:10, 28.07it/s]
Loading 0:  19%|█▉        | 70/363 [00:02<00:11, 25.15it/s]
Loading 0:  20%|██        | 73/363 [00:02<00:13, 21.03it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:10, 26.50it/s]
Loading 0:  23%|██▎       | 82/363 [00:03<00:11, 25.48it/s]
Loading 0:  24%|██▎       | 86/363 [00:03<00:10, 26.53it/s]
Loading 0:  25%|██▍       | 89/363 [00:03<00:10, 26.35it/s]
Loading 0:  25%|██▌       | 92/363 [00:03<00:12, 20.87it/s]
Loading 0:  27%|██▋       | 99/363 [00:03<00:09, 28.07it/s]
Loading 0:  28%|██▊       | 103/363 [00:03<00:09, 26.55it/s]
Loading 0:  29%|██▉       | 107/363 [00:04<00:11, 22.69it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:09, 25.77it/s]
Loading 0:  31%|███▏      | 114/363 [00:04<00:10, 23.15it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 28.38it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 27.01it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 29.55it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:08, 27.87it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:07, 30.16it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:07, 28.08it/s]
Loading 0:  41%|████      | 148/363 [00:05<00:06, 34.66it/s]
Loading 0:  42%|████▏     | 152/363 [00:05<00:09, 23.29it/s]
Loading 0:  43%|████▎     | 155/363 [00:05<00:08, 23.38it/s]
Loading 0:  44%|████▎     | 158/363 [00:06<00:10, 19.76it/s]
Loading 0:  45%|████▍     | 163/363 [00:06<00:07, 25.02it/s]
Loading 0:  46%|████▌     | 167/363 [00:06<00:08, 22.50it/s]
Loading 0:  48%|████▊     | 174/363 [00:06<00:06, 29.12it/s]
Loading 0:  49%|████▉     | 178/363 [00:06<00:06, 27.66it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:06, 27.74it/s]
Loading 0:  52%|█████▏    | 187/363 [00:07<00:06, 25.43it/s]
Loading 0:  52%|█████▏    | 190/363 [00:07<00:07, 23.08it/s]
Loading 0:  53%|█████▎    | 193/363 [00:07<00:07, 23.94it/s]
Loading 0:  54%|█████▍    | 196/363 [00:07<00:07, 23.86it/s]
Loading 0:  55%|█████▌    | 200/363 [00:21<00:06, 23.86it/s]
Loading 0:  55%|█████▌    | 201/363 [00:21<03:00,  1.11s/it]
Loading 0:  56%|█████▌    | 203/363 [00:22<02:29,  1.07it/s]
Loading 0:  57%|█████▋    | 208/363 [00:22<01:30,  1.72it/s]
Loading 0:  58%|█████▊    | 211/363 [00:22<01:09,  2.19it/s]
Loading 0:  59%|█████▉    | 214/363 [00:22<00:51,  2.88it/s]
Loading 0:  60%|██████    | 218/363 [00:22<00:35,  4.10it/s]
Loading 0:  61%|██████    | 221/363 [00:22<00:27,  5.23it/s]
Loading 0:  62%|██████▏   | 224/363 [00:23<00:22,  6.29it/s]
Loading 0:  63%|██████▎   | 228/363 [00:23<00:15,  8.79it/s]
Loading 0:  64%|██████▎   | 231/363 [00:23<00:12, 10.21it/s]
Loading 0:  65%|██████▌   | 237/363 [00:23<00:08, 15.21it/s]
Loading 0:  66%|██████▌   | 240/363 [00:23<00:07, 15.99it/s]
Loading 0:  68%|██████▊   | 246/363 [00:23<00:05, 21.40it/s]
Loading 0:  69%|██████▉   | 250/363 [00:23<00:05, 22.08it/s]
Loading 0:  70%|███████   | 255/363 [00:24<00:04, 25.53it/s]
Loading 0:  71%|███████▏  | 259/363 [00:24<00:04, 25.35it/s]
Loading 0:  73%|███████▎  | 264/363 [00:24<00:03, 30.22it/s]
Loading 0:  74%|███████▍  | 268/363 [00:24<00:04, 23.50it/s]
Loading 0:  75%|███████▍  | 271/363 [00:24<00:04, 21.32it/s]
Loading 0:  75%|███████▌  | 274/363 [00:24<00:03, 22.31it/s]
Loading 0:  76%|███████▋  | 277/363 [00:25<00:03, 23.03it/s]
Loading 0:  78%|███████▊  | 282/363 [00:25<00:03, 26.67it/s]
Loading 0:  79%|███████▊  | 285/363 [00:25<00:03, 24.05it/s]
Loading 0:  80%|███████▉  | 289/363 [00:25<00:02, 26.98it/s]
Loading 0:  80%|████████  | 292/363 [00:25<00:02, 26.95it/s]
Loading 0:  81%|████████▏ | 295/363 [00:25<00:02, 26.18it/s]
Loading 0:  82%|████████▏ | 299/363 [00:25<00:02, 27.16it/s]
Loading 0:  84%|████████▎ | 304/363 [00:26<00:02, 24.40it/s]
Loading 0:  85%|████████▍ | 307/363 [00:26<00:02, 21.82it/s]
Loading 0:  85%|████████▌ | 310/363 [00:26<00:02, 22.97it/s]
Loading 0:  86%|████████▌ | 313/363 [00:26<00:02, 22.99it/s]
Loading 0:  88%|████████▊ | 318/363 [00:26<00:01, 25.84it/s]
Loading 0:  88%|████████▊ | 321/363 [00:26<00:01, 23.42it/s]
Loading 0:  90%|█████████ | 327/363 [00:26<00:01, 28.47it/s]
Loading 0:  91%|█████████ | 330/363 [00:27<00:01, 25.58it/s]
Loading 0:  92%|█████████▏| 335/363 [00:27<00:00, 28.37it/s]
Loading 0:  93%|█████████▎| 338/363 [00:27<00:00, 26.83it/s]
Loading 0:  94%|█████████▍| 341/363 [00:27<00:01, 15.97it/s]
Loading 0:  96%|█████████▌| 347/363 [00:27<00:00, 21.34it/s]
Loading 0:  96%|█████████▋| 350/363 [00:28<00:00, 22.01it/s]
Loading 0:  98%|█████████▊| 355/363 [00:28<00:00, 25.46it/s]
Loading 0:  99%|█████████▊| 358/363 [00:28<00:00, 23.56it/s]
                                                            
Job chaiml-arataki-itto-roo-94227-v1-mkmlizer completed after 168.72s with status: succeeded
Stopping job with name chaiml-arataki-itto-roo-94227-v1-mkmlizer
Pipeline stage MKMLizer completed in 331.38s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-arataki-itto-roo-94227-v1
Waiting for inference service chaiml-arataki-itto-roo-94227-v1 to be ready
Unable to record family friendly update due to error: HTTPConnectionPool(host='chaiml-nemo-guard-merged-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-arataki-itto-roo-94227-v1 ready after 261.33418011665344s
Pipeline stage MKMLDeployer completed in 261.92s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.206239700317383s
Received healthy response to inference request in 2.419921398162842s
Received healthy response to inference request in 2.0949795246124268s
Received healthy response to inference request in 2.2307729721069336s
5 requests
1 failed requests
5th percentile: 2.1221382141113283
10th percentile: 2.1492969036102294
20th percentile: 2.203614282608032
30th percentile: 2.268602657318115
40th percentile: 2.3442620277404784
50th percentile: 2.419921398162842
60th percentile: 2.734448719024658
70th percentile: 3.0489760398864743
80th percentile: 6.618074750900272
90th percentile: 13.44174485206604
95th percentile: 16.853579902648924
99th percentile: 19.583047943115233
mean time: 6.043465709686279
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4240834712982178s
Received healthy response to inference request in 2.0621840953826904s
Received healthy response to inference request in 2.69284987449646s
Received healthy response to inference request in 1.8853821754455566s
Received healthy response to inference request in 2.370664358139038s
5 requests
0 failed requests
5th percentile: 1.9207425594329834
10th percentile: 1.9561029434204102
20th percentile: 2.026823711395264
30th percentile: 2.12388014793396
40th percentile: 2.247272253036499
50th percentile: 2.370664358139038
60th percentile: 2.39203200340271
70th percentile: 2.413399648666382
80th percentile: 2.477836751937866
90th percentile: 2.5853433132171633
95th percentile: 2.6390965938568116
99th percentile: 2.6820992183685304
mean time: 2.2870327949523928
Pipeline stage StressChecker completed in 45.60s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.83s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.80s
Shutdown handler de-registered
chaiml-arataki-itto-roo_94227_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
chaiml-arataki-itto-roo_94227_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-arataki-itto-roo_94227_v1 status is now torndown due to DeploymentManager action
chaiml-arataki-itto-roo_94227_v1 status is now torndown due to DeploymentManager action
chaiml-arataki-itto-roo_94227_v1 status is now torndown due to DeploymentManager action