chaser-cz-qlora-out

submission_id: chaser-cz-qlora-out_v3
developer_uid: chace9580
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
model_name: chaser-cz-qlora-out_v3
model_repo: Chaser-cz/qlora-out
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
timestamp: 2024-06-29T19:10:29+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaser-cz-qlora-out-v3-mkmlizer
Waiting for job on chaser-cz-qlora-out-v3-mkmlizer to finish
chaser-cz-qlora-out-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaser-cz-qlora-out-v3-mkmlizer: ║     _____            __           __                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaser-cz-qlora-out-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaser-cz-qlora-out-v3-mkmlizer: ║       /___/                                                         ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaser-cz-qlora-out-v3-mkmlizer: ║  https://mk1.ai                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
chaser-cz-qlora-out-v3-mkmlizer: ║  belonging to:                                                      ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Chai Research Corp.                                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaser-cz-qlora-out-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaser-cz-qlora-out-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
chaser-cz-qlora-out-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaser-cz-qlora-out-v3-mkmlizer: ║     _____            __           __                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaser-cz-qlora-out-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaser-cz-qlora-out-v3-mkmlizer: ║       /___/                                                         ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaser-cz-qlora-out-v3-mkmlizer: ║  https://mk1.ai                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
chaser-cz-qlora-out-v3-mkmlizer: ║  belonging to:                                                      ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Chai Research Corp.                                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaser-cz-qlora-out-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaser-cz-qlora-out-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
Job chaser-cz-qlora-out-v3-mkmlizer completed after 21.68s with status: failed
Stopping job with name chaser-cz-qlora-out-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaser-cz-qlora-out-v3-mkmlizer
Waiting for job on chaser-cz-qlora-out-v3-mkmlizer to finish
chaser-cz-qlora-out-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaser-cz-qlora-out-v3-mkmlizer: ║     _____            __           __                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaser-cz-qlora-out-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaser-cz-qlora-out-v3-mkmlizer: ║       /___/                                                         ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Version: 0.8.14                                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaser-cz-qlora-out-v3-mkmlizer: ║  https://mk1.ai                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
chaser-cz-qlora-out-v3-mkmlizer: ║  belonging to:                                                      ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Chai Research Corp.                                                ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaser-cz-qlora-out-v3-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaser-cz-qlora-out-v3-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaser-cz-qlora-out-v3-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaser-cz-qlora-out-v3-mkmlizer:   warnings.warn(warning_message, FutureWarning)
chaser-cz-qlora-out-v3-mkmlizer: Downloaded to shared memory in 1.771s
chaser-cz-qlora-out-v3-mkmlizer: quantizing model to /dev/shm/model_cache
chaser-cz-qlora-out-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaser-cz-qlora-out-v3-mkmlizer: Traceback (most recent call last):
chaser-cz-qlora-out-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
chaser-cz-qlora-out-v3-mkmlizer:     cli()
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
chaser-cz-qlora-out-v3-mkmlizer:     return self.main(*args, **kwargs)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
chaser-cz-qlora-out-v3-mkmlizer:     rv = self.invoke(ctx)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
chaser-cz-qlora-out-v3-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
chaser-cz-qlora-out-v3-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
chaser-cz-qlora-out-v3-mkmlizer:     return __callback(*args, **kwargs)
chaser-cz-qlora-out-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
chaser-cz-qlora-out-v3-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
chaser-cz-qlora-out-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
chaser-cz-qlora-out-v3-mkmlizer:     flywheel.instrument(
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 65, in instrument
chaser-cz-qlora-out-v3-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
chaser-cz-qlora-out-v3-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 38, in save_st_pretrained
chaser-cz-qlora-out-v3-mkmlizer:     for name, tensor in model_iterator:
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 213, in tensor_compiler
chaser-cz-qlora-out-v3-mkmlizer:     for name, tensor in tensor_iterator:
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 196, in tensor_slicer
chaser-cz-qlora-out-v3-mkmlizer:     for name, tensor_slice in tensor_iterator:
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 54, in hfpt_model_iterator
chaser-cz-qlora-out-v3-mkmlizer:     tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
chaser-cz-qlora-out-v3-mkmlizer:     with _open_file_like(f, 'rb') as opened_file:
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
chaser-cz-qlora-out-v3-mkmlizer:     return _open_file(name_or_buffer, mode)
chaser-cz-qlora-out-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
chaser-cz-qlora-out-v3-mkmlizer:     super().__init__(open(name, mode))
chaser-cz-qlora-out-v3-mkmlizer: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmps88vmuv_/pytorch_model-00001-of-00003.bin'
Job chaser-cz-qlora-out-v3-mkmlizer completed after 12.81s with status: failed
Stopping job with name chaser-cz-qlora-out-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaser-cz-qlora-out-v3-mkmlizer
Waiting for job on chaser-cz-qlora-out-v3-mkmlizer to finish
Job chaser-cz-qlora-out-v3-mkmlizer completed after 10.54s with status: failed
Stopping job with name chaser-cz-qlora-out-v3-mkmlizer
MKMLizerError('')
chaser-cz-qlora-out_v3 status is now failed due to DeploymentManager action
admin requested tearing down of chaser-cz-qlora-out_v3
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.20s
Running pipeline stage MKMLModelDeleter
Skipping deletion as no model was successfully uploaded
Pipeline stage MKMLModelDeleter completed in 0.71s
chaser-cz-qlora-out_v3 status is now torndown due to DeploymentManager action