chaser-cz-qlora-out

submission_id: chaser-cz-qlora-out_v2
developer_uid: chace9580
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
model_name: chaser-cz-qlora-out_v2
model_repo: Chaser-cz/qlora-out
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/reward_gpt2_medium_preference_24m_e2
status: torndown
timestamp: 2024-06-29T06:56:04+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name chaser-cz-qlora-out-v2-mkmlizer
Waiting for job on chaser-cz-qlora-out-v2-mkmlizer to finish
chaser-cz-qlora-out-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaser-cz-qlora-out-v2-mkmlizer: ║     _____            __           __                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaser-cz-qlora-out-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaser-cz-qlora-out-v2-mkmlizer: ║       /___/                                                         ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Version: 0.8.14                                                    ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaser-cz-qlora-out-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaser-cz-qlora-out-v2-mkmlizer: ║  belonging to:                                                      ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaser-cz-qlora-out-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaser-cz-qlora-out-v2-mkmlizer:   warnings.warn(warning_message, FutureWarning)
chaser-cz-qlora-out-v2-mkmlizer: Downloaded to shared memory in 5.696s
chaser-cz-qlora-out-v2-mkmlizer: quantizing model to /dev/shm/model_cache
chaser-cz-qlora-out-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaser-cz-qlora-out-v2-mkmlizer: Traceback (most recent call last):
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
chaser-cz-qlora-out-v2-mkmlizer:     cli()
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
chaser-cz-qlora-out-v2-mkmlizer:     return self.main(*args, **kwargs)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
chaser-cz-qlora-out-v2-mkmlizer:     rv = self.invoke(ctx)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return __callback(*args, **kwargs)
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
chaser-cz-qlora-out-v2-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
chaser-cz-qlora-out-v2-mkmlizer:     flywheel.instrument(
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 65, in instrument
chaser-cz-qlora-out-v2-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
chaser-cz-qlora-out-v2-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 38, in save_st_pretrained
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor in model_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 213, in tensor_compiler
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor in tensor_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 196, in tensor_slicer
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor_slice in tensor_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 54, in hfpt_model_iterator
chaser-cz-qlora-out-v2-mkmlizer:     tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
chaser-cz-qlora-out-v2-mkmlizer:     with _open_file_like(f, 'rb') as opened_file:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
chaser-cz-qlora-out-v2-mkmlizer:     return _open_file(name_or_buffer, mode)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
chaser-cz-qlora-out-v2-mkmlizer:     super().__init__(open(name, mode))
chaser-cz-qlora-out-v2-mkmlizer: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp8fh81c8s/pytorch_model-00001-of-00003.bin'
Job chaser-cz-qlora-out-v2-mkmlizer completed after 22.42s with status: failed
Stopping job with name chaser-cz-qlora-out-v2-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaser-cz-qlora-out-v2-mkmlizer
Waiting for job on chaser-cz-qlora-out-v2-mkmlizer to finish
chaser-cz-qlora-out-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaser-cz-qlora-out-v2-mkmlizer: ║     _____            __           __                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaser-cz-qlora-out-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaser-cz-qlora-out-v2-mkmlizer: ║       /___/                                                         ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Version: 0.8.14                                                    ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaser-cz-qlora-out-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaser-cz-qlora-out-v2-mkmlizer: ║  belonging to:                                                      ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaser-cz-qlora-out-v2-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
chaser-cz-qlora-out-v2-mkmlizer: ║                                                                     ║
chaser-cz-qlora-out-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaser-cz-qlora-out-v2-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
chaser-cz-qlora-out-v2-mkmlizer:   warnings.warn(warning_message, FutureWarning)
chaser-cz-qlora-out-v2-mkmlizer: Downloaded to shared memory in 3.158s
chaser-cz-qlora-out-v2-mkmlizer: quantizing model to /dev/shm/model_cache
chaser-cz-qlora-out-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaser-cz-qlora-out-v2-mkmlizer: Traceback (most recent call last):
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
chaser-cz-qlora-out-v2-mkmlizer:     cli()
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
chaser-cz-qlora-out-v2-mkmlizer:     return self.main(*args, **kwargs)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
chaser-cz-qlora-out-v2-mkmlizer:     rv = self.invoke(ctx)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
chaser-cz-qlora-out-v2-mkmlizer:     return __callback(*args, **kwargs)
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
chaser-cz-qlora-out-v2-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
chaser-cz-qlora-out-v2-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
chaser-cz-qlora-out-v2-mkmlizer:     flywheel.instrument(
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 65, in instrument
chaser-cz-qlora-out-v2-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
chaser-cz-qlora-out-v2-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 38, in save_st_pretrained
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor in model_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 213, in tensor_compiler
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor in tensor_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 196, in tensor_slicer
chaser-cz-qlora-out-v2-mkmlizer:     for name, tensor_slice in tensor_iterator:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 54, in hfpt_model_iterator
chaser-cz-qlora-out-v2-mkmlizer:     tensors = torch.load(model_shard_filename, map_location=torch.device(self.device), mmap=True)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
chaser-cz-qlora-out-v2-mkmlizer:     with _open_file_like(f, 'rb') as opened_file:
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
chaser-cz-qlora-out-v2-mkmlizer:     return _open_file(name_or_buffer, mode)
chaser-cz-qlora-out-v2-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
chaser-cz-qlora-out-v2-mkmlizer:     super().__init__(open(name, mode))
chaser-cz-qlora-out-v2-mkmlizer: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpk7uws9sl/pytorch_model-00001-of-00003.bin'
Job chaser-cz-qlora-out-v2-mkmlizer completed after 12.71s with status: failed
Stopping job with name chaser-cz-qlora-out-v2-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaser-cz-qlora-out-v2-mkmlizer
Waiting for job on chaser-cz-qlora-out-v2-mkmlizer to finish
Job chaser-cz-qlora-out-v2-mkmlizer completed after 10.55s with status: failed
Stopping job with name chaser-cz-qlora-out-v2-mkmlizer
MKMLizerError('')
chaser-cz-qlora-out_v2 status is now failed due to DeploymentManager action
admin requested tearing down of chaser-cz-qlora-out_v2
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.22s
Running pipeline stage MKMLModelDeleter
Skipping deletion as no model was successfully uploaded
Pipeline stage MKMLModelDeleter completed in 0.16s
chaser-cz-qlora-out_v2 status is now torndown due to DeploymentManager action