bbchicago-nana-nemo-12b-v1-0

submission_id: bbchicago-nana-nemo-12b-v1-0_v3
developer_uid: Bbbrun0
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', 'Bot:', 'User:', 'You:', 'Me:', '####'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
model_name: bbchicago-nana-nemo-12b-v1-0_v1
model_repo: BBChicago/Nana-nemo-12B_v1.0
status: torndown
timestamp: 2024-09-25T02:34:04+00:00
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
Waiting for job on bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║     _____            __           __                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║       /___/                                                         ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Version: 0.11.12                                                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  belonging to:                                                      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Downloaded to shared memory in 23.059s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpm65fvsxq, device:0
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Traceback (most recent call last):
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     cli()
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return self.main(*args, **kwargs)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     rv = self.invoke(ctx)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return __callback(*args, **kwargs)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     flywheel.instrument(
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 93, in instrument
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 31, in save_st_pretrained
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     model_iterator = self.factory.model_iterator()
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 255, in model_iterator
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     raise RuntimeError(f"Unable to construct an interator for the model")
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: RuntimeError: Unable to construct an interator for the model
Job bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer completed after 42.45s with status: failed
Stopping job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
Waiting for job on bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer to finish
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║     _____            __           __                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║       /___/                                                         ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Version: 0.11.12                                                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  belonging to:                                                      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Downloaded to shared memory in 22.673s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpzhln2hpm, device:0
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Job bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer completed after 43.75s with status: failed
Stopping job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
Waiting for job on bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer to finish
Failed to get response for submission chaiml-elo-alignment-run-3_v36: ('http://chaiml-elo-alignment-run-3-v36-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║     _____            __           __                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║       /___/                                                         ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Version: 0.11.12                                                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  https://mk1.ai                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  belonging to:                                                      ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Chai Research Corp.                                                ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ║                                                                     ║
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Downloaded to shared memory in 20.861s
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmps_d4ljlx, device:0
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: Traceback (most recent call last):
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     cli()
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return self.main(*args, **kwargs)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     rv = self.invoke(ctx)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     return __callback(*args, **kwargs)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     flywheel.instrument(
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 93, in instrument
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 31, in save_st_pretrained
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     model_iterator = self.factory.model_iterator()
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 255, in model_iterator
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer:     raise RuntimeError(f"Unable to construct an interator for the model")
bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer: RuntimeError: Unable to construct an interator for the model
Job bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer completed after 55.05s with status: failed
Stopping job with name bbchicago-nana-nemo-12b-v1-0-v3-mkmlizer
clean up pipeline due to error=%s
Shutdown handler de-registered
bbchicago-nana-nemo-12b-v1-0_v3 status is now failed due to DeploymentManager action
bbchicago-nana-nemo-12b-v1-0_v3 status is now torndown due to DeploymentManager action