submission_id: sanchuanhehe-end-try_v15
developer_uid: sanchuanhehe
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 1, 'max_output_tokens': 64}
model_name: sanchuanhehe-end-try_v14
model_repo: sanchuanhehe/end_try
status: torndown
timestamp: 2024-09-02T14:01:21+00:00
Resubmit model
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name sanchuanhehe-end-try-v15-mkmlizer
Waiting for job on sanchuanhehe-end-try-v15-mkmlizer to finish
Stopping job with name sanchuanhehe-end-try-v15-mkmlizer
%s, retrying in %s seconds...
Starting job with name sanchuanhehe-end-try-v15-mkmlizer
Waiting for job on sanchuanhehe-end-try-v15-mkmlizer to finish
admin requested tearing down of zonemercy-lexical-nemov6_2011_v2
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
Checking if service zonemercy-lexical-nemov6-2011-v2 is running
Tearing down inference service zonemercy-lexical-nemov6-2011-v2
Service zonemercy-lexical-nemov6-2011-v2 has been torndown
Pipeline stage MKMLDeleter completed in 3.25s
run pipeline stage %s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key zonemercy-lexical-nemov6-2011-v2/config.json from bucket guanaco-mkml-models
Deleting key zonemercy-lexical-nemov6-2011-v2/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key zonemercy-lexical-nemov6-2011-v2/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key zonemercy-lexical-nemov6-2011-v2/tokenizer.json from bucket guanaco-mkml-models
Deleting key zonemercy-lexical-nemov6-2011-v2/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/config.json from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/merges.txt from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/reward.tensors from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key zonemercy-lexical-nemov6-2011-v2_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.00s
zonemercy-lexical-nemov6_2011_v2 status is now torndown due to DeploymentManager action
sanchuanhehe-end-try-v15-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sanchuanhehe-end-try-v15-mkmlizer: ║     _____            __           __                                ║
sanchuanhehe-end-try-v15-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
sanchuanhehe-end-try-v15-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
sanchuanhehe-end-try-v15-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
sanchuanhehe-end-try-v15-mkmlizer: ║       /___/                                                         ║
sanchuanhehe-end-try-v15-mkmlizer: ║                                                                     ║
sanchuanhehe-end-try-v15-mkmlizer: ║  Version: 0.10.1                                                    ║
sanchuanhehe-end-try-v15-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
sanchuanhehe-end-try-v15-mkmlizer: ║  https://mk1.ai                                                     ║
sanchuanhehe-end-try-v15-mkmlizer: ║                                                                     ║
sanchuanhehe-end-try-v15-mkmlizer: ║  The license key for the current software has been verified as      ║
sanchuanhehe-end-try-v15-mkmlizer: ║  belonging to:                                                      ║
sanchuanhehe-end-try-v15-mkmlizer: ║                                                                     ║
sanchuanhehe-end-try-v15-mkmlizer: ║  Chai Research Corp.                                                ║
sanchuanhehe-end-try-v15-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
sanchuanhehe-end-try-v15-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
sanchuanhehe-end-try-v15-mkmlizer: ║                                                                     ║
sanchuanhehe-end-try-v15-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sanchuanhehe-end-try-v15-mkmlizer: unable to retrieve container logs for containerd://fc564180407d98e4fb2f3
Job sanchuanhehe-end-try-v15-mkmlizer completed after 22.04s with status: failed
Stopping job with name sanchuanhehe-end-try-v15-mkmlizer
%s, retrying in %s seconds...
Starting job with name sanchuanhehe-end-try-v15-mkmlizer
Waiting for job on sanchuanhehe-end-try-v15-mkmlizer to finish
sanchuanhehe-end-try-v15-mkmlizer: Downloaded to shared memory in 9.498s
sanchuanhehe-end-try-v15-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpjx9m2a6k, device:0
sanchuanhehe-end-try-v15-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sanchuanhehe-end-try-v15-mkmlizer: Traceback (most recent call last):
sanchuanhehe-end-try-v15-mkmlizer:   File "/code/uploading/mkmlize.py", line 151, in <module>
sanchuanhehe-end-try-v15-mkmlizer:     cli()
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
sanchuanhehe-end-try-v15-mkmlizer:     return self.main(*args, **kwargs)
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
sanchuanhehe-end-try-v15-mkmlizer:     rv = self.invoke(ctx)
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
sanchuanhehe-end-try-v15-mkmlizer:     return _process_result(sub_ctx.command.invoke(sub_ctx))
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
sanchuanhehe-end-try-v15-mkmlizer:     return ctx.invoke(self.callback, **ctx.params)
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
sanchuanhehe-end-try-v15-mkmlizer:     return __callback(*args, **kwargs)
sanchuanhehe-end-try-v15-mkmlizer:   File "/code/uploading/mkmlize.py", line 42, in quantize
sanchuanhehe-end-try-v15-mkmlizer:     quantize_model(temp_folder, output_path, profile, device)
sanchuanhehe-end-try-v15-mkmlizer:   File "/code/uploading/mkmlize.py", line 135, in quantize_model
sanchuanhehe-end-try-v15-mkmlizer:     flywheel.instrument(
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 90, in instrument
sanchuanhehe-end-try-v15-mkmlizer:     compiler.save_pretrained(input_model_path, output_model_path, storage_format)
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
sanchuanhehe-end-try-v15-mkmlizer:     self.save_st_pretrained(input_model_path, output_model_path)
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 31, in save_st_pretrained
sanchuanhehe-end-try-v15-mkmlizer:     model_iterator = self.factory.model_iterator()
sanchuanhehe-end-try-v15-mkmlizer:   File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 255, in model_iterator
sanchuanhehe-end-try-v15-mkmlizer:     raise RuntimeError(f"Unable to construct an interator for the model")
sanchuanhehe-end-try-v15-mkmlizer: RuntimeError: Unable to construct an interator for the model
Job sanchuanhehe-end-try-v15-mkmlizer completed after 23.15s with status: failed
Stopping job with name sanchuanhehe-end-try-v15-mkmlizer
sanchuanhehe-end-try_v15 status is now failed due to DeploymentManager action
sanchuanhehe-end-try_v15 status is now torndown due to DeploymentManager action