Running pipeline stage MKMLizer
Starting job with name heralax-llama-3-llamilitary-v1-mkmlizer
Waiting for job on heralax-llama-3-llamilitary-v1-mkmlizer to finish
heralax-llama-3-llamilitary-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
heralax-llama-3-llamilitary-v1-mkmlizer: ║ _____ __ __ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /___/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Version: 0.8.14 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ https://mk1.ai ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ The license key for the current software has been verified as ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ belonging to: ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Chai Research Corp. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission blend_pefis_2024-07-04: ('http://mistralai-mixtral-8x7b-3473-v33-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"TypeError : SamplingParameters.__init__() got an unexpected keyword argument \'min_p\'"}')
heralax-llama-3-llamilitary-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
heralax-llama-3-llamilitary-v1-mkmlizer: ║ _____ __ __ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /___/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Version: 0.8.14 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ https://mk1.ai ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ The license key for the current software has been verified as ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ belonging to: ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Chai Research Corp. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Job heralax-llama-3-llamilitary-v1-mkmlizer completed after 54.1s with status: failed
Stopping job with name heralax-llama-3-llamilitary-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name heralax-llama-3-llamilitary-v1-mkmlizer
Waiting for job on heralax-llama-3-llamilitary-v1-mkmlizer to finish
heralax-llama-3-llamilitary-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
heralax-llama-3-llamilitary-v1-mkmlizer: ║ _____ __ __ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /___/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Version: 0.8.14 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ https://mk1.ai ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ The license key for the current software has been verified as ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ belonging to: ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Chai Research Corp. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
heralax-llama-3-llamilitary-v1-mkmlizer: Downloaded to shared memory in 16.578s
heralax-llama-3-llamilitary-v1-mkmlizer: quantizing model to /dev/shm/model_cache
heralax-llama-3-llamilitary-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
heralax-llama-3-llamilitary-v1-mkmlizer: Traceback (most recent call last):
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 151, in <module>
heralax-llama-3-llamilitary-v1-mkmlizer: cli()
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
heralax-llama-3-llamilitary-v1-mkmlizer: return self.main(*args, **kwargs)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
heralax-llama-3-llamilitary-v1-mkmlizer: rv = self.invoke(ctx)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return _process_result(sub_ctx.command.invoke(sub_ctx))
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return ctx.invoke(self.callback, **ctx.params)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return __callback(*args, **kwargs)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 42, in quantize
heralax-llama-3-llamilitary-v1-mkmlizer: quantize_model(temp_folder, output_path, profile, device)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 135, in quantize_model
heralax-llama-3-llamilitary-v1-mkmlizer: flywheel.instrument(
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 65, in instrument
heralax-llama-3-llamilitary-v1-mkmlizer: compiler.save_pretrained(input_model_path, output_model_path, storage_format)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
heralax-llama-3-llamilitary-v1-mkmlizer: self.save_st_pretrained(input_model_path, output_model_path)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 31, in save_st_pretrained
heralax-llama-3-llamilitary-v1-mkmlizer: model_iterator = self.factory.model_iterator()
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 247, in model_iterator
heralax-llama-3-llamilitary-v1-mkmlizer: raise RuntimeError(f"Unable to construct an interator for the model")
heralax-llama-3-llamilitary-v1-mkmlizer: RuntimeError: Unable to construct an interator for the model
heralax-llama-3-llamilitary-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
heralax-llama-3-llamilitary-v1-mkmlizer: ║ _____ __ __ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ /___/ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Version: 0.8.14 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ https://mk1.ai ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ The license key for the current software has been verified as ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ belonging to: ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Chai Research Corp. ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
heralax-llama-3-llamilitary-v1-mkmlizer: ║ ║
heralax-llama-3-llamilitary-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Job heralax-llama-3-llamilitary-v1-mkmlizer completed after 66.27s with status: failed
Stopping job with name heralax-llama-3-llamilitary-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name heralax-llama-3-llamilitary-v1-mkmlizer
Waiting for job on heralax-llama-3-llamilitary-v1-mkmlizer to finish
heralax-llama-3-llamilitary-v1-mkmlizer: Downloaded to shared memory in 15.921s
heralax-llama-3-llamilitary-v1-mkmlizer: quantizing model to /dev/shm/model_cache
heralax-llama-3-llamilitary-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
heralax-llama-3-llamilitary-v1-mkmlizer: Traceback (most recent call last):
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 151, in <module>
heralax-llama-3-llamilitary-v1-mkmlizer: cli()
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
heralax-llama-3-llamilitary-v1-mkmlizer: return self.main(*args, **kwargs)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1053, in main
heralax-llama-3-llamilitary-v1-mkmlizer: rv = self.invoke(ctx)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return _process_result(sub_ctx.command.invoke(sub_ctx))
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return ctx.invoke(self.callback, **ctx.params)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 754, in invoke
heralax-llama-3-llamilitary-v1-mkmlizer: return __callback(*args, **kwargs)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 42, in quantize
heralax-llama-3-llamilitary-v1-mkmlizer: quantize_model(temp_folder, output_path, profile, device)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 135, in quantize_model
heralax-llama-3-llamilitary-v1-mkmlizer: flywheel.instrument(
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/instrument.py", line 65, in instrument
heralax-llama-3-llamilitary-v1-mkmlizer: compiler.save_pretrained(input_model_path, output_model_path, storage_format)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 23, in save_pretrained
heralax-llama-3-llamilitary-v1-mkmlizer: self.save_st_pretrained(input_model_path, output_model_path)
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/compiler.py", line 31, in save_st_pretrained
heralax-llama-3-llamilitary-v1-mkmlizer: model_iterator = self.factory.model_iterator()
heralax-llama-3-llamilitary-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/mk1/flywheel/functional/loader.py", line 247, in model_iterator
heralax-llama-3-llamilitary-v1-mkmlizer: raise RuntimeError(f"Unable to construct an interator for the model")
heralax-llama-3-llamilitary-v1-mkmlizer: RuntimeError: Unable to construct an interator for the model
heralax-llama-3-llamilitary-v1-mkmlizer: unable to retrieve container logs for containerd://9fc3f514aebf138ca63ff
Job heralax-llama-3-llamilitary-v1-mkmlizer completed after 42.62s with status: failed
Stopping job with name heralax-llama-3-llamilitary-v1-mkmlizer
MKMLizerError('')
heralax-llama-3-llamilitary_v1 status is now failed due to DeploymentManager action
Running pipeline stage ISVCDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage ISVCDeleter completed in 0.11s
Running pipeline stage MKMLModelDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLModelDeleter completed in 0.09s
heralax-llama-3-llamilitary_v1 status is now torndown due to DeploymentManager action