Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-mistral-24b-sca-46352-v1-mkmlizer
2 validation errors for MultiheadRewardSubmission
platform
unexpected value; permitted: 'multihead_reward' (type=value_error.const; given=reward; permitted=('multihead_reward',))
weights
field required (type=value_error.missing)
2 validation errors for MultiheadRewardSubmission
platform
unexpected value; permitted: 'multihead_reward' (type=value_error.const; given=reward; permitted=('multihead_reward',))
weights
field required (type=value_error.missing)
2 validation errors for MultiheadRewardSubmission
platform
unexpected value; permitted: 'multihead_reward' (type=value_error.const; given=reward; permitted=('multihead_reward',))
weights
field required (type=value_error.missing)
Waiting for job on rirv938-mistral-24b-sca-46352-v1-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
2 validation errors for MultiheadRewardSubmission
platform
unexpected value; permitted: 'multihead_reward' (type=value_error.const; given=reward; permitted=('multihead_reward',))
weights
field required (type=value_error.missing)
Starting job with name rirv938-mistral-24b-scal-2131-v4-mkmlizer
2 validation errors for MultiheadRewardSubmission
platform
unexpected value; permitted: 'multihead_reward' (type=value_error.const; given=reward; permitted=('multihead_reward',))
weights
field required (type=value_error.missing)
Waiting for job on rirv938-mistral-24b-scal-2131-v4-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-scaleup-400k-v3-mkmlizer
Waiting for job on rirv938-llama-8b-scaleup-400k-v3-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-llama-8b-scaleup-200k-v1-mkmlizer
Waiting for job on rirv938-llama-8b-scaleup-200k-v1-mkmlizer to finish
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Version: 0.25.10 ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ https://mk1.ai ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Version: 0.25.10 ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-mistral-24b-sca-46352-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Version: 0.25.10 ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ https://mk1.ai ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ belonging to: ║
rirv938-mistral-24b-scal-2131-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Version: 0.25.10 ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ║ ║
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: Downloaded to shared memory in 26.872s
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpoobp3nn1, device:0
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: Downloaded to shared memory in 33.113s
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpcjulah14, device:0
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: quantized model in 20.063s
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: Processed model rirv938/llama_8b_scaleup_400k in 46.936s
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-400k-v3
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-400k-v3/config.json
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-400k-v3/special_tokens_map.json
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-400k-v3/tokenizer_config.json
rirv938-llama-8b-scaleup-400k-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-400k-v3/tokenizer.json
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: quantized model in 19.942s
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: Processed model rirv938/llama_8b_scaleup_200k in 53.056s
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1/config.json
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1/special_tokens_map.json
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1/tokenizer_config.json
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1/tokenizer.json
rirv938-llama-8b-scaleup-200k-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-llama-8b-scaleup-200k-v1/flywheel_model.0.safetensors
rirv938-mistral-24b-sca-46352-v1-mkmlizer: Downloaded to shared memory in 83.036s
rirv938-mistral-24b-sca-46352-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpcgwc94xl, device:0
rirv938-mistral-24b-sca-46352-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-mistral-24b-scal-2131-v4-mkmlizer: Downloaded to shared memory in 86.872s
rirv938-mistral-24b-scal-2131-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp5c6g4zd_, device:0
rirv938-mistral-24b-scal-2131-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Job rirv938-llama-8b-scaleup-200k-v1-mkmlizer completed after 102.78s with status: succeeded
Stopping job with name rirv938-llama-8b-scaleup-200k-v1-mkmlizer
Pipeline stage MKMLizer completed in 105.31s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.83s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-scaleup-200k-v1
Waiting for inference service rirv938-llama-8b-scaleup-200k-v1 to be ready
Job rirv938-llama-8b-scaleup-400k-v3-mkmlizer completed after 112.55s with status: succeeded
Stopping job with name rirv938-llama-8b-scaleup-400k-v3-mkmlizer
Pipeline stage MKMLizer completed in 115.39s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 1.04s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-llama-8b-scaleup-400k-v3
Waiting for inference service rirv938-llama-8b-scaleup-400k-v3 to be ready
rirv938-mistral-24b-sca-46352-v1-mkmlizer: quantized model in 50.512s
rirv938-mistral-24b-sca-46352-v1-mkmlizer: Processed model rirv938/mistral_24b_scaleup_200k in 133.554s
rirv938-mistral-24b-sca-46352-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-mistral-24b-sca-46352-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-mistral-24b-sca-46352-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/config.json
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/special_tokens_map.json
rirv938-mistral-24b-scal-2131-v4-mkmlizer: quantized model in 50.314s
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/tokenizer_config.json
rirv938-mistral-24b-scal-2131-v4-mkmlizer: Processed model rirv938/mistral_24b_scaleup_400k in 137.187s
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/flywheel_model.2.safetensors
rirv938-mistral-24b-scal-2131-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v4/tokenizer_config.json
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/flywheel_model.0.safetensors
rirv938-mistral-24b-scal-2131-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v4/tokenizer.json
rirv938-mistral-24b-sca-46352-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-sca-46352-v1/flywheel_model.1.safetensors
rirv938-mistral-24b-scal-2131-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v4/flywheel_model.2.safetensors
rirv938-mistral-24b-sca-46352-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:09, 37.63it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:12, 29.15it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:11, 30.53it/s]
Loading 0: 4%|▍ | 16/363 [00:00<00:12, 28.27it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:10, 32.20it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:11, 29.37it/s]
Loading 0: 8%|▊ | 30/363 [00:00<00:09, 34.21it/s]
Loading 0: 9%|▉ | 34/363 [00:01<00:14, 22.18it/s]
Loading 0: 10%|█ | 37/363 [00:01<00:15, 20.40it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:16, 19.64it/s]
Loading 0: 13%|█▎ | 46/363 [00:01<00:12, 24.95it/s]
Loading 0: 14%|█▍ | 50/363 [00:02<00:14, 22.08it/s]
Loading 0: 16%|█▌ | 57/363 [00:02<00:11, 27.51it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:11, 26.02it/s]
Loading 0: 18%|█▊ | 65/363 [00:02<00:11, 25.68it/s]
Loading 0: 19%|█▉ | 70/363 [00:02<00:13, 22.39it/s]
Loading 0: 20%|██ | 73/363 [00:03<00:15, 18.98it/s]
Loading 0: 21%|██▏ | 78/363 [00:03<00:11, 24.05it/s]
Loading 0: 22%|██▏ | 81/363 [00:03<00:13, 21.12it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:11, 23.76it/s]
Loading 0: 25%|██▍ | 89/363 [00:03<00:11, 23.59it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:13, 19.94it/s]
Loading 0: 27%|██▋ | 99/363 [00:04<00:10, 26.32it/s]
Loading 0: 28%|██▊ | 102/363 [00:04<00:11, 23.64it/s]
Loading 0: 29%|██▉ | 107/363 [00:04<00:11, 22.36it/s]
Loading 0: 31%|███ | 111/363 [00:04<00:09, 25.29it/s]
Loading 0: 31%|███▏ | 114/363 [00:04<00:10, 23.30it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:08, 29.05it/s]
Loading 0: 34%|███▍ | 124/363 [00:05<00:08, 27.74it/s]
Loading 0: 36%|███▌ | 129/363 [00:05<00:07, 29.77it/s]
Loading 0: 37%|███▋ | 133/363 [00:05<00:08, 28.74it/s]
Loading 0: 38%|███▊ | 138/363 [00:05<00:07, 30.76it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:07, 28.95it/s]
Loading 0: 39%|███▉ | 142/363 [00:20<00:07, 28.95it/s]
Loading 0: 39%|███▉ | 143/363 [00:20<04:52, 1.33s/it]
Loading 0: 40%|███▉ | 145/363 [00:20<03:53, 1.07s/it]
Loading 0: 41%|████ | 149/363 [00:20<02:28, 1.44it/s]
Loading 0: 42%|████▏ | 152/363 [00:20<01:50, 1.90it/s]
Loading 0: 43%|████▎ | 155/363 [00:20<01:20, 2.59it/s]
Loading 0: 44%|████▎ | 158/363 [00:21<01:00, 3.39it/s]
Loading 0: 45%|████▌ | 165/363 [00:21<00:31, 6.22it/s]
Loading 0: 46%|████▋ | 168/363 [00:21<00:26, 7.30it/s]
Loading 0: 48%|████▊ | 174/363 [00:21<00:17, 10.81it/s]
Loading 0: 49%|████▉ | 177/363 [00:21<00:15, 11.91it/s]
Loading 0: 50%|█████ | 182/363 [00:21<00:11, 15.66it/s]
Loading 0: 52%|█████▏ | 187/363 [00:22<00:10, 17.04it/s]
Loading 0: 52%|█████▏ | 190/363 [00:22<00:10, 17.25it/s]
Loading 0: 53%|█████▎ | 194/363 [00:22<00:09, 17.44it/s]
Loading 0: 55%|█████▌ | 201/363 [00:22<00:06, 23.88it/s]
Loading 0: 56%|█████▌ | 204/363 [00:22<00:07, 22.13it/s]
Loading 0: 58%|█████▊ | 210/363 [00:22<00:05, 27.07it/s]
Loading 0: 59%|█████▉ | 214/363 [00:23<00:05, 26.60it/s]
Loading 0: 60%|██████ | 218/363 [00:23<00:05, 27.30it/s]
Loading 0: 61%|██████ | 221/363 [00:23<00:05, 25.90it/s]
Loading 0: 62%|██████▏ | 224/363 [00:23<00:06, 20.36it/s]
Loading 0: 63%|██████▎ | 228/363 [00:23<00:05, 23.96it/s]
Loading 0: 64%|██████▎ | 231/363 [00:23<00:06, 21.36it/s]
Loading 0: 65%|██████▌ | 237/363 [00:24<00:04, 27.08it/s]
Loading 0: 66%|██████▌ | 240/363 [00:24<00:05, 24.42it/s]
Loading 0: 68%|██████▊ | 246/363 [00:24<00:04, 28.79it/s]
Loading 0: 69%|██████▉ | 250/363 [00:24<00:04, 27.43it/s]
Loading 0: 70%|███████ | 255/363 [00:24<00:03, 29.70it/s]
Loading 0: 71%|███████▏ | 259/363 [00:24<00:03, 27.08it/s]
Loading 0: 73%|███████▎ | 264/363 [00:25<00:03, 30.70it/s]
Loading 0: 74%|███████▍ | 268/363 [00:25<00:04, 22.51it/s]
Loading 0: 75%|███████▍ | 271/363 [00:25<00:04, 20.81it/s]
Loading 0: 75%|███████▌ | 274/363 [00:25<00:03, 22.37it/s]
Loading 0: 76%|███████▋ | 277/363 [00:25<00:03, 22.48it/s]
Loading 0: 78%|███████▊ | 282/363 [00:25<00:03, 25.33it/s]
Loading 0: 79%|███████▊ | 285/363 [00:26<00:03, 22.87it/s]
Loading 0: 80%|████████ | 291/363 [00:26<00:02, 28.08it/s]
Loading 0: 81%|████████ | 294/363 [00:26<00:02, 24.69it/s]
Loading 0: 82%|████████▏ | 299/363 [00:26<00:02, 27.39it/s]
Loading 0: 84%|████████▎ | 304/363 [00:26<00:02, 25.38it/s]
Loading 0: 85%|████████▍ | 307/363 [00:26<00:02, 23.03it/s]
Loading 0: 85%|████████▍ | 308/363 [00:41<00:02, 23.03it/s]
Loading 0: 85%|████████▌ | 309/363 [00:41<01:13, 1.35s/it]
Loading 0: 86%|████████▌ | 311/363 [00:42<00:57, 1.11s/it]
Loading 0: 87%|████████▋ | 316/363 [00:42<00:30, 1.52it/s]
Loading 0: 88%|████████▊ | 319/363 [00:42<00:22, 1.99it/s]
Loading 0: 89%|████████▊ | 322/363 [00:42<00:15, 2.65it/s]
Loading 0: 90%|████████▉ | 325/363 [00:42<00:10, 3.55it/s]
Loading 0: 90%|█████████ | 328/363 [00:42<00:07, 4.69it/s]
Loading 0: 91%|█████████ | 331/363 [00:43<00:05, 6.07it/s]
Loading 0: 92%|█████████▏| 335/363 [00:43<00:03, 8.27it/s]
Loading 0: 93%|█████████▎| 338/363 [00:43<00:02, 9.87it/s]
Loading 0: 94%|█████████▍| 341/363 [00:43<00:02, 10.24it/s]
Loading 0: 95%|█████████▌| 345/363 [00:43<00:01, 13.52it/s]
Loading 0: 96%|█████████▌| 348/363 [00:43<00:01, 13.63it/s]
Loading 0: 98%|█████████▊| 354/363 [00:44<00:00, 18.56it/s]
Loading 0: 98%|█████████▊| 357/363 [00:44<00:00, 17.88it/s]
Loading 0: 99%|█████████▉| 361/363 [00:44<00:00, 21.60it/s]
rirv938-mistral-24b-scal-2131-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v4/flywheel_model.1.safetensors
Job rirv938-mistral-24b-sca-46352-v1-mkmlizer completed after 175.04s with status: succeeded
rirv938-mistral-24b-scal-2131-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v4/flywheel_model.0.safetensors
Stopping job with name rirv938-mistral-24b-sca-46352-v1-mkmlizer
rirv938-mistral-24b-scal-2131-v4-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:10, 34.90it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:10, 32.33it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:11, 31.31it/s]
Loading 0: 4%|▍ | 16/363 [00:00<00:12, 27.34it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:11, 30.80it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:11, 28.20it/s]
Loading 0: 8%|▊ | 30/363 [00:00<00:09, 33.39it/s]
Loading 0: 9%|▉ | 34/363 [00:01<00:13, 24.01it/s]
Loading 0: 10%|█ | 37/363 [00:01<00:14, 21.81it/s]
Loading 0: 11%|█ | 40/363 [00:01<00:13, 23.36it/s]
Loading 0: 12%|█▏ | 43/363 [00:01<00:13, 23.43it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:11, 27.08it/s]
Loading 0: 14%|█▍ | 51/363 [00:01<00:13, 23.81it/s]
Loading 0: 16%|█▌ | 57/363 [00:02<00:10, 29.45it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:10, 27.60it/s]
Loading 0: 18%|█▊ | 65/363 [00:02<00:10, 27.75it/s]
Loading 0: 19%|█▉ | 70/363 [00:02<00:11, 25.18it/s]
Loading 0: 20%|██ | 73/363 [00:02<00:13, 21.30it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:10, 26.44it/s]
Loading 0: 23%|██▎ | 82/363 [00:03<00:10, 26.02it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:10, 27.42it/s]
Loading 0: 25%|██▍ | 89/363 [00:03<00:09, 27.44it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:12, 22.47it/s]
Loading 0: 27%|██▋ | 99/363 [00:03<00:08, 29.64it/s]
Loading 0: 28%|██▊ | 103/363 [00:03<00:09, 28.29it/s]
Loading 0: 29%|██▉ | 107/363 [00:04<00:10, 24.20it/s]
Loading 0: 31%|███ | 111/363 [00:04<00:09, 27.08it/s]
Loading 0: 31%|███▏ | 114/363 [00:04<00:10, 24.18it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:08, 29.36it/s]
Loading 0: 34%|███▍ | 124/363 [00:04<00:08, 27.23it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 29.57it/s]
Loading 0: 37%|███▋ | 133/363 [00:04<00:08, 27.86it/s]
Loading 0: 38%|███▊ | 138/363 [00:05<00:07, 30.11it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:07, 28.40it/s]
Loading 0: 40%|███▉ | 144/363 [00:20<00:07, 28.40it/s]
Loading 0: 40%|███▉ | 145/363 [00:20<04:11, 1.15s/it]
Loading 0: 41%|████ | 149/363 [00:20<02:54, 1.22it/s]
Loading 0: 42%|████▏ | 152/363 [00:20<02:14, 1.56it/s]
Loading 0: 43%|████▎ | 155/363 [00:20<01:40, 2.06it/s]
Loading 0: 44%|████▎ | 158/363 [00:20<01:16, 2.68it/s]
Loading 0: 45%|████▌ | 165/363 [00:21<00:40, 4.85it/s]
Loading 0: 46%|████▋ | 168/363 [00:21<00:33, 5.82it/s]
Loading 0: 48%|████▊ | 174/363 [00:21<00:21, 8.78it/s]
Loading 0: 49%|████▉ | 178/363 [00:21<00:17, 10.63it/s]
Loading 0: 50%|█████ | 182/363 [00:21<00:13, 13.01it/s]
Loading 0: 52%|█████▏ | 187/363 [00:21<00:11, 14.79it/s]
Loading 0: 52%|█████▏ | 190/363 [00:22<00:11, 15.35it/s]
Loading 0: 53%|█████▎ | 194/363 [00:22<00:10, 15.97it/s]
Loading 0: 55%|█████▌ | 201/363 [00:22<00:07, 22.33it/s]
Loading 0: 56%|█████▌ | 204/363 [00:22<00:07, 21.81it/s]
Loading 0: 58%|█████▊ | 210/363 [00:22<00:05, 27.19it/s]
Loading 0: 59%|█████▉ | 214/363 [00:22<00:05, 25.79it/s]
Loading 0: 60%|██████ | 218/363 [00:23<00:05, 25.92it/s]
Loading 0: 61%|██████ | 221/363 [00:23<00:05, 24.93it/s]
Loading 0: 62%|██████▏ | 224/363 [00:23<00:07, 19.43it/s]
Loading 0: 63%|██████▎ | 228/363 [00:23<00:05, 22.74it/s]
Loading 0: 64%|██████▎ | 231/363 [00:23<00:06, 21.00it/s]
Loading 0: 65%|██████▌ | 237/363 [00:23<00:04, 26.63it/s]
Loading 0: 66%|██████▌ | 240/363 [00:24<00:05, 23.77it/s]
Loading 0: 68%|██████▊ | 246/363 [00:24<00:04, 28.58it/s]
Loading 0: 69%|██████▉ | 250/363 [00:24<00:04, 26.84it/s]
Loading 0: 70%|███████ | 255/363 [00:24<00:03, 29.05it/s]
Loading 0: 71%|███████▏ | 259/363 [00:24<00:03, 27.33it/s]
Loading 0: 73%|███████▎ | 264/363 [00:24<00:03, 31.83it/s]
Loading 0: 74%|███████▍ | 268/363 [00:25<00:04, 23.42it/s]
Loading 0: 75%|███████▍ | 271/363 [00:25<00:04, 21.66it/s]
Loading 0: 75%|███████▌ | 274/363 [00:25<00:03, 23.17it/s]
Loading 0: 76%|███████▋ | 277/363 [00:25<00:03, 22.75it/s]
Loading 0: 77%|███████▋ | 280/363 [00:25<00:03, 23.50it/s]
Loading 0: 78%|███████▊ | 283/363 [00:25<00:03, 23.85it/s]
Loading 0: 79%|███████▉ | 286/363 [00:25<00:03, 23.43it/s]
Loading 0: 80%|████████ | 291/363 [00:26<00:02, 27.05it/s]
Loading 0: 81%|████████ | 294/363 [00:26<00:02, 23.16it/s]
Loading 0: 82%|████████▏ | 299/363 [00:26<00:02, 25.29it/s]
Loading 0: 84%|████████▎ | 304/363 [00:26<00:02, 22.08it/s]
Loading 0: 85%|████████▍ | 307/363 [00:26<00:02, 19.92it/s]
Loading 0: 85%|████████▍ | 308/363 [00:42<00:02, 19.92it/s]
Loading 0: 85%|████████▌ | 309/363 [00:42<01:16, 1.41s/it]
Loading 0: 86%|████████▌ | 311/363 [00:42<00:59, 1.14s/it]
Loading 0: 87%|████████▋ | 316/363 [00:42<00:31, 1.49it/s]
Loading 0: 88%|████████▊ | 319/363 [00:42<00:22, 1.95it/s]
Loading 0: 89%|████████▊ | 322/363 [00:42<00:15, 2.61it/s]
Loading 0: 90%|█████████ | 327/363 [00:42<00:08, 4.14it/s]
Loading 0: 91%|█████████ | 330/363 [00:43<00:06, 5.13it/s]
Loading 0: 92%|█████████▏| 334/363 [00:43<00:04, 7.00it/s]
Loading 0: 93%|█████████▎| 337/363 [00:43<00:03, 8.20it/s]
Loading 0: 94%|█████████▍| 341/363 [00:43<00:02, 9.71it/s]
Loading 0: 95%|█████████▌| 345/363 [00:43<00:01, 12.81it/s]
Loading 0: 96%|█████████▌| 348/363 [00:43<00:01, 13.96it/s]
Loading 0: 97%|█████████▋| 352/363 [00:44<00:00, 16.72it/s]
Loading 0: 98%|█████████▊| 355/363 [00:44<00:00, 18.76it/s]
Loading 0: 99%|█████████▊| 358/363 [00:44<00:00, 20.52it/s]
Pipeline stage MKMLizer completed in 177.38s
Job rirv938-mistral-24b-scal-2131-v4-mkmlizer completed after 173.51s with status: succeeded
run pipeline stage %s
Stopping job with name rirv938-mistral-24b-scal-2131-v4-mkmlizer
Running pipeline stage MKMLTemplater
Pipeline stage MKMLizer completed in 175.52s
Pipeline stage MKMLTemplater completed in 0.86s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Running pipeline stage MKMLDeployer
Pipeline stage MKMLTemplater completed in 0.66s
Creating inference service rirv938-mistral-24b-sca-46352-v1
run pipeline stage %s
Waiting for inference service rirv938-mistral-24b-sca-46352-v1 to be ready
Running pipeline stage MKMLDeployer
Creating inference service rirv938-mistral-24b-scal-2131-v4
Waiting for inference service rirv938-mistral-24b-scal-2131-v4 to be ready
Inference service rirv938-llama-8b-scaleup-200k-v1 ready after 91.1041796207428s
Pipeline stage MKMLDeployer completed in 94.13s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service rirv938-llama-8b-scaleup-400k-v3 ready after 91.02529239654541s
Pipeline stage MKMLDeployer completed in 94.11s
Received healthy response to inference request in 6.875030279159546s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.999146938323975s
Received healthy response to inference request in 5.838494062423706s
Received healthy response to inference request in 2.5466840267181396s
Received healthy response to inference request in 3.137603282928467s
Received healthy response to inference request in 4.695525646209717s
Received healthy response to inference request in 4.727989196777344s
Received healthy response to inference request in 3.140216588973999s
5 requests
0 failed requests
5th percentile: 2.6653905391693113
10th percentile: 2.7840970516204835
20th percentile: 3.0215100765228273
30th percentile: 3.4512784004211428
40th percentile: 4.073402023315429
Received healthy response to inference request in 4.965777635574341s
50th percentile: 4.695525646209717
60th percentile: 4.81697416305542
70th percentile: 4.938422679901123
80th percentile: 5.374323606491089
90th percentile: 6.124676942825317
95th percentile: 6.499853610992432
Received healthy response to inference request in 2.7075490951538086s
99th percentile: 6.799994945526123
5 requests
mean time: 4.451320695877075
0 failed requests
Pipeline stage StressChecker completed in 34.67s
5th percentile: 2.79355993270874
run pipeline stage %s
10th percentile: 2.8795707702636717
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
20th percentile: 3.0515924453735352
run_pipeline:run_in_cloud %s
30th percentile: 3.4556804656982423
starting trigger_guanaco_pipeline args=%s
40th percentile: 4.091834831237793
50th percentile: 4.727989196777344
triggered trigger_guanaco_pipeline args=%s
60th percentile: 4.823104572296143
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.37s
70th percentile: 4.918219947814942
run pipeline stage %s
80th percentile: 5.140320920944214
Running pipeline stage TriggerMKMLProfilingPipeline
90th percentile: 5.48940749168396
run_pipeline:run_in_cloud %s
95th percentile: 5.663950777053833
starting trigger_guanaco_pipeline args=%s
99th percentile: 5.8035854053497316
mean time: 4.275482654571533
triggered trigger_guanaco_pipeline args=%s
Pipeline stage StressChecker completed in 34.95s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.04s
run pipeline stage %s
Shutdown handler de-registered
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
rirv938-llama-8b-scaleup-200k_v1 status is now deployed due to DeploymentManager action
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.55s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.22s
Shutdown handler de-registered
rirv938-llama-8b-scaleup-400k_v3 status is now deployed due to DeploymentManager action
Inference service rirv938-mistral-24b-sca-46352-v1 ready after 91.1574923992157s
Pipeline stage MKMLDeployer completed in 93.72s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service rirv938-mistral-24b-scal-2131-v4 ready after 91.0551586151123s
Pipeline stage MKMLDeployer completed in 93.59s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.692603826522827s
Received healthy response to inference request in 3.2257208824157715s
Received healthy response to inference request in 6.135567903518677s
Received healthy response to inference request in 3.2589404582977295s
Received healthy response to inference request in 3.810298442840576s
Received healthy response to inference request in 3.2009634971618652s
Received healthy response to inference request in 5.690638542175293s
Received healthy response to inference request in 3.3886682987213135s
5 requests
0 failed requests
5th percentile: 3.2059149742126465
10th percentile: 3.2108664512634277
20th percentile: 3.2207694053649902
30th percentile: 3.232364797592163
40th percentile: 3.2456526279449465
50th percentile: 3.2589404582977295
60th percentile: 3.310831594467163
70th percentile: 3.3627227306365968
80th percentile: 3.449455404281616
90th percentile: 3.571029615402222
95th percentile: 3.6318167209625245
99th percentile: 3.6804464054107666
mean time: 3.3533793926239013
Received healthy response to inference request in 4.785853862762451s
Pipeline stage StressChecker completed in 23.57s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.90s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Received healthy response to inference request in 3.1368019580841064s
run_pipeline:run_in_cloud %s
5 requests
starting trigger_guanaco_pipeline args=%s
0 failed requests
5th percentile: 3.2715012550354006
10th percentile: 3.4062005519866942
triggered trigger_guanaco_pipeline args=%s
20th percentile: 3.675599145889282
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.50s
30th percentile: 4.005409526824951
Shutdown handler de-registered
40th percentile: 4.3956316947937015
40th percentile: 4.3956316947937015
rirv938-mistral-24b-sca_46352_v1 status is now deployed due to DeploymentManager action
50th percentile: 4.785853862762451
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-sca-46352-v1-profiler
Waiting for inference service rirv938-mistral-24b-sca-46352-v1-profiler to be ready
Inference service rirv938-mistral-24b-sca-46352-v1-profiler ready after 110.44333100318909s
Pipeline stage MKMLProfilerDeployer completed in 110.99s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deploqddmd:/code/chaiverse_profiler_1741724235 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deploqddmd --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741724235 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1741724235/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-sca-46352-v1-profiler is running
Tearing down inference service rirv938-mistral-24b-sca-46352-v1-profiler
Service rirv938-mistral-24b-sca-46352-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.51s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-sca-46352-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.30s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.21s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-sca-46352-v1-profiler
Waiting for inference service rirv938-mistral-24b-sca-46352-v1-profiler to be ready
Inference service rirv938-mistral-24b-sca-46352-v1-profiler ready after 60.28454089164734s
Pipeline stage MKMLProfilerDeployer completed in 60.79s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deplojkvpd:/code/chaiverse_profiler_1741727811 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deplojkvpd --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741727811 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1741727811/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-sca-46352-v1-profiler is running
Tearing down inference service rirv938-mistral-24b-sca-46352-v1-profiler
Service rirv938-mistral-24b-sca-46352-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.37s
Shutdown handler de-registered
rirv938-mistral-24b-sca_46352_v1 status is now inactive due to auto deactivation removed underperforming models
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-sca-46352-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.42s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-sca-46352-v1-profiler
Waiting for inference service rirv938-mistral-24b-sca-46352-v1-profiler to be ready
Inference service rirv938-mistral-24b-sca-46352-v1-profiler ready after 110.44562911987305s
Pipeline stage MKMLProfilerDeployer completed in 110.96s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deplovnzg4:/code/chaiverse_profiler_1741731501 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-91547ceaa258800965f4d16281ac97e3-deplovnzg4 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741731501 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1741731501/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-sca-46352-v1-profiler is running
Tearing down inference service rirv938-mistral-24b-sca-46352-v1-profiler
Service rirv938-mistral-24b-sca-46352-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.64s
Shutdown handler de-registered
rirv938-mistral-24b-sca_46352_v1 status is now torndown due to DeploymentManager action