Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-mistral-24b-scal-2131-v5-mkmlizer
Waiting for job on rirv938-mistral-24b-scal-2131-v5-mkmlizer to finish
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Version: 0.25.10 ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Version: 0.25.10 ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-mistral-24b-1m-86706-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Chai Research Corp. ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Version: 0.25.10 ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ https://mk1.ai ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Version: 0.25.10 ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ belonging to: ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ https://mk1.ai ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Chai Research Corp. ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ belonging to: ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ║
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Chai Research Corp. ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ║ ║
rirv938-mistral-24b-scal-2131-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: Downloaded to shared memory in 31.804s
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpmwisoiwu, device:0
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: Downloaded to shared memory in 26.412s
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpj0l18ve0, device:0
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: quantized model in 20.850s
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: Processed model rirv938/llama_8b_1m_scaleup_20250312 in 47.262s
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v2
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: quantized model in 21.085s
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v2/config.json
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: Processed model rirv938/llama_8b_1m_scaleup_20250312 in 52.890s
rirv938-llama-8b-1m-sca-88531-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v2/special_tokens_map.json
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v1
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v1/special_tokens_map.json
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v1/config.json
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v1/tokenizer_config.json
rirv938-llama-8b-1m-sca-88531-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-llama-8b-1m-sca-88531-v1/tokenizer.json
rirv938-mistral-24b-scal-2131-v5-mkmlizer: Downloaded to shared memory in 63.367s
rirv938-mistral-24b-scal-2131-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpqreb7woz, device:0
rirv938-mistral-24b-scal-2131-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-mistral-24b-1m-86706-v2-mkmlizer: Downloaded to shared memory in 85.912s
rirv938-mistral-24b-1m-86706-v1-mkmlizer: Downloaded to shared memory in 88.458s
rirv938-mistral-24b-1m-86706-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpdukacz9t, device:0
rirv938-mistral-24b-1m-86706-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpnlftdyp1, device:0
rirv938-mistral-24b-1m-86706-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-mistral-24b-1m-86706-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Job rirv938-llama-8b-1m-sca-88531-v1-mkmlizer completed after 129.62s with status: succeeded
Stopping job with name rirv938-llama-8b-1m-sca-88531-v1-mkmlizer
Pipeline stage MKMLizer completed in 133.06s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Job rirv938-llama-8b-1m-sca-88531-v2-mkmlizer completed after 132.44s with status: succeeded
Pipeline stage MKMLTemplater completed in 1.25s
Stopping job with name rirv938-llama-8b-1m-sca-88531-v2-mkmlizer
run pipeline stage %s
Pipeline stage MKMLizer completed in 135.91s
Running pipeline stage MKMLDeployer
run pipeline stage %s
Running pipeline stage MKMLTemplater
Creating inference service rirv938-llama-8b-1m-sca-88531-v1
Pipeline stage MKMLTemplater completed in 1.12s
Waiting for inference service rirv938-llama-8b-1m-sca-88531-v1 to be ready
run pipeline stage %s
Running pipeline stage MKMLDeployer
rirv938-mistral-24b-scal-2131-v5-mkmlizer: quantized model in 47.914s
Creating inference service rirv938-llama-8b-1m-sca-88531-v2
rirv938-mistral-24b-scal-2131-v5-mkmlizer: Processed model rirv938/mistral_24b_scaleup_400k in 111.285s
rirv938-mistral-24b-scal-2131-v5-mkmlizer: creating bucket guanaco-mkml-models
Waiting for inference service rirv938-llama-8b-1m-sca-88531-v2 to be ready
rirv938-mistral-24b-scal-2131-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-mistral-24b-scal-2131-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5
rirv938-mistral-24b-scal-2131-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5/config.json
rirv938-mistral-24b-scal-2131-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5/special_tokens_map.json
rirv938-mistral-24b-1m-86706-v2-mkmlizer: quantized model in 49.551s
rirv938-mistral-24b-scal-2131-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5/tokenizer_config.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: quantized model in 49.666s
rirv938-mistral-24b-1m-86706-v2-mkmlizer: Processed model rirv938/mistral_24b_1m_reward_model_512_tokens in 135.467s
rirv938-mistral-24b-scal-2131-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5/tokenizer.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: Processed model rirv938/mistral_24b_1m_reward_model_512_tokens in 138.125s
rirv938-mistral-24b-1m-86706-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-mistral-24b-scal-2131-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-scal-2131-v5/flywheel_model.2.safetensors
rirv938-mistral-24b-1m-86706-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-mistral-24b-1m-86706-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-mistral-24b-1m-86706-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-mistral-24b-1m-86706-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2
rirv938-mistral-24b-1m-86706-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/config.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/config.json
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/special_tokens_map.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/special_tokens_map.json
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/tokenizer_config.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/tokenizer_config.json
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/tokenizer.json
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/tokenizer.json
rirv938-mistral-24b-scal-2131-v5-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:09, 37.39it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:10, 33.21it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:11, 31.40it/s]
Loading 0: 4%|▍ | 16/363 [00:00<00:12, 28.50it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:10, 32.21it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:11, 29.51it/s]
Loading 0: 9%|▉ | 32/363 [00:00<00:09, 35.02it/s]
Loading 0: 10%|▉ | 36/363 [00:01<00:14, 21.87it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:14, 22.84it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:10, 28.88it/s]
Loading 0: 14%|█▍ | 52/363 [00:01<00:11, 27.76it/s]
Loading 0: 16%|█▌ | 57/363 [00:01<00:10, 30.11it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:10, 28.50it/s]
Loading 0: 18%|█▊ | 65/363 [00:02<00:10, 28.81it/s]
Loading 0: 19%|█▉ | 70/363 [00:02<00:12, 24.14it/s]
Loading 0: 20%|██ | 73/363 [00:02<00:13, 20.92it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:10, 26.12it/s]
Loading 0: 23%|██▎ | 82/363 [00:03<00:11, 25.16it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:10, 26.89it/s]
Loading 0: 25%|██▍ | 89/363 [00:03<00:09, 27.48it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:11, 22.99it/s]
Loading 0: 27%|██▋ | 99/363 [00:03<00:08, 30.39it/s]
Loading 0: 28%|██▊ | 103/363 [00:03<00:09, 28.37it/s]
Loading 0: 29%|██▉ | 107/363 [00:04<00:10, 23.42it/s]
Loading 0: 31%|███ | 112/363 [00:04<00:09, 26.33it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:09, 25.80it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:08, 29.22it/s]
Loading 0: 34%|███▍ | 124/363 [00:04<00:08, 28.23it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 30.76it/s]
Loading 0: 37%|███▋ | 133/363 [00:04<00:08, 28.59it/s]
Loading 0: 38%|███▊ | 138/363 [00:04<00:07, 31.06it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:07, 28.50it/s]
Loading 0: 39%|███▉ | 142/363 [00:19<00:07, 28.50it/s]
Loading 0: 39%|███▉ | 143/363 [00:19<04:48, 1.31s/it]
Loading 0: 40%|███▉ | 145/363 [00:19<03:50, 1.06s/it]
Loading 0: 41%|████ | 149/363 [00:19<02:25, 1.47it/s]
Loading 0: 42%|████▏ | 152/363 [00:19<01:47, 1.95it/s]
Loading 0: 43%|████▎ | 155/363 [00:20<01:18, 2.66it/s]
Loading 0: 44%|████▎ | 158/363 [00:20<00:58, 3.49it/s]
Loading 0: 45%|████▌ | 165/363 [00:20<00:30, 6.43it/s]
Loading 0: 46%|████▋ | 168/363 [00:20<00:25, 7.57it/s]
Loading 0: 48%|████▊ | 174/363 [00:20<00:16, 11.25it/s]
Loading 0: 49%|████▉ | 178/363 [00:20<00:14, 13.18it/s]
Loading 0: 50%|█████ | 182/363 [00:21<00:11, 15.76it/s]
Loading 0: 52%|█████▏ | 187/363 [00:21<00:10, 17.24it/s]
Loading 0: 52%|█████▏ | 190/363 [00:21<00:09, 17.56it/s]
Loading 0: 53%|█████▎ | 194/363 [00:21<00:09, 17.86it/s]
Loading 0: 55%|█████▌ | 201/363 [00:21<00:06, 24.42it/s]
Loading 0: 56%|█████▋ | 205/363 [00:21<00:06, 24.34it/s]
Loading 0: 58%|█████▊ | 210/363 [00:22<00:05, 27.56it/s]
Loading 0: 59%|█████▉ | 214/363 [00:22<00:05, 26.77it/s]
Loading 0: 60%|██████ | 218/363 [00:22<00:05, 27.29it/s]
Loading 0: 61%|██████ | 221/363 [00:22<00:05, 26.89it/s]
Loading 0: 62%|██████▏ | 224/363 [00:22<00:06, 21.41it/s]
Loading 0: 63%|██████▎ | 229/363 [00:22<00:05, 24.76it/s]
Loading 0: 64%|██████▍ | 232/363 [00:23<00:05, 23.96it/s]
Loading 0: 65%|██████▌ | 237/363 [00:23<00:04, 27.42it/s]
Loading 0: 66%|██████▌ | 240/363 [00:23<00:05, 24.58it/s]
Loading 0: 68%|██████▊ | 246/363 [00:23<00:03, 29.28it/s]
Loading 0: 69%|██████▉ | 250/363 [00:23<00:04, 27.57it/s]
Loading 0: 70%|███████ | 255/363 [00:23<00:03, 30.48it/s]
Loading 0: 71%|███████▏ | 259/363 [00:23<00:03, 28.30it/s]
Loading 0: 73%|███████▎ | 265/363 [00:24<00:02, 35.14it/s]
Loading 0: 74%|███████▍ | 269/363 [00:24<00:04, 22.68it/s]
Loading 0: 75%|███████▌ | 273/363 [00:24<00:03, 23.23it/s]
Loading 0: 76%|███████▌ | 276/363 [00:24<00:03, 22.05it/s]
Loading 0: 78%|███████▊ | 282/363 [00:24<00:03, 26.42it/s]
Loading 0: 79%|███████▊ | 285/363 [00:25<00:03, 24.43it/s]
Loading 0: 80%|████████ | 291/363 [00:25<00:02, 30.06it/s]
Loading 0: 81%|████████▏ | 295/363 [00:25<00:02, 28.62it/s]
Loading 0: 82%|████████▏ | 299/363 [00:25<00:02, 28.20it/s]
Loading 0: 84%|████████▎ | 304/363 [00:25<00:02, 23.38it/s]
Loading 0: 85%|████████▍ | 307/363 [00:25<00:02, 21.94it/s]
Loading 0: 85%|████████▍ | 308/363 [00:40<00:02, 21.94it/s]
Loading 0: 85%|████████▌ | 309/363 [00:40<01:09, 1.29s/it]
Loading 0: 86%|████████▌ | 311/363 [00:40<00:54, 1.05s/it]
Loading 0: 87%|████████▋ | 316/363 [00:40<00:29, 1.60it/s]
Loading 0: 88%|████████▊ | 319/363 [00:40<00:21, 2.08it/s]
Loading 0: 89%|████████▊ | 322/363 [00:40<00:14, 2.77it/s]
Loading 0: 90%|█████████ | 327/363 [00:41<00:08, 4.35it/s]
Loading 0: 91%|█████████ | 330/363 [00:41<00:06, 5.39it/s]
Loading 0: 92%|█████████▏| 335/363 [00:41<00:03, 7.86it/s]
Loading 0: 93%|█████████▎| 338/363 [00:41<00:02, 9.43it/s]
Loading 0: 94%|█████████▍| 341/363 [00:41<00:02, 10.05it/s]
Loading 0: 95%|█████████▌| 345/363 [00:41<00:01, 13.33it/s]
Loading 0: 96%|█████████▌| 348/363 [00:42<00:01, 14.43it/s]
Loading 0: 98%|█████████▊| 354/363 [00:42<00:00, 20.27it/s]
Loading 0: 99%|█████████▊| 358/363 [00:42<00:00, 21.37it/s]
Job rirv938-mistral-24b-scal-2131-v5-mkmlizer completed after 157.17s with status: succeeded
Stopping job with name rirv938-mistral-24b-scal-2131-v5-mkmlizer
Pipeline stage MKMLizer completed in 160.75s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 1.02s
run pipeline stage %s
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/flywheel_model.0.safetensors
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/flywheel_model.0.safetensors
Running pipeline stage MKMLDeployer
rirv938-mistral-24b-1m-86706-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v2/flywheel_model.1.safetensors
rirv938-mistral-24b-1m-86706-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-mistral-24b-1m-86706-v1/flywheel_model.1.safetensors
Creating inference service rirv938-mistral-24b-scal-2131-v5
rirv938-mistral-24b-1m-86706-v2-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:09, 36.47it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:12, 29.36it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:12, 28.74it/s]
Loading 0: 4%|▍ | 15/363 [00:00<00:14, 23.93it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:11, 30.22it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:12, 27.45it/s]
Loading 0: 8%|▊ | 30/363 [00:01<00:10, 31.85it/s]
Loading 0: 9%|▉ | 34/363 [00:01<00:14, 22.43it/s]
Loading 0: 10%|█ | 37/363 [00:01<00:15, 21.11it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:15, 20.20it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:11, 26.58it/s]
Loading 0: 14%|█▍ | 51/363 [00:02<00:13, 23.93it/s]
Loading 0: 16%|█▌ | 57/363 [00:02<00:10, 27.85it/s]
Loading 0: 17%|█▋ | 60/363 [00:02<00:12, 24.73it/s]
Loading 0: 18%|█▊ | 65/363 [00:02<00:10, 27.10it/s]
Loading 0: 19%|█▉ | 70/363 [00:02<00:11, 24.64it/s]
Loading 0: 20%|██ | 73/363 [00:02<00:14, 20.55it/s]
Loading 0: 22%|██▏ | 79/363 [00:03<00:11, 25.42it/s]
Loading 0: 23%|██▎ | 82/363 [00:03<00:11, 24.91it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:10, 25.82it/s]
Loading 0: 25%|██▍ | 89/363 [00:03<00:10, 25.64it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:12, 20.97it/s]
Loading 0: 27%|██▋ | 99/363 [00:03<00:09, 27.45it/s]
Loading 0: 28%|██▊ | 102/363 [00:04<00:10, 24.67it/s]
Loading 0: 29%|██▉ | 107/363 [00:04<00:11, 23.20it/s]
Loading 0: 31%|███ | 112/363 [00:04<00:09, 25.92it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:09, 25.52it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:08, 28.33it/s]
Loading 0: 34%|███▍ | 123/363 [00:04<00:09, 24.94it/s]
Loading 0: 36%|███▌ | 129/363 [00:05<00:08, 29.11it/s]
Loading 0: 36%|███▋ | 132/363 [00:05<00:09, 25.42it/s]
Loading 0: 38%|███▊ | 138/363 [00:05<00:07, 29.39it/s]
Loading 0: 39%|███▉ | 141/363 [00:05<00:08, 26.07it/s]
Loading 0: 39%|███▉ | 142/363 [00:15<00:08, 26.07it/s]
Loading 0: 39%|███▉ | 143/363 [00:19<04:38, 1.27s/it]
Loading 0: 40%|███▉ | 145/363 [00:20<03:43, 1.03s/it]
Loading 0: 41%|████ | 149/363 [00:20<02:23, 1.49it/s]
Loading 0: 42%|████▏ | 152/363 [00:20<01:47, 1.97it/s]
Loading 0: 43%|████▎ | 155/363 [00:20<01:18, 2.66it/s]
Loading 0: 44%|████▎ | 158/363 [00:20<00:59, 3.44it/s]
Loading 0: 45%|████▍ | 163/363 [00:20<00:36, 5.53it/s]
Loading 0: 46%|████▌ | 166/363 [00:21<00:28, 6.98it/s]
Loading 0: 47%|████▋ | 169/363 [00:21<00:22, 8.66it/s]
Loading 0: 48%|████▊ | 174/363 [00:21<00:15, 12.19it/s]
Loading 0: 49%|████▉ | 177/363 [00:21<00:14, 13.21it/s]
Loading 0: 50%|█████ | 182/363 [00:21<00:10, 17.26it/s]
Loading 0: 52%|█████▏ | 187/363 [00:21<00:09, 18.11it/s]
Loading 0: 52%|█████▏ | 190/363 [00:22<00:09, 17.86it/s]
Loading 0: 53%|█████▎ | 194/363 [00:22<00:09, 18.01it/s]
Loading 0: 55%|█████▌ | 201/363 [00:22<00:06, 24.08it/s]
Loading 0: 56%|█████▌ | 204/363 [00:22<00:07, 22.45it/s]
Loading 0: 58%|█████▊ | 210/363 [00:22<00:05, 26.94it/s]
Loading 0: 59%|█████▊ | 213/363 [00:22<00:06, 23.74it/s]
Loading 0: 60%|██████ | 218/363 [00:23<00:05, 26.20it/s]
Loading 0: 61%|██████ | 221/363 [00:23<00:05, 25.57it/s]
Loading 0: 62%|██████▏ | 224/363 [00:23<00:06, 20.38it/s]
Loading 0: 63%|██████▎ | 228/363 [00:23<00:05, 24.08it/s]
Loading 0: 64%|██████▎ | 231/363 [00:23<00:05, 22.10it/s]
Loading 0: 65%|██████▌ | 237/363 [00:23<00:04, 27.42it/s]
Loading 0: 66%|██████▌ | 240/363 [00:24<00:05, 24.38it/s]
Loading 0: 68%|██████▊ | 246/363 [00:24<00:04, 28.91it/s]
Loading 0: 69%|██████▉ | 250/363 [00:24<00:04, 27.53it/s]
Loading 0: 70%|███████ | 255/363 [00:24<00:03, 29.95it/s]
Loading 0: 71%|███████▏ | 259/363 [00:24<00:03, 28.52it/s]
Loading 0: 73%|███████▎ | 264/363 [00:24<00:03, 32.74it/s]
Loading 0: 74%|███████▍ | 268/363 [00:25<00:03, 23.76it/s]
Loading 0: 75%|███████▍ | 271/363 [00:25<00:04, 22.06it/s]
Loading 0: 76%|███████▌ | 275/363 [00:25<00:04, 21.10it/s]
Loading 0: 78%|███████▊ | 282/363 [00:25<00:03, 26.98it/s]
Loading 0: 79%|███████▊ | 285/363 [00:25<00:03, 24.57it/s]
Loading 0: 80%|████████ | 291/363 [00:25<00:02, 29.16it/s]
Loading 0: 81%|████████▏ | 295/363 [00:26<00:02, 27.43it/s]
Loading 0: 82%|████████▏ | 299/363 [00:26<00:02, 27.10it/s]
Loading 0: 84%|████████▎ | 304/363 [00:26<00:02, 24.43it/s]
Loading 0: 85%|████████▍ | 307/363 [00:26<00:02, 22.52it/s]
Loading 0: 85%|████████▍ | 308/363 [00:41<00:02, 22.52it/s]
Loading 0: 85%|████████▌ | 309/363 [00:41<01:09, 1.28s/it]
Loading 0: 86%|████████▌ | 311/363 [00:41<00:54, 1.05s/it]
Loading 0: 87%|████████▋ | 316/363 [00:41<00:29, 1.59it/s]
Loading 0: 88%|████████▊ | 319/363 [00:41<00:21, 2.09it/s]
Loading 0: 89%|████████▊ | 322/363 [00:41<00:14, 2.77it/s]
Loading 0: 90%|█████████ | 327/363 [00:41<00:08, 4.34it/s]
Loading 0: 91%|█████████ | 330/363 [00:42<00:06, 5.31it/s]
Loading 0: 92%|█████████▏| 335/363 [00:42<00:03, 7.65it/s]
Loading 0: 93%|█████████▎| 338/363 [00:42<00:02, 9.03it/s]
Loading 0: 94%|█████████▍| 341/363 [00:42<00:02, 9.37it/s]
Loading 0: 95%|█████████▌| 345/363 [00:42<00:01, 12.36it/s]
Loading 0: 96%|█████████▌| 348/363 [00:43<00:01, 12.66it/s]
Loading 0: 97%|█████████▋| 352/363 [00:43<00:00, 15.90it/s]
Loading 0: 98%|█████████▊| 355/363 [00:43<00:00, 17.18it/s]
Loading 0: 99%|█████████▊| 358/363 [00:43<00:00, 18.22it/s]
rirv938-mistral-24b-1m-86706-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:09, 36.85it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:11, 29.59it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:12, 29.12it/s]
Loading 0: 4%|▍ | 15/363 [00:00<00:14, 24.02it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:11, 30.64it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:12, 27.97it/s]
Loading 0: 8%|▊ | 30/363 [00:00<00:10, 32.85it/s]
Loading 0: 9%|▉ | 34/363 [00:01<00:14, 23.37it/s]
Loading 0: 10%|█ | 37/363 [00:01<00:15, 21.59it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:15, 20.50it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:11, 26.97it/s]
Loading 0: 14%|█▍ | 51/363 [00:01<00:12, 24.21it/s]
Loading 0: 16%|█▌ | 57/363 [00:02<00:10, 28.48it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:11, 26.67it/s]
Loading 0: 18%|█▊ | 65/363 [00:02<00:11, 26.80it/s]
Loading 0: 19%|█▉ | 70/363 [00:02<00:12, 24.01it/s]
Loading 0: 20%|██ | 73/363 [00:02<00:14, 20.57it/s]
Loading 0: 22%|██▏ | 79/363 [00:03<00:11, 25.46it/s]
Loading 0: 23%|██▎ | 82/363 [00:03<00:11, 24.95it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:10, 25.81it/s]
Loading 0: 25%|██▍ | 89/363 [00:03<00:10, 25.41it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:12, 21.01it/s]
Loading 0: 27%|██▋ | 99/363 [00:03<00:09, 27.79it/s]
Loading 0: 28%|██▊ | 102/363 [00:04<00:10, 24.77it/s]
Loading 0: 29%|██▉ | 107/363 [00:04<00:11, 23.25it/s]
Loading 0: 31%|███ | 112/363 [00:04<00:09, 25.98it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:09, 25.61it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:08, 28.36it/s]
Loading 0: 34%|███▍ | 123/363 [00:04<00:09, 24.92it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 29.34it/s]
Loading 0: 37%|███▋ | 133/363 [00:05<00:08, 27.35it/s]
Loading 0: 38%|███▊ | 138/363 [00:05<00:07, 29.18it/s]
Loading 0: 39%|███▉ | 141/363 [00:05<00:08, 25.62it/s]
Loading 0: 40%|███▉ | 144/363 [00:19<04:18, 1.18s/it]
Loading 0: 40%|███▉ | 145/363 [00:20<03:54, 1.07s/it]
Loading 0: 41%|████ | 149/363 [00:20<02:27, 1.45it/s]
Loading 0: 42%|████▏ | 152/363 [00:20<01:49, 1.92it/s]
Loading 0: 42%|████▏ | 154/363 [00:20<01:28, 2.36it/s]
Loading 0: 44%|████▎ | 158/363 [00:20<00:58, 3.51it/s]
Loading 0: 45%|████▌ | 165/363 [00:20<00:31, 6.36it/s]
Loading 0: 46%|████▋ | 168/363 [00:21<00:26, 7.44it/s]
Loading 0: 48%|████▊ | 174/363 [00:21<00:17, 10.96it/s]
Loading 0: 49%|████▉ | 177/363 [00:21<00:15, 11.99it/s]
Loading 0: 50%|█████ | 182/363 [00:21<00:11, 15.52it/s]
Loading 0: 52%|█████▏ | 187/363 [00:21<00:10, 16.65it/s]
Loading 0: 52%|█████▏ | 190/363 [00:22<00:10, 16.62it/s]
Loading 0: 53%|█████▎ | 194/363 [00:22<00:09, 17.14it/s]
Loading 0: 55%|█████▌ | 201/363 [00:22<00:07, 23.02it/s]
Loading 0: 56%|█████▌ | 204/363 [00:22<00:07, 21.68it/s]
Loading 0: 58%|█████▊ | 210/363 [00:22<00:05, 26.22it/s]
Loading 0: 59%|█████▊ | 213/363 [00:22<00:06, 23.77it/s]
Loading 0: 60%|██████ | 218/363 [00:23<00:05, 26.01it/s]
Loading 0: 61%|██████ | 221/363 [00:23<00:05, 25.36it/s]
Loading 0: 62%|██████▏ | 224/363 [00:23<00:07, 19.82it/s]
Loading 0: 63%|██████▎ | 229/363 [00:23<00:05, 23.31it/s]
Loading 0: 64%|██████▍ | 232/363 [00:23<00:05, 23.50it/s]
Loading 0: 65%|██████▌ | 237/363 [00:23<00:04, 26.80it/s]
Loading 0: 66%|██████▌ | 240/363 [00:24<00:05, 24.08it/s]
Loading 0: 68%|██████▊ | 246/363 [00:24<00:04, 29.08it/s]
Loading 0: 69%|██████▉ | 250/363 [00:24<00:04, 27.43it/s]
Loading 0: 70%|███████ | 255/363 [00:24<00:03, 30.00it/s]
Loading 0: 71%|███████▏ | 259/363 [00:24<00:03, 27.76it/s]
Loading 0: 73%|███████▎ | 264/363 [00:24<00:03, 31.78it/s]
Loading 0: 74%|███████▍ | 268/363 [00:25<00:04, 22.88it/s]
Loading 0: 75%|███████▍ | 271/363 [00:25<00:04, 21.05it/s]
Loading 0: 76%|███████▌ | 275/363 [00:25<00:04, 20.36it/s]
Loading 0: 78%|███████▊ | 282/363 [00:25<00:03, 26.26it/s]
Loading 0: 79%|███████▊ | 285/363 [00:25<00:03, 24.17it/s]
Loading 0: 80%|████████ | 291/363 [00:25<00:02, 29.06it/s]
Loading 0: 81%|████████▏ | 295/363 [00:26<00:02, 27.55it/s]
Loading 0: 82%|████████▏ | 299/363 [00:26<00:02, 27.43it/s]
Loading 0: 84%|████████▎ | 304/363 [00:26<00:02, 24.98it/s]
Loading 0: 85%|████████▍ | 307/363 [00:26<00:02, 22.56it/s]
Loading 0: 85%|████████▍ | 308/363 [00:41<00:02, 22.56it/s]
Loading 0: 85%|████████▌ | 309/363 [00:41<01:10, 1.31s/it]
Loading 0: 86%|████████▌ | 311/363 [00:41<00:55, 1.07s/it]
Loading 0: 87%|████████▋ | 316/363 [00:41<00:29, 1.57it/s]
Loading 0: 88%|████████▊ | 319/363 [00:41<00:21, 2.05it/s]
Loading 0: 89%|████████▊ | 322/363 [00:42<00:15, 2.73it/s]
Loading 0: 90%|█████████ | 327/363 [00:42<00:08, 4.27it/s]
Loading 0: 91%|█████████ | 330/363 [00:42<00:06, 5.27it/s]
Loading 0: 92%|█████████▏| 335/363 [00:42<00:03, 7.69it/s]
Loading 0: 93%|█████████▎| 338/363 [00:42<00:02, 9.15it/s]
Loading 0: 94%|█████████▍| 341/363 [00:42<00:02, 9.56it/s]
Loading 0: 95%|█████████▌| 345/363 [00:43<00:01, 12.65it/s]
Loading 0: 96%|█████████▌| 348/363 [00:43<00:01, 13.26it/s]
Loading 0: 98%|█████████▊| 354/363 [00:43<00:00, 18.25it/s]
Loading 0: 98%|█████████▊| 357/363 [00:43<00:00, 17.75it/s]
Waiting for inference service rirv938-mistral-24b-scal-2131-v5 to be ready
Job rirv938-mistral-24b-1m-86706-v2-mkmlizer completed after 174.96s with status: succeeded
Job rirv938-mistral-24b-1m-86706-v1-mkmlizer completed after 178.5s with status: succeeded
Stopping job with name rirv938-mistral-24b-1m-86706-v2-mkmlizer
Stopping job with name rirv938-mistral-24b-1m-86706-v1-mkmlizer
Pipeline stage MKMLizer completed in 178.59s
Pipeline stage MKMLizer completed in 182.31s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 1.42s
Pipeline stage MKMLTemplater completed in 1.99s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Running pipeline stage MKMLDeployer
Creating inference service rirv938-mistral-24b-1m-86706-v2
Creating inference service rirv938-mistral-24b-1m-86706-v1
Waiting for inference service rirv938-mistral-24b-1m-86706-v2 to be ready
Waiting for inference service rirv938-mistral-24b-1m-86706-v1 to be ready
Inference service rirv938-llama-8b-1m-sca-88531-v1 ready after 91.74766731262207s
Pipeline stage MKMLDeployer completed in 97.16s
run pipeline stage %s
Running pipeline stage StressChecker
Inference service rirv938-llama-8b-1m-sca-88531-v2 ready after 91.15564346313477s
Pipeline stage MKMLDeployer completed in 96.26s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 9.696150302886963s
Received healthy response to inference request in 6.139187574386597s
Received healthy response to inference request in 3.6020054817199707s
Received healthy response to inference request in 3.779045581817627s
Received healthy response to inference request in 7.714804410934448s
Received healthy response to inference request in 4.864864110946655s
Received healthy response to inference request in 6.875163793563843s
Inference service rirv938-mistral-24b-scal-2131-v5 ready after 91.10835313796997s
Pipeline stage MKMLDeployer completed in 97.27s
run pipeline stage %s
Received healthy response to inference request in 3.997162103652954s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8202083110809326s
5 requests
0 failed requests
5th percentile: 3.637413501739502
10th percentile: 3.672821521759033
Received healthy response to inference request in 5.518310308456421s
20th percentile: 3.7436375617980957
5 requests
Received healthy response to inference request in 6.449045419692993s
30th percentile: 3.7872781276702883
Inference service rirv938-mistral-24b-1m-86706-v2 ready after 91.08054852485657s
0 failed requests
Inference service rirv938-mistral-24b-1m-86706-v1 ready after 91.24917912483215s
40th percentile: 3.8037432193756104
Pipeline stage MKMLDeployer completed in 97.89s
5th percentile: 4.170702505111694
Pipeline stage MKMLDeployer completed in 99.10s
50th percentile: 3.8202083110809326
run pipeline stage %s
10th percentile: 4.344242906570434
run pipeline stage %s
60th percentile: 5.042190504074096
Running pipeline stage StressChecker
Received healthy response to inference request in 7.194443464279175s
20th percentile: 4.691323709487915
Running pipeline stage StressChecker
70th percentile: 6.26417269706726
30th percentile: 4.995553350448608
80th percentile: 7.439361095428468
40th percentile: 5.2569318294525145
90th percentile: 8.567755699157715
50th percentile: 5.518310308456421
95th percentile: 9.131953001022339
60th percentile: 5.766661214828491
Received healthy response to inference request in 5.355046987533569s
99th percentile: 9.583310842514038
70th percentile: 6.015012121200561
Received healthy response to inference request in 7.1233742237091064s
Received healthy response to inference request in 5.419728994369507s
mean time: 5.554514694213867
80th percentile: 6.454310941696168
Pipeline stage StressChecker completed in 65.91s
90th percentile: 7.084557676315308
run pipeline stage %s
95th percentile: 7.399681043624877
Received healthy response to inference request in 4.028217077255249s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Received healthy response to inference request in 4.161670684814453s
99th percentile: 7.651779737472534
Received healthy response to inference request in 4.2908034324646s
run_pipeline:run_in_cloud %s
mean time: 5.646865701675415
starting trigger_guanaco_pipeline args=%s
Pipeline stage StressChecker completed in 68.08s
run pipeline stage %s
triggered trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 4.3179261684417725s
Received healthy response to inference request in 2.9550232887268066s
Received healthy response to inference request in 7.579199314117432s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 10.32s
5 requests
run_pipeline:run_in_cloud %s
run pipeline stage %s
0 failed requests
starting trigger_guanaco_pipeline args=%s
Running pipeline stage TriggerMKMLProfilingPipeline
5th percentile: 4.293583059310913
triggered trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 6.474468469619751s
run_pipeline:run_in_cloud %s
10th percentile: 4.558949041366577
Received healthy response to inference request in 7.9870874881744385s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 10.16s
starting trigger_guanaco_pipeline args=%s
20th percentile: 5.089681005477905
run pipeline stage %s
30th percentile: 5.573846673965454
triggered trigger_guanaco_pipeline args=%s
Running pipeline stage TriggerMKMLProfilingPipeline
Received healthy response to inference request in 5.490382432937622s
Received healthy response to inference request in 3.766071081161499s
40th percentile: 6.0114460468292235
Pipeline stage TriggerMKMLProfilingPipeline completed in 12.28s
run_pipeline:run_in_cloud %s
5 requests
5 requests
50th percentile: 6.449045419692993
Shutdown handler de-registered
0 failed requests
0 failed requests
60th percentile: 6.7472046375274655
rirv938-llama-8b-1m-sca_88531_v1 status is now deployed due to DeploymentManager action
5th percentile: 4.192921781539917
triggered trigger_guanaco_pipeline args=%s
5th percentile: 3.117232847213745
70th percentile: 7.045363855361939
10th percentile: 4.224172878265381
Pipeline stage TriggerMKMLProfilingPipeline completed in 14.96s
10th percentile: 3.2794424057006837
80th percentile: 7.2713946342468265
20th percentile: 4.286675071716308
Shutdown handler de-registered
20th percentile: 3.6038615226745607
30th percentile: 4.552417421340943
rirv938-llama-8b-1m-sca_88531_v2 status is now deployed due to DeploymentManager action
30th percentile: 3.8710175514221192
95th percentile: 7.50224814414978
40th percentile: 5.021399927139282
40th percentile: 4.080910491943359
99th percentile: 7.563809080123901
50th percentile: 5.490382432937622
50th percentile: 4.2908034324646
mean time: 6.121190452575684
60th percentile: 5.884016847610473
60th percentile: 4.742373657226563
Pipeline stage StressChecker completed in 87.18s
70th percentile: 6.277651262283325
70th percentile: 5.193943881988525
run pipeline stage %s
80th percentile: 6.604249620437622
80th percentile: 5.933200693130494
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
90th percentile: 6.863811922073364
90th percentile: 6.9601440906524665
run_pipeline:run_in_cloud %s
95th percentile: 6.993593072891235
95th percentile: 7.473615789413452
starting trigger_guanaco_pipeline args=%s
99th percentile: 7.097417993545532
99th percentile: 7.884393148422241
mean time: 5.513564395904541
triggered trigger_guanaco_pipeline args=%s
mean time: 4.88374285697937
Pipeline stage StressChecker completed in 82.43s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 8.93s
Pipeline stage StressChecker completed in 80.63s
run pipeline stage %s
run pipeline stage %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Running pipeline stage TriggerMKMLProfilingPipeline
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
run_pipeline:run_in_cloud %s
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
starting trigger_guanaco_pipeline args=%s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 6.84s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.93s
run pipeline stage %s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 8.02s
Shutdown handler de-registered
Running pipeline stage TriggerMKMLProfilingPipeline
run pipeline stage %s
rirv938-mistral-24b-scal_2131_v5 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.29s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.21s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-scal-2131-v5-profiler
Waiting for inference service rirv938-mistral-24b-scal-2131-v5-profiler to be ready
Inference service rirv938-mistral-24b-scal-2131-v5-profiler ready after 110.4567015171051s
Pipeline stage MKMLProfilerDeployer completed in 111.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplo9mcxg:/code/chaiverse_profiler_1741803504 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplo9mcxg --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741803504 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1741803504/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-scal-2131-v5-profiler is running
Tearing down inference service rirv938-mistral-24b-scal-2131-v5-profiler
Service rirv938-mistral-24b-scal-2131-v5-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.42s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-scal-2131-v5-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.41s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.20s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-scal-2131-v5-profiler
Waiting for inference service rirv938-mistral-24b-scal-2131-v5-profiler to be ready
Inference service rirv938-mistral-24b-scal-2131-v5-profiler ready after 100.45462918281555s
Pipeline stage MKMLProfilerDeployer completed in 101.02s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplowpzt6:/code/chaiverse_profiler_1741807112 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplowpzt6 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741807112 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1741807112/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-scal-2131-v5-profiler is running
Tearing down inference service rirv938-mistral-24b-scal-2131-v5-profiler
Service rirv938-mistral-24b-scal-2131-v5-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.39s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-scal-2131-v5-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.52s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-mistral-24b-scal-2131-v5-profiler
Waiting for inference service rirv938-mistral-24b-scal-2131-v5-profiler to be ready
Inference service rirv938-mistral-24b-scal-2131-v5-profiler ready after 110.43759608268738s
Pipeline stage MKMLProfilerDeployer completed in 111.01s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplowvpgp:/code/chaiverse_profiler_1741810747 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-mistral-24b-af9a0551b1997f0bd9d3214876581fbc-deplowvpgp --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1741810747 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1741810747/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-mistral-24b-scal-2131-v5-profiler is running
Tearing down inference service rirv938-mistral-24b-scal-2131-v5-profiler
Service rirv938-mistral-24b-scal-2131-v5-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.50s
Shutdown handler de-registered
rirv938-mistral-24b-scal_2131_v5 status is now inactive due to auto deactivation removed underperforming models
rirv938-mistral-24b-scal_2131_v5 status is now torndown due to DeploymentManager action