Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer
Waiting for job on riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer to finish
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ https://mk1.ai ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ The license key for the current software has been verified as ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ belonging to: ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Chai Research Corp. ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ║ ║
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission riverise-3k-1e-4-sft-cap_v1: HTTPConnectionPool(host='riverise-3k-1e-4-sft-cap-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission riverise-3k-1e-4-sft-cap_v1: HTTPConnectionPool(host='riverise-3k-1e-4-sft-cap-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission riverise-3k-1e-4-sft-cap_v1: HTTPConnectionPool(host='riverise-3k-1e-4-sft-cap-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: Downloaded to shared memory in 45.332s
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpmyy41cax, device:0
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission riverise-3k-1e-4-sft-cap_v1: HTTPConnectionPool(host='riverise-3k-1e-4-sft-cap-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: quantized model in 31.280s
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: Processed model Riverise/7k_1e-4_2epk_sft_Cap in 76.612s
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: creating bucket guanaco-mkml-models
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1/config.json
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1/special_tokens_map.json
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1/tokenizer_config.json
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1/tokenizer.json
riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/riverise-7k-1e-4-2epk-sft-cap-v1/flywheel_model.0.safetensors
Job riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer completed after 105.26s with status: succeeded
Stopping job with name riverise-7k-1e-4-2epk-sft-cap-v1-mkmlizer
Pipeline stage MKMLizer completed in 105.76s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-7k-1e-4-2epk-sft-cap-v1
Waiting for inference service riverise-7k-1e-4-2epk-sft-cap-v1 to be ready
Failed to get response for submission riverise-3k-1e-4-sft-cap_v1: HTTPConnectionPool(host='riverise-3k-1e-4-sft-cap-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service riverise-7k-1e-4-2epk-sft-cap-v1 ready after 120.49916791915894s
Pipeline stage MKMLDeployer completed in 121.07s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.389143228530884s
Received healthy response to inference request in 1.5285329818725586s
Received healthy response to inference request in 1.8516900539398193s
Received healthy response to inference request in 1.6480228900909424s
Received healthy response to inference request in 1.9738502502441406s
5 requests
0 failed requests
5th percentile: 1.5524309635162354
10th percentile: 1.576328945159912
20th percentile: 1.6241249084472655
30th percentile: 1.6887563228607179
40th percentile: 1.7702231884002686
50th percentile: 1.8516900539398193
60th percentile: 1.9005541324615478
70th percentile: 1.9494182109832763
80th percentile: 2.0569088459014893
90th percentile: 2.2230260372161865
95th percentile: 2.306084632873535
99th percentile: 2.372531509399414
mean time: 1.8782478809356689
Pipeline stage StressChecker completed in 10.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.65s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.65s
Shutdown handler de-registered
riverise-7k-1e-4-2epk-sft-cap_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 4958.85s
Shutdown handler de-registered
riverise-7k-1e-4-2epk-sft-cap_v1 status is now inactive due to auto deactivation removed underperforming models
riverise-7k-1e-4-2epk-sft-cap_v1 status is now torndown due to DeploymentManager action