Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name lanlanlan23-mv-total-pk-71370-v1-mkmlizer
Waiting for job on lanlanlan23-mv-total-pk-71370-v1-mkmlizer to finish
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ https://mk1.ai ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ The license key for the current software has been verified as ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ belonging to: ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Chai Research Corp. ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ║ ║
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: Downloaded to shared memory in 47.654s
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpp9bf6i7r, device:0
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: quantized model in 31.170s
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: Processed model lanlanlan23/mV_total_pk_100r_filter50_2700truncated_step5000 in 78.825s
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: creating bucket guanaco-mkml-models
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1/config.json
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1/special_tokens_map.json
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1/tokenizer_config.json
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1/tokenizer.json
lanlanlan23-mv-total-pk-71370-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/lanlanlan23-mv-total-pk-71370-v1/flywheel_model.0.safetensors
lanlanlan23-mv-total-pk-71370-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:11, 31.68it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:07, 48.09it/s]
Loading 0: 5%|▍ | 18/363 [00:00<00:07, 48.98it/s]
Loading 0: 7%|▋ | 24/363 [00:00<00:08, 40.70it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 46.82it/s]
Loading 0: 10%|█ | 37/363 [00:00<00:07, 44.31it/s]
Loading 0: 12%|█▏ | 42/363 [00:00<00:07, 43.64it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:06, 48.64it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:06, 44.87it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:08, 34.21it/s]
Loading 0: 18%|█▊ | 65/363 [00:01<00:08, 33.37it/s]
Loading 0: 20%|█▉ | 72/363 [00:01<00:07, 39.45it/s]
Loading 0: 21%|██▏ | 78/363 [00:01<00:07, 38.59it/s]
Loading 0: 23%|██▎ | 83/363 [00:02<00:07, 38.33it/s]
Loading 0: 25%|██▍ | 89/363 [00:02<00:06, 42.97it/s]
Loading 0: 26%|██▌ | 94/363 [00:02<00:06, 42.75it/s]
Loading 0: 27%|██▋ | 99/363 [00:02<00:06, 43.42it/s]
Loading 0: 29%|██▊ | 104/363 [00:02<00:05, 45.05it/s]
Loading 0: 30%|███ | 110/363 [00:02<00:05, 42.71it/s]
Loading 0: 32%|███▏ | 115/363 [00:02<00:05, 41.36it/s]
Loading 0: 33%|███▎ | 120/363 [00:02<00:06, 39.45it/s]
Loading 0: 34%|███▍ | 125/363 [00:03<00:05, 41.54it/s]
Loading 0: 36%|███▌ | 130/363 [00:03<00:05, 42.11it/s]
Loading 0: 37%|███▋ | 135/363 [00:03<00:05, 41.70it/s]
Loading 0: 39%|███▊ | 140/363 [00:03<00:05, 43.48it/s]
Loading 0: 40%|███▉ | 145/363 [00:03<00:07, 27.39it/s]
Loading 0: 41%|████ | 149/363 [00:03<00:07, 27.75it/s]
Loading 0: 43%|████▎ | 156/363 [00:03<00:05, 35.54it/s]
Loading 0: 44%|████▍ | 161/363 [00:04<00:05, 37.03it/s]
Loading 0: 46%|████▌ | 166/363 [00:04<00:05, 38.30it/s]
Loading 0: 47%|████▋ | 171/363 [00:04<00:04, 40.32it/s]
Loading 0: 48%|████▊ | 176/363 [00:04<00:05, 33.94it/s]
Loading 0: 50%|█████ | 183/363 [00:04<00:04, 40.74it/s]
Loading 0: 52%|█████▏ | 188/363 [00:04<00:04, 40.47it/s]
Loading 0: 53%|█████▎ | 193/363 [00:04<00:04, 40.13it/s]
Loading 0: 55%|█████▍ | 198/363 [00:04<00:03, 41.33it/s]
Loading 0: 56%|█████▌ | 203/363 [00:05<00:04, 35.57it/s]
Loading 0: 58%|█████▊ | 210/363 [00:05<00:03, 42.22it/s]
Loading 0: 59%|█████▉ | 215/363 [00:05<00:03, 41.69it/s]
Loading 0: 61%|██████ | 220/363 [00:05<00:03, 43.37it/s]
Loading 0: 62%|██████▏ | 225/363 [00:05<00:05, 27.13it/s]
Loading 0: 63%|██████▎ | 230/363 [00:05<00:04, 29.35it/s]
Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 36.58it/s]
Loading 0: 67%|██████▋ | 242/363 [00:06<00:03, 38.12it/s]
Loading 0: 68%|██████▊ | 247/363 [00:06<00:02, 39.23it/s]
Loading 0: 69%|██████▉ | 252/363 [00:06<00:02, 41.49it/s]
Loading 0: 71%|███████ | 257/363 [00:06<00:02, 35.75it/s]
Loading 0: 73%|███████▎ | 264/363 [00:06<00:02, 42.57it/s]
Loading 0: 74%|███████▍ | 269/363 [00:06<00:02, 42.72it/s]
Loading 0: 75%|███████▌ | 274/363 [00:06<00:02, 42.85it/s]
Loading 0: 77%|███████▋ | 279/363 [00:07<00:01, 44.22it/s]
Loading 0: 78%|███████▊ | 284/363 [00:07<00:02, 36.05it/s]
Loading 0: 80%|████████ | 291/363 [00:07<00:01, 42.58it/s]
Loading 0: 82%|████████▏ | 296/363 [00:07<00:01, 41.93it/s]
Loading 0: 83%|████████▎ | 301/363 [00:07<00:01, 43.05it/s]
Loading 0: 84%|████████▍ | 306/363 [00:08<00:02, 23.83it/s]
Loading 0: 85%|████████▌ | 310/363 [00:08<00:02, 24.54it/s]
Loading 0: 87%|████████▋ | 314/363 [00:08<00:01, 27.09it/s]
Loading 0: 88%|████████▊ | 320/363 [00:08<00:01, 32.71it/s]
Loading 0: 90%|████████▉ | 326/363 [00:08<00:01, 34.27it/s]
Loading 0: 91%|█████████ | 330/363 [00:08<00:00, 33.75it/s]
Loading 0: 93%|█████████▎| 338/363 [00:08<00:00, 42.40it/s]
Loading 0: 95%|█████████▍| 344/363 [00:08<00:00, 40.38it/s]
Loading 0: 96%|█████████▌| 349/363 [00:09<00:00, 39.78it/s]
Loading 0: 98%|█████████▊| 355/363 [00:09<00:00, 44.10it/s]
Loading 0: 99%|█████████▉| 360/363 [00:09<00:00, 43.61it/s]
Job lanlanlan23-mv-total-pk-71370-v1-mkmlizer completed after 104.66s with status: succeeded
Stopping job with name lanlanlan23-mv-total-pk-71370-v1-mkmlizer
Pipeline stage MKMLizer completed in 105.18s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service lanlanlan23-mv-total-pk-71370-v1
Waiting for inference service lanlanlan23-mv-total-pk-71370-v1 to be ready
Inference service lanlanlan23-mv-total-pk-71370-v1 ready after 110.48913717269897s
Pipeline stage MKMLDeployer completed in 111.06s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.7777347564697266s
Received healthy response to inference request in 1.5543947219848633s
Received healthy response to inference request in 1.7195289134979248s
Received healthy response to inference request in 1.6463608741760254s
Received healthy response to inference request in 1.6385109424591064s
5 requests
0 failed requests
5th percentile: 1.5712179660797119
10th percentile: 1.5880412101745605
20th percentile: 1.6216876983642579
30th percentile: 1.6400809288024902
40th percentile: 1.6432209014892578
50th percentile: 1.6463608741760254
60th percentile: 1.675628089904785
70th percentile: 1.704895305633545
80th percentile: 1.9311700820922852
90th percentile: 2.354452419281006
95th percentile: 2.566093587875366
99th percentile: 2.7354065227508544
mean time: 1.8673060417175293
Pipeline stage StressChecker completed in 10.74s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.75s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.72s
Shutdown handler de-registered
lanlanlan23-mv-total-pk_71370_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service lanlanlan23-mv-total-pk-71370-v1-profiler
Waiting for inference service lanlanlan23-mv-total-pk-71370-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2648.93s
Shutdown handler de-registered
lanlanlan23-mv-total-pk_71370_v1 status is now inactive due to auto deactivation removed underperforming models
lanlanlan23-mv-total-pk_71370_v1 status is now torndown due to DeploymentManager action