Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-humof-95870-v1-mkmlizer
Waiting for job on alexdaoud-trainer-humof-95870-v1-mkmlizer to finish
alexdaoud-trainer-humof-95870-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║     _____            __           __                                ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║       /___/                                                         ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  Version: 0.11.37                                                   ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  https://mk1.ai                                                     ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  belonging to:                                                      ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  Chai Research Corp.                                                ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ║                                                                     ║
alexdaoud-trainer-humof-95870-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-humof-95870-v1-mkmlizer: quantized model in 15.433s
alexdaoud-trainer-humof-95870-v1-mkmlizer: Processed model alexdaoud/trainer_humof_2025-01-27-checkpoint-47 in 48.024s
alexdaoud-trainer-humof-95870-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-humof-95870-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-humof-95870-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1
alexdaoud-trainer-humof-95870-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1/config.json
alexdaoud-trainer-humof-95870-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1/special_tokens_map.json
alexdaoud-trainer-humof-95870-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1/tokenizer_config.json
alexdaoud-trainer-humof-95870-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1/tokenizer.json
alexdaoud-trainer-humof-95870-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-humof-95870-v1/flywheel_model.0.safetensors
alexdaoud-trainer-humof-95870-v1-mkmlizer: 
Loading 0:   0%|          | 0/292 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/292 [00:00<00:07, 36.57it/s]
Loading 0:   4%|▍         | 13/292 [00:00<00:04, 58.29it/s]
Loading 0:   7%|▋         | 20/292 [00:00<00:04, 55.74it/s]
Loading 0:   9%|▉         | 26/292 [00:00<00:04, 53.82it/s]
Loading 0:  11%|█         | 32/292 [00:00<00:05, 47.28it/s]
Loading 0:  14%|█▎        | 40/292 [00:00<00:04, 56.09it/s]
Loading 0:  16%|█▌        | 46/292 [00:00<00:04, 52.66it/s]
Loading 0:  18%|█▊        | 52/292 [00:00<00:04, 51.75it/s]
Loading 0:  20%|██        | 59/292 [00:01<00:04, 48.29it/s]
Loading 0:  23%|██▎       | 67/292 [00:01<00:04, 54.38it/s]
Loading 0:  25%|██▌       | 73/292 [00:01<00:04, 51.77it/s]
Loading 0:  27%|██▋       | 79/292 [00:01<00:04, 52.38it/s]
Loading 0:  29%|██▉       | 85/292 [00:01<00:05, 37.56it/s]
Loading 0:  31%|███       | 90/292 [00:01<00:05, 39.85it/s]
Loading 0:  33%|███▎      | 95/292 [00:02<00:05, 37.33it/s]
Loading 0:  35%|███▌      | 103/292 [00:02<00:04, 46.69it/s]
Loading 0:  37%|███▋      | 109/292 [00:02<00:03, 46.81it/s]
Loading 0:  39%|███▉      | 115/292 [00:02<00:03, 49.42it/s]
Loading 0:  41%|████▏     | 121/292 [00:02<00:03, 50.29it/s]
Loading 0:  43%|████▎     | 127/292 [00:02<00:03, 49.10it/s]
Loading 0:  46%|████▌     | 133/292 [00:02<00:03, 51.29it/s]
Loading 0:  48%|████▊     | 140/292 [00:02<00:03, 48.03it/s]
Loading 0:  51%|█████     | 148/292 [00:03<00:02, 55.10it/s]
Loading 0:  53%|█████▎    | 154/292 [00:03<00:02, 52.66it/s]
Loading 0:  55%|█████▍    | 160/292 [00:03<00:02, 53.45it/s]
Loading 0:  57%|█████▋    | 166/292 [00:03<00:02, 54.48it/s]
Loading 0:  59%|█████▉    | 172/292 [00:03<00:02, 51.71it/s]
Loading 0:  61%|██████▏   | 179/292 [00:03<00:02, 56.05it/s]
Loading 0:  64%|██████▎   | 186/292 [00:03<00:02, 52.25it/s]
Loading 0:  66%|██████▌   | 192/292 [00:04<00:02, 35.73it/s]
Loading 0:  68%|██████▊   | 199/292 [00:04<00:02, 38.95it/s]
Loading 0:  70%|██████▉   | 204/292 [00:04<00:02, 40.49it/s]
Loading 0:  72%|███████▏  | 211/292 [00:04<00:01, 46.40it/s]
Loading 0:  74%|███████▍  | 217/292 [00:04<00:01, 46.41it/s]
Loading 0:  76%|███████▋  | 223/292 [00:04<00:01, 49.19it/s]
Loading 0:  79%|███████▉  | 230/292 [00:04<00:01, 46.46it/s]
Loading 0:  82%|████████▏ | 238/292 [00:04<00:01, 53.00it/s]
Loading 0:  84%|████████▎ | 244/292 [00:05<00:00, 49.47it/s]
Loading 0:  86%|████████▌ | 250/292 [00:05<00:00, 51.39it/s]
Loading 0:  88%|████████▊ | 257/292 [00:05<00:00, 49.10it/s]
Loading 0:  91%|█████████ | 265/292 [00:05<00:00, 55.72it/s]
Loading 0:  93%|█████████▎| 271/292 [00:05<00:00, 52.58it/s]
Loading 0:  95%|█████████▍| 277/292 [00:05<00:00, 53.90it/s]
Loading 0:  97%|█████████▋| 283/292 [00:05<00:00, 49.04it/s]
Loading 0:  99%|█████████▉| 289/292 [00:06<00:00, 34.71it/s]
                                                            
Job alexdaoud-trainer-humof-95870-v1-mkmlizer completed after 74.17s with status: succeeded
Stopping job with name alexdaoud-trainer-humof-95870-v1-mkmlizer
Pipeline stage MKMLizer completed in 74.63s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-humof-95870-v1
Waiting for inference service alexdaoud-trainer-humof-95870-v1 to be ready
Failed to get response for submission rirv938-anthropic-40k-2_43413_v7: HTTPConnectionPool(host='rirv938-anthropic-40k-2-43413-v7-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service alexdaoud-trainer-humof-95870-v1 ready after 200.91637992858887s
Pipeline stage MKMLDeployer completed in 201.44s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.879218339920044s
Received healthy response to inference request in 4.855034112930298s
Received healthy response to inference request in 2.4891388416290283s
Received healthy response to inference request in 4.221195459365845s
Received healthy response to inference request in 2.410215377807617s
5 requests
0 failed requests
5th percentile: 2.4260000705718996
10th percentile: 2.4417847633361816
20th percentile: 2.473354148864746
30th percentile: 2.7671547412872313
40th percentile: 3.323186540603638
50th percentile: 3.879218339920044
60th percentile: 4.016009187698364
70th percentile: 4.152800035476685
80th percentile: 4.347963190078736
90th percentile: 4.601498651504516
95th percentile: 4.728266382217407
99th percentile: 4.82968056678772
mean time: 3.5709604263305663
%s, retrying in %s seconds...
Received healthy response to inference request in 2.8589751720428467s
Received healthy response to inference request in 3.9847381114959717s
Received healthy response to inference request in 3.453721046447754s
Received healthy response to inference request in 3.209476947784424s
Received healthy response to inference request in 4.699348449707031s
5 requests
0 failed requests
5th percentile: 2.929075527191162
10th percentile: 2.9991758823394776
20th percentile: 3.1393765926361086
30th percentile: 3.25832576751709
40th percentile: 3.356023406982422
50th percentile: 3.453721046447754
60th percentile: 3.666127872467041
70th percentile: 3.878534698486328
80th percentile: 4.127660179138184
90th percentile: 4.413504314422608
95th percentile: 4.556426382064819
99th percentile: 4.670764036178589
mean time: 3.6412519454956054
%s, retrying in %s seconds...
Received healthy response to inference request in 2.373394012451172s
Received healthy response to inference request in 2.467503309249878s
Received healthy response to inference request in 2.3339250087738037s
Received healthy response to inference request in 2.9018993377685547s
Received healthy response to inference request in 4.054178953170776s
5 requests
0 failed requests
5th percentile: 2.3418188095092773
10th percentile: 2.349712610244751
20th percentile: 2.3655002117156982
30th percentile: 2.392215871810913
40th percentile: 2.4298595905303957
50th percentile: 2.467503309249878
60th percentile: 2.6412617206573485
70th percentile: 2.815020132064819
80th percentile: 3.1323552608489993
90th percentile: 3.5932671070098876
95th percentile: 3.8237230300903318
99th percentile: 4.008087768554687
mean time: 2.826180124282837
Pipeline stage StressChecker completed in 53.59s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.64s
Shutdown handler de-registered
alexdaoud-trainer-humof_95870_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-humof-95870-v1-profiler
Waiting for inference service alexdaoud-trainer-humof-95870-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3493.36s
Shutdown handler de-registered
alexdaoud-trainer-humof_95870_v1 status is now inactive due to auto deactivation removed underperforming models
alexdaoud-trainer-humof_95870_v1 status is now torndown due to DeploymentManager action