Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name junhua024-chai-1-full-94469-v112-mkmlizer
Waiting for job on junhua024-chai-1-full-94469-v112-mkmlizer to finish
junhua024-chai-1-full-94469-v112-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Version: 0.29.15 ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ https://mk1.ai ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ The license key for the current software has been verified as ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ belonging to: ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Chai Research Corp. ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
junhua024-chai-1-full-94469-v112-mkmlizer: ║ ║
junhua024-chai-1-full-94469-v112-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-94469-v112-mkmlizer: Downloaded to shared memory in 77.443s
junhua024-chai-1-full-94469-v112-mkmlizer: Checking if junhua024/chai-1-full-066126 already exists in ChaiML
junhua024-chai-1-full-94469-v112-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpgwu1l6w7, device:0
junhua024-chai-1-full-94469-v112-mkmlizer: Saving flywheel model at /dev/shm/model_cache
junhua024-chai-1-full-94469-v112-mkmlizer: quantized model in 31.286s
junhua024-chai-1-full-94469-v112-mkmlizer: Processed model junhua024/chai-1-full-066126 in 108.814s
junhua024-chai-1-full-94469-v112-mkmlizer: creating bucket guanaco-mkml-models
junhua024-chai-1-full-94469-v112-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/junhua024-chai-1-full-94469-v112/nvidia/config.json
junhua024-chai-1-full-94469-v112-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/junhua024-chai-1-full-94469-v112/nvidia/special_tokens_map.json
junhua024-chai-1-full-94469-v112-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/junhua024-chai-1-full-94469-v112/nvidia/tokenizer_config.json
junhua024-chai-1-full-94469-v112-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/junhua024-chai-1-full-94469-v112/nvidia/tokenizer.json
junhua024-chai-1-full-94469-v112-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/junhua024-chai-1-full-94469-v112/nvidia/flywheel_model.0.safetensors
junhua024-chai-1-full-94469-v112-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:00<00:22, 15.83it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:19, 18.13it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:11, 29.36it/s]
Loading 0: 5%|▍ | 17/363 [00:00<00:11, 30.70it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:10, 33.23it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 44.33it/s]
Loading 0: 10%|▉ | 36/363 [00:01<00:09, 32.72it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:09, 33.14it/s]
Loading 0: 14%|█▍ | 50/363 [00:01<00:07, 39.99it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:08, 34.53it/s]
Loading 0: 16%|█▋ | 59/363 [00:01<00:08, 33.98it/s]
Loading 0: 18%|█▊ | 65/363 [00:01<00:08, 35.04it/s]
Loading 0: 19%|█▉ | 69/363 [00:02<00:08, 34.27it/s]
Loading 0: 20%|██ | 74/363 [00:02<00:07, 37.75it/s]
Loading 0: 21%|██▏ | 78/363 [00:02<00:07, 37.22it/s]
Loading 0: 23%|██▎ | 82/363 [00:02<00:08, 32.05it/s]
Loading 0: 24%|██▎ | 86/363 [00:02<00:08, 33.84it/s]
Loading 0: 25%|██▌ | 91/363 [00:02<00:07, 34.82it/s]
Loading 0: 27%|██▋ | 97/363 [00:02<00:07, 36.20it/s]
Loading 0: 28%|██▊ | 101/363 [00:02<00:07, 35.97it/s]
Loading 0: 29%|██▉ | 105/363 [00:03<00:07, 33.62it/s]
Loading 0: 31%|███ | 113/363 [00:03<00:06, 39.65it/s]
Loading 0: 32%|███▏ | 117/363 [00:03<00:07, 30.92it/s]
Loading 0: 34%|███▎ | 122/363 [00:03<00:07, 32.52it/s]
Loading 0: 35%|███▌ | 128/363 [00:03<00:06, 34.18it/s]
Loading 0: 36%|███▋ | 132/363 [00:03<00:06, 33.41it/s]
Loading 0: 38%|███▊ | 137/363 [00:03<00:06, 36.79it/s]
Loading 0: 39%|███▉ | 141/363 [00:04<00:05, 37.47it/s]
Loading 0: 40%|███▉ | 145/363 [00:04<00:06, 32.80it/s]
Loading 0: 41%|████ | 149/363 [00:04<00:06, 31.85it/s]
Loading 0: 43%|████▎ | 155/363 [00:04<00:05, 38.00it/s]
Loading 0: 44%|████▍ | 160/363 [00:04<00:05, 35.71it/s]
Loading 0: 45%|████▌ | 164/363 [00:04<00:05, 35.33it/s]
Loading 0: 46%|████▋ | 168/363 [00:04<00:05, 32.55it/s]
Loading 0: 48%|████▊ | 176/363 [00:05<00:04, 38.20it/s]
Loading 0: 50%|████▉ | 180/363 [00:05<00:05, 30.95it/s]
Loading 0: 51%|█████ | 185/363 [00:05<00:05, 32.99it/s]
Loading 0: 53%|█████▎ | 191/363 [00:05<00:05, 33.85it/s]
Loading 0: 54%|█████▎ | 195/363 [00:05<00:05, 33.23it/s]
Loading 0: 55%|█████▌ | 200/363 [00:05<00:04, 36.85it/s]
Loading 0: 56%|█████▌ | 204/363 [00:05<00:04, 36.72it/s]
Loading 0: 57%|█████▋ | 208/363 [00:06<00:05, 30.81it/s]
Loading 0: 58%|█████▊ | 212/363 [00:06<00:04, 30.30it/s]
Loading 0: 60%|██████ | 218/363 [00:06<00:03, 37.11it/s]
Loading 0: 61%|██████▏ | 223/363 [00:06<00:03, 35.60it/s]
Loading 0: 63%|██████▎ | 227/363 [00:06<00:03, 35.30it/s]
Loading 0: 64%|██████▎ | 231/363 [00:06<00:03, 33.08it/s]
Loading 0: 66%|██████▌ | 239/363 [00:06<00:03, 39.31it/s]
Loading 0: 67%|██████▋ | 243/363 [00:07<00:03, 31.11it/s]
Loading 0: 68%|██████▊ | 248/363 [00:07<00:03, 33.29it/s]
Loading 0: 70%|██████▉ | 254/363 [00:07<00:03, 34.60it/s]
Loading 0: 71%|███████ | 258/363 [00:07<00:03, 33.88it/s]
Loading 0: 72%|███████▏ | 263/363 [00:07<00:02, 37.27it/s]
Loading 0: 74%|███████▍ | 268/363 [00:07<00:02, 35.43it/s]
Loading 0: 75%|███████▌ | 273/363 [00:07<00:02, 36.86it/s]
Loading 0: 76%|███████▋ | 277/363 [00:08<00:02, 36.00it/s]
Loading 0: 77%|███████▋ | 281/363 [00:08<00:02, 36.70it/s]
Loading 0: 79%|███████▉ | 286/363 [00:08<00:02, 33.38it/s]
Loading 0: 80%|███████▉ | 290/363 [00:08<00:02, 33.40it/s]
Loading 0: 81%|████████ | 294/363 [00:08<00:02, 31.69it/s]
Loading 0: 83%|████████▎ | 301/363 [00:08<00:01, 40.60it/s]
Loading 0: 84%|████████▍ | 306/363 [00:08<00:01, 29.67it/s]
Loading 0: 86%|████████▌ | 311/363 [00:09<00:01, 31.79it/s]
Loading 0: 87%|████████▋ | 317/363 [00:09<00:01, 33.33it/s]
Loading 0: 88%|████████▊ | 321/363 [00:09<00:01, 32.69it/s]
Loading 0: 90%|█████████ | 327/363 [00:09<00:01, 33.00it/s]
Loading 0: 91%|█████████▏| 332/363 [00:09<00:00, 32.41it/s]
Loading 0: 93%|█████████▎| 338/363 [00:09<00:00, 33.72it/s]
Loading 0: 95%|█████████▍| 344/363 [00:09<00:00, 38.64it/s]
Loading 0: 96%|█████████▌| 349/363 [00:10<00:00, 27.04it/s]
Loading 0: 97%|█████████▋| 353/363 [00:10<00:00, 24.82it/s]
Loading 0: 98%|█████████▊| 357/363 [00:10<00:00, 26.40it/s]
Job junhua024-chai-1-full-94469-v112-mkmlizer completed after 138.8s with status: succeeded
Stopping job with name junhua024-chai-1-full-94469-v112-mkmlizer
Pipeline stage MKMLizer completed in 139.32s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service junhua024-chai-1-full-94469-v112
Waiting for inference service junhua024-chai-1-full-94469-v112 to be ready
Failed to get response for submission junhua024-chai-1-full_94469_v111: HTTPConnectionPool(host='junhua024-chai-1-full-94469-v111-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission junhua024-chai-1-full_94469_v111: HTTPConnectionPool(host='junhua024-chai-1-full-94469-v111-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission junhua024-chai-1-full_94469_v111: HTTPConnectionPool(host='junhua024-chai-1-full-94469-v111-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service junhua024-chai-1-full-94469-v112 ready after 261.5221199989319s
Pipeline stage MKMLDeployer completed in 262.41s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"HTTPConnectionPool(host='junhua024-chai-1-full-94469-v112-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
Received healthy response to inference request in 1.880479097366333s
Received healthy response to inference request in 2.244572162628174s
Received healthy response to inference request in 2.332653760910034s
Received healthy response to inference request in 1.6823210716247559s
5 requests
1 failed requests
5th percentile: 1.7219526767730713
10th percentile: 1.7615842819213867
20th percentile: 1.8408474922180176
30th percentile: 1.9532977104187013
40th percentile: 2.0989349365234373
50th percentile: 2.244572162628174
60th percentile: 2.279804801940918
70th percentile: 2.3150374412536623
80th percentile: 4.339199590682985
90th percentile: 8.352291250228884
95th percentile: 10.35883708000183
99th percentile: 11.96407374382019
mean time: 4.101081800460816
%s, retrying in %s seconds...
Received healthy response to inference request in 1.7158119678497314s
Received healthy response to inference request in 2.022481679916382s
Received healthy response to inference request in 1.7333886623382568s
Received healthy response to inference request in 2.042142391204834s
Received healthy response to inference request in 1.5972607135772705s
5 requests
0 failed requests
5th percentile: 1.6209709644317627
10th percentile: 1.6446812152862549
20th percentile: 1.6921017169952393
30th percentile: 1.7193273067474366
40th percentile: 1.7263579845428467
50th percentile: 1.7333886623382568
60th percentile: 1.8490258693695067
70th percentile: 1.9646630764007569
80th percentile: 2.026413822174072
90th percentile: 2.0342781066894533
95th percentile: 2.0382102489471436
99th percentile: 2.041355962753296
mean time: 1.822217082977295
Pipeline stage StressChecker completed in 33.72s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.13s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.77s
Shutdown handler de-registered
junhua024-chai-1-full_94469_v112 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2697.67s
Shutdown handler de-registered
junhua024-chai-1-full_94469_v112 status is now inactive due to auto deactivation removed underperforming models
junhua024-chai-1-full_94469_v112 status is now torndown due to DeploymentManager action
junhua024-chai-1-full_94469_v112 status is now torndown due to DeploymentManager action