Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name junhua024-chai-1-full-066126-v40-mkmlizer
Waiting for job on junhua024-chai-1-full-066126-v40-mkmlizer to finish
junhua024-chai-1-full-066126-v40-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Version: 0.29.15 ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ https://mk1.ai ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ The license key for the current software has been verified as ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ belonging to: ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Chai Research Corp. ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
junhua024-chai-1-full-066126-v40-mkmlizer: ║ ║
junhua024-chai-1-full-066126-v40-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: `pip install huggingface_hub[hf_xet]` or `pip install hf_xet`
junhua024-chai-1-full-066126-v40-mkmlizer: Downloaded to shared memory in 73.810s
junhua024-chai-1-full-066126-v40-mkmlizer: Checking if junhua024/chai-1-full-066126 already exists in ChaiML
junhua024-chai-1-full-066126-v40-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqoo0x556, device:0
junhua024-chai-1-full-066126-v40-mkmlizer: Saving flywheel model at /dev/shm/model_cache
junhua024-chai-1-full-066126-v40-mkmlizer: quantized model in 31.539s
junhua024-chai-1-full-066126-v40-mkmlizer: Processed model junhua024/chai-1-full-066126 in 105.437s
junhua024-chai-1-full-066126-v40-mkmlizer: creating bucket guanaco-mkml-models
junhua024-chai-1-full-066126-v40-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
junhua024-chai-1-full-066126-v40-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia
junhua024-chai-1-full-066126-v40-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia/special_tokens_map.json
junhua024-chai-1-full-066126-v40-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia/config.json
junhua024-chai-1-full-066126-v40-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia/tokenizer_config.json
junhua024-chai-1-full-066126-v40-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia/tokenizer.json
junhua024-chai-1-full-066126-v40-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/junhua024-chai-1-full-066126-v40/nvidia/flywheel_model.0.safetensors
junhua024-chai-1-full-066126-v40-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:00<00:23, 15.18it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:19, 17.97it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:12, 28.98it/s]
Loading 0: 5%|▍ | 17/363 [00:00<00:11, 29.91it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:10, 32.63it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 43.80it/s]
Loading 0: 10%|▉ | 36/363 [00:01<00:10, 31.43it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:09, 32.65it/s]
Loading 0: 14%|█▍ | 50/363 [00:01<00:07, 40.32it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:09, 34.22it/s]
Loading 0: 16%|█▋ | 59/363 [00:01<00:08, 33.78it/s]
Loading 0: 18%|█▊ | 65/363 [00:01<00:08, 34.21it/s]
Loading 0: 19%|█▉ | 69/363 [00:02<00:08, 32.88it/s]
Loading 0: 20%|██ | 74/363 [00:02<00:08, 35.76it/s]
Loading 0: 21%|██▏ | 78/363 [00:02<00:08, 34.79it/s]
Loading 0: 23%|██▎ | 82/363 [00:02<00:09, 30.89it/s]
Loading 0: 25%|██▍ | 89/363 [00:02<00:07, 37.71it/s]
Loading 0: 26%|██▌ | 93/363 [00:02<00:07, 36.69it/s]
Loading 0: 27%|██▋ | 97/363 [00:02<00:08, 33.17it/s]
Loading 0: 28%|██▊ | 101/363 [00:03<00:07, 33.80it/s]
Loading 0: 29%|██▉ | 105/363 [00:03<00:08, 30.94it/s]
Loading 0: 31%|███ | 112/363 [00:03<00:06, 40.10it/s]
Loading 0: 32%|███▏ | 117/363 [00:03<00:08, 29.60it/s]
Loading 0: 34%|███▎ | 122/363 [00:03<00:07, 31.64it/s]
Loading 0: 35%|███▌ | 128/363 [00:03<00:07, 33.12it/s]
Loading 0: 36%|███▋ | 132/363 [00:03<00:07, 32.73it/s]
Loading 0: 38%|███▊ | 137/363 [00:04<00:06, 36.39it/s]
Loading 0: 39%|███▉ | 141/363 [00:04<00:06, 36.86it/s]
Loading 0: 40%|███▉ | 145/363 [00:04<00:06, 32.12it/s]
Loading 0: 41%|████ | 149/363 [00:04<00:06, 31.41it/s]
Loading 0: 43%|████▎ | 155/363 [00:04<00:05, 37.39it/s]
Loading 0: 44%|████▍ | 160/363 [00:04<00:05, 35.10it/s]
Loading 0: 45%|████▌ | 164/363 [00:04<00:05, 34.95it/s]
Loading 0: 46%|████▋ | 168/363 [00:05<00:06, 31.92it/s]
Loading 0: 48%|████▊ | 176/363 [00:05<00:04, 38.67it/s]
Loading 0: 50%|████▉ | 180/363 [00:05<00:05, 31.58it/s]
Loading 0: 51%|█████ | 185/363 [00:05<00:05, 33.53it/s]
Loading 0: 53%|█████▎ | 191/363 [00:05<00:04, 35.19it/s]
Loading 0: 54%|█████▎ | 195/363 [00:05<00:04, 33.80it/s]
Loading 0: 55%|█████▌ | 200/363 [00:05<00:04, 37.25it/s]
Loading 0: 56%|█████▋ | 205/363 [00:06<00:04, 35.25it/s]
Loading 0: 58%|█████▊ | 209/363 [00:06<00:04, 36.27it/s]
Loading 0: 59%|█████▊ | 213/363 [00:06<00:04, 33.94it/s]
Loading 0: 60%|██████ | 219/363 [00:06<00:03, 40.13it/s]
Loading 0: 62%|██████▏ | 224/363 [00:06<00:03, 35.80it/s]
Loading 0: 63%|██████▎ | 228/363 [00:06<00:03, 35.43it/s]
Loading 0: 64%|██████▍ | 232/363 [00:06<00:03, 34.64it/s]
Loading 0: 66%|██████▌ | 239/363 [00:06<00:03, 37.19it/s]
Loading 0: 67%|██████▋ | 243/363 [00:07<00:03, 30.04it/s]
Loading 0: 68%|██████▊ | 248/363 [00:07<00:03, 32.67it/s]
Loading 0: 70%|██████▉ | 254/363 [00:07<00:03, 33.53it/s]
Loading 0: 71%|███████ | 258/363 [00:07<00:03, 33.27it/s]
Loading 0: 73%|███████▎ | 264/363 [00:07<00:02, 33.34it/s]
Loading 0: 74%|███████▍ | 269/363 [00:07<00:02, 33.15it/s]
Loading 0: 76%|███████▌ | 275/363 [00:08<00:02, 34.55it/s]
Loading 0: 78%|███████▊ | 283/363 [00:08<00:01, 43.89it/s]
Loading 0: 79%|███████▉ | 288/363 [00:08<00:02, 33.35it/s]
Loading 0: 81%|████████ | 293/363 [00:08<00:02, 33.96it/s]
Loading 0: 83%|████████▎ | 302/363 [00:08<00:01, 40.76it/s]
Loading 0: 85%|████████▍ | 307/363 [00:08<00:01, 35.03it/s]
Loading 0: 86%|████████▌ | 311/363 [00:09<00:01, 34.14it/s]
Loading 0: 87%|████████▋ | 317/363 [00:09<00:01, 35.28it/s]
Loading 0: 88%|████████▊ | 321/363 [00:09<00:01, 34.39it/s]
Loading 0: 90%|█████████ | 327/363 [00:09<00:01, 34.16it/s]
Loading 0: 91%|█████████▏| 332/363 [00:09<00:00, 33.43it/s]
Loading 0: 93%|█████████▎| 338/363 [00:09<00:00, 34.54it/s]
Loading 0: 95%|█████████▍| 344/363 [00:09<00:00, 38.98it/s]
Loading 0: 96%|█████████▌| 349/363 [00:10<00:00, 26.99it/s]
Loading 0: 97%|█████████▋| 353/363 [00:10<00:00, 24.91it/s]
Loading 0: 98%|█████████▊| 357/363 [00:10<00:00, 26.51it/s]
Job junhua024-chai-1-full-066126-v40-mkmlizer completed after 128.35s with status: succeeded
Stopping job with name junhua024-chai-1-full-066126-v40-mkmlizer
Pipeline stage MKMLizer completed in 128.93s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service junhua024-chai-1-full-066126-v40
Waiting for inference service junhua024-chai-1-full-066126-v40 to be ready
Failed to get response for submission junhua024-chai-1-full-066126_v39: HTTPConnectionPool(host='junhua024-chai-1-full-066126-v39-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission junhua024-chai-1-full-066126_v39: HTTPConnectionPool(host='junhua024-chai-1-full-066126-v39-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service junhua024-chai-1-full-066126-v40 ready after 231.12000107765198s
Pipeline stage MKMLDeployer completed in 231.74s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.4704160690307617s
Received healthy response to inference request in 1.797492265701294s
Failed to get response for submission albertwang8192-2025-07-14-0_v1: HTTPConnectionPool(host='albertwang8192-2025-07-14-0-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 1.8034822940826416s
Received healthy response to inference request in 1.5306355953216553s
Received healthy response to inference request in 1.7413811683654785s
5 requests
0 failed requests
5th percentile: 1.57278470993042
10th percentile: 1.6149338245391847
20th percentile: 1.6992320537567138
30th percentile: 1.7526033878326417
40th percentile: 1.7750478267669678
50th percentile: 1.797492265701294
60th percentile: 1.799888277053833
70th percentile: 1.8022842884063721
80th percentile: 1.9368690490722658
90th percentile: 2.203642559051514
95th percentile: 2.337029314041138
99th percentile: 2.443738718032837
mean time: 1.8686814785003663
Pipeline stage StressChecker completed in 11.43s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.90s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.75s
Shutdown handler de-registered
junhua024-chai-1-full-066126_v40 status is now deployed due to DeploymentManager action
Failed to get response for submission albertwang8192-2025-07-14-0_v1: HTTPConnectionPool(host='albertwang8192-2025-07-14-0-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 8113.20s
Shutdown handler de-registered
junhua024-chai-1-full-066126_v40 status is now inactive due to auto deactivation removed underperforming models
junhua024-chai-1-full-066126_v40 status is now torndown due to DeploymentManager action