Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer
Waiting for job on zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer to finish
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ _____ __ __ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ /___/ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ Version: 0.11.12 ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ https://mk1.ai ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ belonging to: ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ Chai Research Corp. ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ║ ║
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: Downloaded to shared memory in 118.679s
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmplj88j1d4, device:0
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: quantized model in 53.354s
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: Processed model zonemercy/Viral-ss-v0-1e5ep2 in 172.033s
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v78
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v78/config.json
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v78/special_tokens_map.json
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v78/flywheel_model.1.safetensors
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-viral-ss-v0-1e5ep2-v78/flywheel_model.0.safetensors
zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer:
Loading 0: 0%| | 0/507 [00:00<?, ?it/s]
Loading 0: 1%| | 4/507 [00:00<00:16, 29.95it/s]
Loading 0: 1%|▏ | 7/507 [00:00<00:21, 23.64it/s]
Loading 0: 2%|▏ | 10/507 [00:00<00:20, 24.66it/s]
Loading 0: 3%|▎ | 13/507 [00:00<00:21, 23.44it/s]
Loading 0: 3%|▎ | 16/507 [00:00<00:21, 22.89it/s]
Loading 0: 4%|▎ | 19/507 [00:00<00:20, 24.14it/s]
Loading 0: 4%|▍ | 22/507 [00:00<00:18, 25.61it/s]
Loading 0: 5%|▍ | 25/507 [00:01<00:26, 18.41it/s]
Loading 0: 6%|▌ | 28/507 [00:01<00:28, 16.89it/s]
Loading 0: 6%|▌ | 31/507 [00:01<00:25, 18.54it/s]
Loading 0: 7%|▋ | 34/507 [00:01<00:24, 19.43it/s]
Loading 0: 7%|▋ | 37/507 [00:01<00:22, 21.19it/s]
Loading 0: 8%|▊ | 40/507 [00:01<00:21, 21.75it/s]
Loading 0: 8%|▊ | 43/507 [00:02<00:21, 21.72it/s]
Loading 0: 9%|▉ | 46/507 [00:02<00:19, 23.46it/s]
Loading 0: 10%|▉ | 49/507 [00:02<00:18, 24.94it/s]
Loading 0: 10%|█ | 52/507 [00:02<00:23, 18.96it/s]
Loading 0: 11%|█ | 55/507 [00:02<00:25, 17.74it/s]
Loading 0: 11%|█▏ | 58/507 [00:02<00:23, 18.83it/s]
Loading 0: 12%|█▏ | 61/507 [00:02<00:22, 19.63it/s]
Loading 0: 13%|█▎ | 64/507 [00:03<00:21, 20.60it/s]
Loading 0: 13%|█▎ | 67/507 [00:03<00:20, 21.03it/s]
Loading 0: 14%|█▍ | 70/507 [00:03<00:20, 21.19it/s]
Loading 0: 15%|█▍ | 74/507 [00:03<00:19, 22.16it/s]
Loading 0: 15%|█▌ | 78/507 [00:03<00:16, 25.39it/s]
Loading 0: 16%|█▌ | 81/507 [00:03<00:22, 19.05it/s]
Loading 0: 17%|█▋ | 84/507 [00:04<00:21, 19.62it/s]
Loading 0: 17%|█▋ | 87/507 [00:04<00:19, 21.65it/s]
Loading 0: 18%|█▊ | 90/507 [00:04<00:17, 23.48it/s]
Loading 0: 19%|█▊ | 94/507 [00:04<00:17, 23.15it/s]
Loading 0: 19%|█▉ | 97/507 [00:04<00:17, 22.87it/s]
Loading 0: 20%|█▉ | 100/507 [00:04<00:22, 18.05it/s]
Loading 0: 21%|██ | 105/507 [00:04<00:17, 23.33it/s]
Loading 0: 21%|██▏ | 108/507 [00:05<00:24, 16.22it/s]
Loading 0: 22%|██▏ | 112/507 [00:05<00:20, 19.31it/s]
Loading 0: 23%|██▎ | 115/507 [00:05<00:19, 19.76it/s]
Loading 0: 23%|██▎ | 118/507 [00:05<00:18, 21.57it/s]
Loading 0: 24%|██▍ | 121/507 [00:05<00:17, 22.05it/s]
Loading 0: 24%|██▍ | 124/507 [00:05<00:17, 22.38it/s]
Loading 0: 25%|██▌ | 127/507 [00:05<00:15, 23.96it/s]
Loading 0: 26%|██▌ | 130/507 [00:06<00:14, 25.45it/s]
Loading 0: 26%|██▌ | 133/507 [00:06<00:19, 19.54it/s]
Loading 0: 27%|██▋ | 136/507 [00:06<00:20, 18.20it/s]
Loading 0: 27%|██▋ | 139/507 [00:06<00:19, 19.29it/s]
Loading 0: 28%|██▊ | 142/507 [00:06<00:17, 20.35it/s]
Loading 0: 29%|██▉ | 146/507 [00:06<00:14, 24.74it/s]
Loading 0: 29%|██▉ | 149/507 [00:07<00:18, 19.41it/s]
Loading 0: 30%|███ | 154/507 [00:07<00:14, 25.12it/s]
Loading 0: 31%|███ | 157/507 [00:07<00:13, 26.11it/s]
Loading 0: 32%|███▏ | 160/507 [00:07<00:17, 19.48it/s]
Loading 0: 32%|███▏ | 163/507 [00:07<00:19, 17.71it/s]
Loading 0: 33%|███▎ | 166/507 [00:07<00:17, 19.05it/s]
Loading 0: 33%|███▎ | 169/507 [00:08<00:17, 19.29it/s]
Loading 0: 34%|███▍ | 172/507 [00:08<00:15, 21.28it/s]
Loading 0: 35%|███▍ | 175/507 [00:08<00:15, 21.68it/s]
Loading 0: 35%|███▌ | 178/507 [00:08<00:15, 21.27it/s]
Loading 0: 36%|███▌ | 181/507 [00:08<00:14, 23.01it/s]
Loading 0: 36%|███▋ | 184/507 [00:08<00:13, 24.46it/s]
Loading 0: 37%|███▋ | 187/507 [00:08<00:18, 16.88it/s]
Loading 0: 37%|███▋ | 190/507 [00:09<00:19, 16.08it/s]
Loading 0: 38%|███▊ | 193/507 [00:09<00:17, 17.70it/s]
Loading 0: 39%|███▊ | 196/507 [00:09<00:16, 18.57it/s]
Loading 0: 39%|███▉ | 199/507 [00:09<00:14, 20.78it/s]
Loading 0: 40%|███▉ | 202/507 [00:09<00:14, 21.46it/s]
Loading 0: 40%|████ | 205/507 [00:09<00:14, 21.28it/s]
Loading 0: 41%|████ | 208/507 [00:09<00:13, 22.84it/s]
Loading 0: 42%|████▏ | 211/507 [00:10<00:12, 24.59it/s]
Loading 0: 42%|████▏ | 214/507 [00:10<00:15, 18.68it/s]
Loading 0: 43%|████▎ | 217/507 [00:10<00:16, 17.54it/s]
Loading 0: 43%|████▎ | 220/507 [00:10<00:15, 18.85it/s]
Loading 0: 44%|████▍ | 223/507 [00:10<00:14, 19.21it/s]
Loading 0: 45%|████▍ | 226/507 [00:10<00:13, 21.22it/s]
Loading 0: 45%|████▌ | 229/507 [00:11<00:12, 21.54it/s]
Loading 0: 46%|████▌ | 232/507 [00:11<00:12, 21.31it/s]
Loading 0: 46%|████▋ | 235/507 [00:11<00:11, 23.03it/s]
Loading 0: 47%|████▋ | 239/507 [00:11<00:10, 26.48it/s]
Loading 0: 48%|████▊ | 242/507 [00:11<00:14, 18.91it/s]
Loading 0: 48%|████▊ | 245/507 [00:11<00:12, 20.20it/s]
Loading 0: 49%|████▉ | 248/507 [00:12<00:15, 17.26it/s]
Loading 0: 50%|████▉ | 253/507 [00:12<00:11, 22.63it/s]
Loading 0: 50%|█████ | 256/507 [00:12<00:11, 22.41it/s]
Loading 0: 51%|█████ | 259/507 [00:12<00:11, 22.24it/s]
Loading 0: 52%|█████▏ | 262/507 [00:12<00:10, 23.88it/s]
Loading 0: 52%|█████▏ | 266/507 [00:12<00:09, 26.36it/s]
Loading 0: 53%|█████▎ | 269/507 [00:12<00:12, 18.88it/s]
Loading 0: 54%|█████▎ | 272/507 [00:13<00:11, 20.03it/s]
Loading 0: 54%|█████▍ | 275/507 [00:13<00:13, 17.03it/s]
Loading 0: 55%|█████▌ | 280/507 [00:13<00:09, 22.77it/s]
Loading 0: 56%|█████▌ | 283/507 [00:13<00:09, 23.11it/s]
Loading 0: 56%|█████▋ | 286/507 [00:13<00:09, 23.26it/s]
Loading 0: 57%|█████▋ | 289/507 [00:13<00:08, 24.53it/s]
Loading 0: 58%|█████▊ | 293/507 [00:13<00:07, 26.92it/s]
Loading 0: 58%|█████▊ | 296/507 [00:14<00:10, 19.62it/s]
Loading 0: 59%|█████▉ | 299/507 [00:14<00:10, 20.63it/s]
Loading 0: 60%|█████▉ | 302/507 [00:29<04:59, 1.46s/it]
Loading 0: 60%|██████ | 305/507 [00:29<03:31, 1.05s/it]
Loading 0: 61%|██████ | 308/507 [00:29<02:30, 1.32it/s]
Loading 0: 61%|██████▏ | 311/507 [00:29<01:49, 1.79it/s]
Loading 0: 62%|██████▏ | 316/507 [00:29<01:04, 2.96it/s]
Loading 0: 63%|██████▎ | 320/507 [00:30<00:44, 4.20it/s]
Loading 0: 64%|██████▍ | 324/507 [00:30<00:35, 5.19it/s]
Loading 0: 65%|██████▍ | 328/507 [00:30<00:25, 6.98it/s]
Loading 0: 65%|██████▌ | 331/507 [00:30<00:20, 8.39it/s]
Loading 0: 66%|██████▋ | 336/507 [00:30<00:14, 11.41it/s]
Loading 0: 67%|██████▋ | 339/507 [00:31<00:13, 12.51it/s]
Loading 0: 68%|██████▊ | 343/507 [00:31<00:10, 15.69it/s]
Loading 0: 68%|██████▊ | 347/507 [00:31<00:08, 18.87it/s]
Loading 0: 69%|██████▉ | 350/507 [00:31<00:09, 16.22it/s]
Loading 0: 70%|██████▉ | 353/507 [00:31<00:08, 17.68it/s]
Loading 0: 70%|███████ | 356/507 [00:31<00:09, 16.15it/s]
Loading 0: 71%|███████ | 361/507 [00:31<00:06, 21.44it/s]
Loading 0: 72%|███████▏ | 364/507 [00:32<00:06, 22.00it/s]
Loading 0: 72%|███████▏ | 367/507 [00:32<00:06, 22.52it/s]
Loading 0: 73%|███████▎ | 371/507 [00:32<00:05, 22.96it/s]
Loading 0: 74%|███████▍ | 375/507 [00:32<00:05, 26.22it/s]
Loading 0: 75%|███████▍ | 378/507 [00:32<00:07, 17.82it/s]
Loading 0: 75%|███████▌ | 382/507 [00:32<00:06, 20.62it/s]
Loading 0: 76%|███████▌ | 385/507 [00:33<00:05, 21.09it/s]
Loading 0: 77%|███████▋ | 390/507 [00:33<00:04, 23.79it/s]
Loading 0: 78%|███████▊ | 393/507 [00:33<00:05, 21.90it/s]
Loading 0: 78%|███████▊ | 397/507 [00:33<00:04, 25.14it/s]
Loading 0: 79%|███████▉ | 401/507 [00:33<00:03, 27.49it/s]
Loading 0: 80%|███████▉ | 404/507 [00:33<00:05, 20.18it/s]
Loading 0: 80%|████████ | 407/507 [00:34<00:04, 21.06it/s]
Loading 0: 81%|████████ | 410/507 [00:34<00:05, 17.43it/s]
Loading 0: 82%|████████▏ | 415/507 [00:34<00:04, 22.77it/s]
Loading 0: 82%|████████▏ | 418/507 [00:34<00:03, 22.81it/s]
Loading 0: 83%|████████▎ | 421/507 [00:34<00:03, 22.92it/s]
Loading 0: 84%|████████▍ | 425/507 [00:34<00:03, 23.15it/s]
Loading 0: 85%|████████▍ | 429/507 [00:34<00:02, 26.34it/s]
Loading 0: 85%|████████▌ | 432/507 [00:35<00:04, 17.22it/s]
Loading 0: 86%|████████▌ | 436/507 [00:35<00:03, 20.27it/s]
Loading 0: 87%|████████▋ | 439/507 [00:35<00:03, 20.90it/s]
Loading 0: 87%|████████▋ | 442/507 [00:35<00:02, 22.76it/s]
Loading 0: 88%|████████▊ | 445/507 [00:35<00:02, 22.84it/s]
Loading 0: 88%|████████▊ | 448/507 [00:35<00:02, 22.86it/s]
Loading 0: 89%|████████▉ | 451/507 [00:36<00:02, 24.33it/s]
Loading 0: 90%|████████▉ | 455/507 [00:36<00:01, 26.94it/s]
Loading 0: 90%|█████████ | 458/507 [00:36<00:02, 18.48it/s]
Loading 0: 91%|█████████ | 461/507 [00:36<00:02, 19.73it/s]
Loading 0: 92%|█████████▏| 464/507 [00:36<00:02, 16.91it/s]
Loading 0: 93%|█████████▎| 469/507 [00:36<00:01, 22.23it/s]
Loading 0: 93%|█████████▎| 472/507 [00:37<00:01, 22.07it/s]
Loading 0: 94%|█████████▎| 475/507 [00:37<00:01, 22.14it/s]
Loading 0: 94%|█████████▍| 478/507 [00:37<00:01, 23.70it/s]
Loading 0: 95%|█████████▍| 481/507 [00:37<00:01, 25.06it/s]
Loading 0: 95%|█████████▌| 484/507 [00:39<00:05, 3.92it/s]
Loading 0: 96%|█████████▌| 487/507 [00:39<00:03, 5.02it/s]
Loading 0: 97%|█████████▋| 491/507 [00:40<00:02, 7.09it/s]
Loading 0: 97%|█████████▋| 494/507 [00:40<00:01, 8.81it/s]
Loading 0: 98%|█████████▊| 497/507 [00:40<00:00, 10.99it/s]
Loading 0: 99%|█████████▊| 500/507 [00:40<00:00, 12.80it/s]
Loading 0: 99%|█████████▉| 503/507 [00:40<00:00, 14.40it/s]
Loading 0: 100%|█████████▉| 506/507 [00:40<00:00, 16.54it/s]
Job zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer completed after 206.52s with status: succeeded
Stopping job with name zonemercy-viral-ss-v0-1e5ep2-v78-mkmlizer
Pipeline stage MKMLizer completed in 207.12s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-viral-ss-v0-1e5ep2-v78
Waiting for inference service zonemercy-viral-ss-v0-1e5ep2-v78 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service zonemercy-viral-ss-v0-1e5ep2-v78 ready after 140.5235674381256s
Pipeline stage MKMLDeployer completed in 141.09s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.758049249649048s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.264181613922119s
Received healthy response to inference request in 2.722855806350708s
Received healthy response to inference request in 2.626049757003784s
5 requests
1 failed requests
5th percentile: 2.336555242538452
10th percentile: 2.408928871154785
20th percentile: 2.553676128387451
30th percentile: 2.645410966873169
40th percentile: 2.6841333866119386
50th percentile: 2.722855806350708
60th percentile: 2.736933183670044
70th percentile: 2.75101056098938
80th percentile: 6.236992025375369
90th percentile: 13.194877576828006
95th percentile: 16.67382035255432
99th percentile: 19.456974573135376
mean time: 6.10477991104126
%s, retrying in %s seconds...
Received healthy response to inference request in 2.6616697311401367s
Received healthy response to inference request in 2.9086787700653076s
Received healthy response to inference request in 2.8299367427825928s
Received healthy response to inference request in 2.7089810371398926s
Received healthy response to inference request in 2.2722156047821045s
5 requests
0 failed requests
5th percentile: 2.350106430053711
10th percentile: 2.4279972553253173
20th percentile: 2.58377890586853
30th percentile: 2.671131992340088
40th percentile: 2.6900565147399904
50th percentile: 2.7089810371398926
60th percentile: 2.7573633193969727
70th percentile: 2.8057456016540527
80th percentile: 2.8456851482391357
90th percentile: 2.877181959152222
95th percentile: 2.8929303646087647
99th percentile: 2.905529088973999
mean time: 2.6762963771820067
Pipeline stage StressChecker completed in 47.80s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.62s
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5ep2_v78 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3208.15s
Shutdown handler de-registered
zonemercy-viral-ss-v0-1e5ep2_v78 status is now inactive due to auto deactivation removed underperforming models