Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alpindale-wizardlm-2-8x22b-v1-mkmlizer
Waiting for job on alpindale-wizardlm-2-8x22b-v1-mkmlizer to finish
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ _____ __ __ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ /___/ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ Version: 0.11.12 ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ https://mk1.ai ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ belonging to: ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ Chai Research Corp. ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ║ ║
alpindale-wizardlm-2-8x22b-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alpindale-wizardlm-2-8x22b-v1-mkmlizer: Downloaded to shared memory in 528.085s
alpindale-wizardlm-2-8x22b-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp4khtxd8d, device:0
alpindale-wizardlm-2-8x22b-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alpindale-wizardlm-2-8x22b-v1-mkmlizer:
Loading 0: 0%| | 0/1739 [00:00<?, ?it/s]
Loading 0: 0%| | 4/1739 [00:00<01:21, 21.27it/s]
Loading 0: 0%| | 7/1739 [00:00<01:31, 18.88it/s]
Loading 0: 1%| | 10/1739 [00:00<01:33, 18.45it/s]
Loading 0: 1%| | 13/1739 [00:00<01:32, 18.73it/s]
Loading 0: 1%| | 16/1739 [00:00<01:33, 18.39it/s]
Loading 0: 1%| | 19/1739 [00:01<01:33, 18.33it/s]
Loading 0: 1%|▏ | 22/1739 [00:01<01:32, 18.47it/s]
Loading 0: 2%|▏ | 28/1739 [00:01<01:06, 25.91it/s]
Loading 0: 2%|▏ | 31/1739 [00:01<01:32, 18.44it/s]
Loading 0: 2%|▏ | 35/1739 [00:01<01:28, 19.36it/s]
Loading 0: 2%|▏ | 38/1739 [00:01<01:29, 19.06it/s]
Loading 0: 2%|▏ | 41/1739 [00:02<01:30, 18.75it/s]
Loading 0: 3%|▎ | 44/1739 [00:02<01:31, 18.58it/s]
Loading 0: 3%|▎ | 47/1739 [00:02<01:30, 18.62it/s]
Loading 0: 3%|▎ | 50/1739 [00:02<01:30, 18.58it/s]
Loading 0: 3%|▎ | 53/1739 [00:02<01:33, 18.07it/s]
Loading 0: 3%|▎ | 58/1739 [00:02<01:09, 24.26it/s]
Loading 0: 4%|▎ | 61/1739 [00:03<01:38, 16.99it/s]
Loading 0: 4%|▍ | 66/1739 [00:03<01:22, 20.29it/s]
Loading 0: 4%|▍ | 69/1739 [00:03<01:25, 19.62it/s]
Loading 0: 4%|▍ | 72/1739 [00:03<01:25, 19.56it/s]
Loading 0: 4%|▍ | 75/1739 [00:03<01:26, 19.19it/s]
Loading 0: 4%|▍ | 78/1739 [00:04<01:25, 19.37it/s]
Loading 0: 5%|▍ | 81/1739 [00:04<01:29, 18.61it/s]
Loading 0: 5%|▌ | 87/1739 [00:04<01:01, 26.86it/s]
Loading 0: 5%|▌ | 91/1739 [00:04<01:32, 17.75it/s]
Loading 0: 5%|▌ | 95/1739 [00:04<01:21, 20.20it/s]
Loading 0: 6%|▌ | 98/1739 [00:05<01:24, 19.46it/s]
Loading 0: 6%|▌ | 101/1739 [00:05<01:25, 19.05it/s]
Loading 0: 6%|▌ | 104/1739 [00:05<01:26, 18.87it/s]
Loading 0: 6%|▌ | 107/1739 [00:05<01:29, 18.31it/s]
Loading 0: 6%|▋ | 109/1739 [00:05<01:33, 17.40it/s]
Loading 0: 6%|▋ | 112/1739 [00:05<01:32, 17.68it/s]
Loading 0: 7%|▋ | 118/1739 [00:05<01:04, 25.08it/s]
Loading 0: 7%|▋ | 121/1739 [00:06<01:34, 17.21it/s]
Loading 0: 7%|▋ | 124/1739 [00:06<01:28, 18.29it/s]
Loading 0: 7%|▋ | 128/1739 [00:06<01:23, 19.20it/s]
Loading 0: 8%|▊ | 131/1739 [00:06<01:27, 18.39it/s]
Loading 0: 8%|▊ | 134/1739 [00:06<01:28, 18.14it/s]
Loading 0: 8%|▊ | 137/1739 [00:07<01:28, 18.13it/s]
Loading 0: 8%|▊ | 140/1739 [00:07<01:30, 17.69it/s]
Loading 0: 8%|▊ | 143/1739 [00:07<01:29, 17.92it/s]
Loading 0: 9%|▊ | 149/1739 [00:07<01:23, 19.14it/s]
Loading 0: 9%|▊ | 151/1739 [00:07<01:27, 18.15it/s]
Loading 0: 9%|▉ | 154/1739 [00:08<01:28, 17.86it/s]
Loading 0: 9%|▉ | 159/1739 [00:08<01:13, 21.39it/s]
Loading 0: 9%|▉ | 161/1739 [00:22<01:13, 21.39it/s]
Loading 0: 9%|▉ | 162/1739 [00:22<33:20, 1.27s/it]
Loading 0: 9%|▉ | 163/1739 [00:22<30:02, 1.14s/it]
Loading 0: 9%|▉ | 165/1739 [00:23<23:47, 1.10it/s]
Loading 0: 10%|▉ | 168/1739 [00:23<16:18, 1.61it/s]
Loading 0: 10%|▉ | 171/1739 [00:23<11:37, 2.25it/s]
Loading 0: 10%|█ | 178/1739 [00:23<05:53, 4.41it/s]
Loading 0: 10%|█ | 181/1739 [00:24<05:10, 5.02it/s]
Loading 0: 11%|█ | 183/1739 [00:24<04:31, 5.73it/s]
Loading 0: 11%|█ | 185/1739 [00:24<03:54, 6.62it/s]
Loading 0: 11%|█ | 190/1739 [00:24<02:35, 9.95it/s]
Loading 0: 11%|█ | 193/1739 [00:24<02:16, 11.37it/s]
Loading 0: 11%|█▏ | 196/1739 [00:24<02:03, 12.52it/s]
Loading 0: 11%|█▏ | 199/1739 [00:25<01:53, 13.54it/s]
Loading 0: 12%|█▏ | 202/1739 [00:25<01:44, 14.70it/s]
Loading 0: 12%|█▏ | 208/1739 [00:25<01:11, 21.45it/s]
Loading 0: 12%|█▏ | 211/1739 [00:25<01:36, 15.91it/s]
Loading 0: 12%|█▏ | 214/1739 [00:25<01:33, 16.32it/s]
Loading 0: 12%|█▏ | 217/1739 [00:25<01:26, 17.63it/s]
Loading 0: 13%|█▎ | 221/1739 [00:26<01:18, 19.34it/s]
Loading 0: 13%|█▎ | 224/1739 [00:26<01:20, 18.92it/s]
Loading 0: 13%|█▎ | 227/1739 [00:26<01:19, 19.03it/s]
Loading 0: 13%|█▎ | 230/1739 [00:26<01:19, 18.96it/s]
Loading 0: 13%|█▎ | 233/1739 [00:26<01:20, 18.81it/s]
Loading 0: 14%|█▎ | 239/1739 [00:27<01:14, 20.10it/s]
Loading 0: 14%|█▍ | 242/1739 [00:27<01:18, 19.10it/s]
Loading 0: 14%|█▍ | 244/1739 [00:27<01:22, 18.09it/s]
Loading 0: 14%|█▍ | 247/1739 [00:27<01:23, 17.84it/s]
Loading 0: 14%|█▍ | 252/1739 [00:27<01:09, 21.33it/s]
Loading 0: 15%|█▍ | 255/1739 [00:27<01:12, 20.46it/s]
Loading 0: 15%|█▍ | 258/1739 [00:28<01:16, 19.40it/s]
Loading 0: 15%|█▌ | 261/1739 [00:28<01:17, 18.96it/s]
Loading 0: 15%|█▌ | 268/1739 [00:28<00:56, 25.96it/s]
Loading 0: 16%|█▌ | 271/1739 [00:28<01:23, 17.58it/s]
Loading 0: 16%|█▌ | 274/1739 [00:28<01:23, 17.63it/s]
Loading 0: 16%|█▌ | 276/1739 [00:29<01:27, 16.75it/s]
Loading 0: 16%|█▌ | 278/1739 [00:29<01:28, 16.53it/s]
Loading 0: 16%|█▋ | 283/1739 [00:29<01:09, 21.04it/s]
Loading 0: 16%|█▋ | 286/1739 [00:29<01:11, 20.19it/s]
Loading 0: 17%|█▋ | 289/1739 [00:29<01:13, 19.84it/s]
Loading 0: 17%|█▋ | 292/1739 [00:29<01:13, 19.75it/s]
Loading 0: 17%|█▋ | 298/1739 [00:29<00:54, 26.46it/s]
Loading 0: 17%|█▋ | 301/1739 [00:30<01:21, 17.73it/s]
Loading 0: 17%|█▋ | 304/1739 [00:30<01:24, 17.03it/s]
Loading 0: 18%|█▊ | 307/1739 [00:30<01:25, 16.79it/s]
Loading 0: 18%|█▊ | 310/1739 [00:30<01:26, 16.57it/s]
Loading 0: 18%|█▊ | 315/1739 [00:30<01:03, 22.35it/s]
Loading 0: 18%|█▊ | 318/1739 [00:31<01:07, 21.09it/s]
Loading 0: 18%|█▊ | 321/1739 [00:31<01:10, 19.98it/s]
Loading 0: 19%|█▊ | 324/1739 [00:31<01:13, 19.34it/s]
Loading 0: 19%|█▊ | 325/1739 [00:46<01:13, 19.34it/s]
Loading 0: 19%|█▊ | 326/1739 [00:46<37:04, 1.57s/it]
Loading 0: 19%|█▉ | 329/1739 [00:46<26:11, 1.11s/it]
Loading 0: 19%|█▉ | 331/1739 [00:46<20:35, 1.14it/s]
Loading 0: 19%|█▉ | 334/1739 [00:46<14:14, 1.64it/s]
Loading 0: 19%|█▉ | 337/1739 [00:46<10:05, 2.31it/s]
Loading 0: 20%|█▉ | 340/1739 [00:47<07:21, 3.17it/s]
Loading 0: 20%|█▉ | 345/1739 [00:47<04:33, 5.10it/s]
Loading 0: 20%|██ | 348/1739 [00:47<03:40, 6.32it/s]
Loading 0: 20%|██ | 351/1739 [00:47<03:01, 7.65it/s]
Loading 0: 21%|██ | 358/1739 [00:47<01:48, 12.75it/s]
Loading 0: 21%|██ | 361/1739 [00:48<01:59, 11.50it/s]
Loading 0: 21%|██ | 364/1739 [00:48<01:48, 12.65it/s]
Loading 0: 21%|██ | 367/1739 [00:48<01:41, 13.53it/s]
Loading 0: 21%|██ | 369/1739 [00:48<01:41, 13.47it/s]
Loading 0: 21%|██▏ | 371/1739 [00:48<01:38, 13.92it/s]
Loading 0: 22%|██▏ | 376/1739 [00:48<01:14, 18.29it/s]
Loading 0: 22%|██▏ | 379/1739 [00:49<01:14, 18.17it/s]
Loading 0: 22%|██▏ | 382/1739 [00:49<01:13, 18.36it/s]
Loading 0: 22%|██▏ | 388/1739 [00:49<00:54, 24.89it/s]
Loading 0: 22%|██▏ | 391/1739 [00:49<01:19, 16.96it/s]
Loading 0: 23%|██▎ | 394/1739 [00:49<01:19, 16.96it/s]
Loading 0: 23%|██▎ | 397/1739 [00:50<01:20, 16.71it/s]
Loading 0: 23%|██▎ | 399/1739 [00:50<01:21, 16.45it/s]
Loading 0: 23%|██▎ | 402/1739 [00:50<01:17, 17.28it/s]
Loading 0: 23%|██▎ | 407/1739 [00:50<01:04, 20.64it/s]
Loading 0: 24%|██▎ | 410/1739 [00:50<01:06, 20.12it/s]
Loading 0: 24%|██▎ | 413/1739 [00:50<01:07, 19.71it/s]
Loading 0: 24%|██▍ | 418/1739 [00:51<00:52, 25.27it/s]
Loading 0: 24%|██▍ | 421/1739 [00:51<01:18, 16.70it/s]
Loading 0: 24%|██▍ | 424/1739 [00:51<01:18, 16.80it/s]
Loading 0: 25%|██▍ | 427/1739 [00:51<01:18, 16.72it/s]
Loading 0: 25%|██▍ | 430/1739 [00:51<01:19, 16.56it/s]
Loading 0: 25%|██▍ | 433/1739 [00:52<01:19, 16.39it/s]
Loading 0: 25%|██▌ | 438/1739 [00:52<01:05, 19.91it/s]
Loading 0: 25%|██▌ | 441/1739 [00:52<01:08, 19.00it/s]
Loading 0: 26%|██▌ | 448/1739 [00:52<00:50, 25.39it/s]
Loading 0: 26%|██▌ | 451/1739 [00:52<01:13, 17.62it/s]
Loading 0: 26%|██▌ | 454/1739 [00:53<01:12, 17.64it/s]
Loading 0: 26%|██▌ | 456/1739 [00:53<01:15, 17.07it/s]
Loading 0: 26%|██▋ | 458/1739 [00:53<01:16, 16.71it/s]
Loading 0: 27%|██▋ | 461/1739 [00:53<01:15, 16.91it/s]
Loading 0: 27%|██▋ | 464/1739 [00:53<01:17, 16.45it/s]
Loading 0: 27%|██▋ | 469/1739 [00:53<01:02, 20.22it/s]
Loading 0: 27%|██▋ | 472/1739 [00:54<01:07, 18.90it/s]
Loading 0: 27%|██▋ | 478/1739 [00:54<00:51, 24.58it/s]
Loading 0: 28%|██▊ | 481/1739 [00:54<01:11, 17.48it/s]
Loading 0: 28%|██▊ | 484/1739 [00:54<01:10, 17.68it/s]
Loading 0: 28%|██▊ | 486/1739 [00:54<01:11, 17.43it/s]
Loading 0: 28%|██▊ | 488/1739 [01:09<01:11, 17.43it/s]
Loading 0: 28%|██▊ | 489/1739 [01:09<30:27, 1.46s/it]
Loading 0: 28%|██▊ | 490/1739 [01:09<26:59, 1.30s/it]
Loading 0: 28%|██▊ | 492/1739 [01:09<20:46, 1.00it/s]
Loading 0: 28%|██▊ | 495/1739 [01:10<13:45, 1.51it/s]
Loading 0: 29%|██▉ | 500/1739 [01:10<07:44, 2.67it/s]
Loading 0: 29%|██▉ | 503/1739 [01:10<05:50, 3.52it/s]
Loading 0: 29%|██▉ | 509/1739 [01:10<03:39, 5.59it/s]
Loading 0: 29%|██▉ | 511/1739 [01:10<03:15, 6.30it/s]
Loading 0: 30%|██▉ | 514/1739 [01:10<02:39, 7.68it/s]
Loading 0: 30%|██▉ | 517/1739 [01:11<02:15, 9.05it/s]
Loading 0: 30%|██▉ | 520/1739 [01:11<01:55, 10.58it/s]
Loading 0: 30%|███ | 523/1739 [01:11<01:41, 12.01it/s]
Loading 0: 30%|███ | 526/1739 [01:11<01:32, 13.14it/s]
Loading 0: 31%|███ | 531/1739 [01:11<01:10, 17.07it/s]
Loading 0: 31%|███ | 538/1739 [01:11<00:51, 23.23it/s]
Loading 0: 31%|███ | 541/1739 [01:12<01:09, 17.27it/s]
Loading 0: 31%|███▏ | 544/1739 [01:12<01:08, 17.36it/s]
Loading 0: 31%|███▏ | 547/1739 [01:12<01:09, 17.08it/s]
Loading 0: 32%|███▏ | 549/1739 [01:12<01:13, 16.14it/s]
Loading 0: 32%|███▏ | 551/1739 [01:12<01:16, 15.58it/s]
Loading 0: 32%|███▏ | 554/1739 [01:13<01:15, 15.78it/s]
Loading 0: 32%|███▏ | 557/1739 [01:13<01:13, 16.04it/s]
Loading 0: 32%|███▏ | 562/1739 [01:13<01:00, 19.48it/s]
Loading 0: 33%|███▎ | 568/1739 [01:13<00:46, 25.38it/s]
Loading 0: 33%|███▎ | 571/1739 [01:14<01:07, 17.39it/s]
Loading 0: 33%|███▎ | 574/1739 [01:14<01:06, 17.58it/s]
Loading 0: 33%|███▎ | 577/1739 [01:14<01:05, 17.64it/s]
Loading 0: 33%|███▎ | 579/1739 [01:14<01:07, 17.14it/s]
Loading 0: 33%|███▎ | 582/1739 [01:14<01:07, 17.19it/s]
Loading 0: 34%|███▎ | 585/1739 [01:14<01:08, 16.97it/s]
Loading 0: 34%|███▍ | 588/1739 [01:15<01:08, 16.72it/s]
Loading 0: 34%|███▍ | 593/1739 [01:15<00:57, 20.08it/s]
Loading 0: 34%|███▍ | 598/1739 [01:15<00:44, 25.78it/s]
Loading 0: 35%|███▍ | 601/1739 [01:15<01:03, 17.79it/s]
Loading 0: 35%|███▍ | 604/1739 [01:15<01:03, 17.97it/s]
Loading 0: 35%|███▍ | 607/1739 [01:15<01:02, 17.98it/s]
Loading 0: 35%|███▌ | 610/1739 [01:16<01:03, 17.76it/s]
Loading 0: 35%|███▌ | 613/1739 [01:16<01:02, 17.95it/s]
Loading 0: 35%|███▌ | 616/1739 [01:16<01:03, 17.80it/s]
Loading 0: 36%|███▌ | 619/1739 [01:16<01:02, 18.03it/s]
Loading 0: 36%|███▌ | 628/1739 [01:16<00:38, 28.78it/s]
Loading 0: 36%|███▋ | 631/1739 [01:17<00:56, 19.55it/s]
Loading 0: 36%|███▋ | 634/1739 [01:17<00:57, 19.23it/s]
Loading 0: 37%|███▋ | 637/1739 [01:17<00:58, 18.79it/s]
Loading 0: 37%|███▋ | 640/1739 [01:17<00:58, 18.86it/s]
Loading 0: 37%|███▋ | 643/1739 [01:17<00:58, 18.68it/s]
Loading 0: 37%|███▋ | 645/1739 [01:17<01:01, 17.82it/s]
Loading 0: 37%|███▋ | 647/1739 [01:18<01:02, 17.44it/s]
Loading 0: 37%|███▋ | 649/1739 [01:32<01:02, 17.44it/s]
Loading 0: 37%|███▋ | 650/1739 [01:32<29:48, 1.64s/it]
Loading 0: 38%|███▊ | 653/1739 [01:32<20:28, 1.13s/it]
Loading 0: 38%|███▊ | 658/1739 [01:33<11:51, 1.52it/s]
Loading 0: 38%|███▊ | 661/1739 [01:33<09:10, 1.96it/s]
Loading 0: 38%|███▊ | 663/1739 [01:33<07:30, 2.39it/s]
Loading 0: 38%|███▊ | 666/1739 [01:33<05:29, 3.25it/s]
Loading 0: 38%|███▊ | 669/1739 [01:33<04:06, 4.34it/s]
Loading 0: 39%|███▊ | 672/1739 [01:33<03:09, 5.63it/s]
Loading 0: 39%|███▉ | 675/1739 [01:34<02:31, 7.03it/s]
Loading 0: 39%|███▉ | 678/1739 [01:34<02:05, 8.47it/s]
Loading 0: 39%|███▉ | 681/1739 [01:34<01:47, 9.86it/s]
Loading 0: 40%|███▉ | 688/1739 [01:34<01:01, 17.14it/s]
Loading 0: 40%|███▉ | 692/1739 [01:34<01:11, 14.74it/s]
Loading 0: 40%|███▉ | 695/1739 [01:35<01:08, 15.22it/s]
Loading 0: 40%|████ | 698/1739 [01:35<01:06, 15.63it/s]
Loading 0: 40%|████ | 701/1739 [01:35<01:04, 15.97it/s]
Loading 0: 40%|████ | 703/1739 [01:35<01:06, 15.68it/s]
Loading 0: 41%|████ | 706/1739 [01:35<01:03, 16.17it/s]
Loading 0: 41%|████ | 709/1739 [01:36<01:02, 16.46it/s]
Loading 0: 41%|████ | 712/1739 [01:36<01:01, 16.63it/s]
Loading 0: 41%|████ | 717/1739 [01:36<00:44, 23.03it/s]
Loading 0: 41%|████▏ | 720/1739 [01:36<01:03, 16.01it/s]
Loading 0: 42%|████▏ | 723/1739 [01:36<01:02, 16.19it/s]
Loading 0: 42%|████▏ | 726/1739 [01:36<01:00, 16.78it/s]
Loading 0: 42%|████▏ | 729/1739 [01:37<00:58, 17.24it/s]
Loading 0: 42%|████▏ | 732/1739 [01:37<00:59, 16.93it/s]
Loading 0: 42%|████▏ | 735/1739 [01:37<00:58, 17.07it/s]
Loading 0: 42%|████▏ | 738/1739 [01:37<00:57, 17.27it/s]
Loading 0: 43%|████▎ | 741/1739 [01:37<00:58, 17.19it/s]
Loading 0: 43%|████▎ | 748/1739 [01:38<00:46, 21.39it/s]
Loading 0: 43%|████▎ | 751/1739 [01:38<00:49, 19.96it/s]
Loading 0: 43%|████▎ | 753/1739 [01:38<00:52, 18.87it/s]
Loading 0: 43%|████▎ | 756/1739 [01:38<00:52, 18.60it/s]
Loading 0: 44%|████▎ | 759/1739 [01:38<00:54, 18.14it/s]
Loading 0: 44%|████▍ | 762/1739 [01:38<00:55, 17.72it/s]
Loading 0: 44%|████▍ | 765/1739 [01:39<00:55, 17.69it/s]
Loading 0: 44%|████▍ | 768/1739 [01:39<00:57, 16.93it/s]
Loading 0: 44%|████▍ | 771/1739 [01:39<00:57, 16.88it/s]
Loading 0: 45%|████▍ | 777/1739 [01:39<00:39, 24.54it/s]
Loading 0: 45%|████▍ | 780/1739 [01:39<00:56, 17.05it/s]
Loading 0: 45%|████▌ | 783/1739 [01:40<00:55, 17.10it/s]
Loading 0: 45%|████▌ | 786/1739 [01:40<00:56, 17.00it/s]
Loading 0: 45%|████▌ | 789/1739 [01:40<00:55, 17.19it/s]
Loading 0: 46%|████▌ | 792/1739 [01:40<00:55, 17.14it/s]
Loading 0: 46%|████▌ | 795/1739 [01:40<00:55, 17.15it/s]
Loading 0: 46%|████▌ | 798/1739 [01:40<00:56, 16.72it/s]
Loading 0: 46%|████▌ | 801/1739 [01:41<00:56, 16.72it/s]
Loading 0: 46%|████▋ | 807/1
alpindale-wizardlm-2-8x22b-v1-mkmlizer: quantized model in 256.921s
alpindale-wizardlm-2-8x22b-v1-mkmlizer: Processed model alpindale/WizardLM-2-8x22B in 785.007s
alpindale-wizardlm-2-8x22b-v1-mkmlizer: creating bucket guanaco-mkml-models
alpindale-wizardlm-2-8x22b-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alpindale-wizardlm-2-8x22b-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/config.json
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/tokenizer.model
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/tokenizer.json
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.10.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.10.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.0.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.5.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.5.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.3.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.8.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.8.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.9.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.9.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.2.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.1.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.4.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.4.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.6.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.6.safetensors
alpindale-wizardlm-2-8x22b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.7.safetensors s3://guanaco-mkml-models/alpindale-wizardlm-2-8x22b-v1/flywheel_model.7.safetensors
Job alpindale-wizardlm-2-8x22b-v1-mkmlizer completed after 865.24s with status: succeeded
Stopping job with name alpindale-wizardlm-2-8x22b-v1-mkmlizer
Pipeline stage MKMLizer completed in 866.15s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alpindale-wizardlm-2-8x22b-v1
Waiting for inference service alpindale-wizardlm-2-8x22b-v1 to be ready
Inference service alpindale-wizardlm-2-8x22b-v1 ready after 140.32335114479065s
Pipeline stage MKMLDeployer completed in 140.70s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.3553450107574463s
Received healthy response to inference request in 3.4020228385925293s
Received healthy response to inference request in 3.849824905395508s
Received healthy response to inference request in 4.173811197280884s
Received healthy response to inference request in 3.955357074737549s
5 requests
0 failed requests
5th percentile: 3.3646805763244627
10th percentile: 3.3740161418914796
20th percentile: 3.392687273025513
30th percentile: 3.491583251953125
40th percentile: 3.6707040786743166
50th percentile: 3.849824905395508
60th percentile: 3.892037773132324
70th percentile: 3.9342506408691404
80th percentile: 3.9990478992462157
90th percentile: 4.08642954826355
95th percentile: 4.130120372772216
99th percentile: 4.1650730323791505
mean time: 3.7472722053527834
%s, retrying in %s seconds...
Received healthy response to inference request in 3.33345627784729s
Received healthy response to inference request in 3.5338857173919678s
Received healthy response to inference request in 4.810327529907227s
Received healthy response to inference request in 3.6788926124572754s
Received healthy response to inference request in 3.9476258754730225s
5 requests
0 failed requests
5th percentile: 3.3735421657562257
10th percentile: 3.4136280536651613
20th percentile: 3.493799829483032
30th percentile: 3.562887096405029
40th percentile: 3.6208898544311525
50th percentile: 3.6788926124572754
60th percentile: 3.786385917663574
70th percentile: 3.893879222869873
80th percentile: 4.120166206359864
90th percentile: 4.465246868133545
95th percentile: 4.637787199020385
99th percentile: 4.775819463729858
mean time: 3.8608376026153564
%s, retrying in %s seconds...
Received healthy response to inference request in 2.612572431564331s
Received healthy response to inference request in 4.003966808319092s
Received healthy response to inference request in 3.2157392501831055s
Received healthy response to inference request in 3.241497039794922s
Received healthy response to inference request in 3.9511430263519287s
5 requests
0 failed requests
5th percentile: 2.733205795288086
10th percentile: 2.853839159011841
20th percentile: 3.0951058864593506
30th percentile: 3.2208908081054686
40th percentile: 3.2311939239501952
50th percentile: 3.241497039794922
60th percentile: 3.5253554344177247
70th percentile: 3.809213829040527
80th percentile: 3.9617077827453615
90th percentile: 3.9828372955322267
95th percentile: 3.993402051925659
99th percentile: 4.001853857040405
mean time: 3.404983711242676
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 3.809213829040527s')
Shutdown handler de-registered
alpindale-wizardlm-2-8x22b_v1 status is now failed due to DeploymentManager action
alpindale-wizardlm-2-8x22b_v1 status is now torndown due to DeploymentManager action