Running pipeline stage MKMLizer
Starting job with name khanhnto-khanhnto-v38-mkmlizer
Waiting for job on khanhnto-khanhnto-v38-mkmlizer to finish
khanhnto-khanhnto-v38-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
khanhnto-khanhnto-v38-mkmlizer: ║ _____ __ __ ║
khanhnto-khanhnto-v38-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
khanhnto-khanhnto-v38-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
khanhnto-khanhnto-v38-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
khanhnto-khanhnto-v38-mkmlizer: ║ /___/ ║
khanhnto-khanhnto-v38-mkmlizer: ║ ║
khanhnto-khanhnto-v38-mkmlizer: ║ Version: 0.6.11 ║
khanhnto-khanhnto-v38-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
khanhnto-khanhnto-v38-mkmlizer: ║ ║
khanhnto-khanhnto-v38-mkmlizer: ║ The license key for the current software has been verified as ║
khanhnto-khanhnto-v38-mkmlizer: ║ belonging to: ║
khanhnto-khanhnto-v38-mkmlizer: ║ ║
khanhnto-khanhnto-v38-mkmlizer: ║ Chai Research Corp. ║
khanhnto-khanhnto-v38-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
khanhnto-khanhnto-v38-mkmlizer: ║ Expiration: 2024-04-15 23:59:59 ║
khanhnto-khanhnto-v38-mkmlizer: ║ ║
khanhnto-khanhnto-v38-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
khanhnto-khanhnto-v38-mkmlizer:
.gitattributes: 0%| | 0.00/1.52k [00:00<?, ?B/s]
.gitattributes: 100%|██████████| 1.52k/1.52k [00:00<00:00, 14.0MB/s]
khanhnto-khanhnto-v38-mkmlizer:
added_tokens.json: 0%| | 0.00/21.0 [00:00<?, ?B/s]
added_tokens.json: 100%|██████████| 21.0/21.0 [00:00<00:00, 214kB/s]
khanhnto-khanhnto-v38-mkmlizer:
config.json: 0%| | 0.00/702 [00:00<?, ?B/s]
config.json: 100%|██████████| 702/702 [00:00<00:00, 8.02MB/s]
khanhnto-khanhnto-v38-mkmlizer:
generation_config.json: 0%| | 0.00/137 [00:00<?, ?B/s]
generation_config.json: 100%|██████████| 137/137 [00:00<00:00, 1.08MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model-00001-of-00006.safetensors: 0%| | 0.00/4.98G [00:00<?, ?B/s]
model-00001-of-00006.safetensors: 0%| | 10.5M/4.98G [00:01<10:53, 7.60MB/s]
model-00001-of-00006.safetensors: 0%| | 21.0M/4.98G [00:01<06:47, 12.2MB/s]
model-00001-of-00006.safetensors: 1%|▏ | 62.9M/4.98G [00:01<01:42, 47.8MB/s]
model-00001-of-00006.safetensors: 2%|▏ | 83.9M/4.98G [00:02<01:16, 63.9MB/s]
model-00001-of-00006.safetensors: 3%|▎ | 126M/4.98G [00:02<00:44, 109MB/s]
model-00001-of-00006.safetensors: 4%|▎ | 178M/4.98G [00:02<00:28, 168MB/s]
model-00001-of-00006.safetensors: 6%|▌ | 294M/4.98G [00:02<00:15, 305MB/s]
model-00001-of-00006.safetensors: 7%|▋ | 346M/4.98G [00:02<00:15, 300MB/s]
model-00001-of-00006.safetensors: 10%|▉ | 482M/4.98G [00:02<00:09, 494MB/s]
model-00001-of-00006.safetensors: 13%|█▎ | 629M/4.98G [00:02<00:06, 690MB/s]
model-00001-of-00006.safetensors: 25%|██▍ | 1.23G/4.98G [00:03<00:01, 1.88GB/s]
model-00001-of-00006.safetensors: 29%|██▉ | 1.47G/4.98G [00:03<00:02, 1.18GB/s]
model-00001-of-00006.safetensors: 33%|███▎ | 1.66G/4.98G [00:04<00:04, 699MB/s]
model-00001-of-00006.safetensors: 36%|███▌ | 1.80G/4.98G [00:04<00:04, 692MB/s]
model-00001-of-00006.safetensors: 39%|███▉ | 1.95G/4.98G [00:04<00:03, 789MB/s]
model-00001-of-00006.safetensors: 42%|████▏ | 2.08G/4.98G [00:04<00:03, 774MB/s]
model-00001-of-00006.safetensors: 45%|████▌ | 2.26G/4.98G [00:04<00:02, 952MB/s]
model-00001-of-00006.safetensors: 54%|█████▎ | 2.66G/4.98G [00:04<00:01, 1.51GB/s]
model-00001-of-00006.safetensors: 58%|█████▊ | 2.87G/4.98G [00:05<00:01, 1.08GB/s]
model-00001-of-00006.safetensors: 61%|██████ | 3.04G/4.98G [00:05<00:02, 936MB/s]
model-00001-of-00006.safetensors: 64%|██████▍ | 3.18G/4.98G [00:05<00:01, 913MB/s]
model-00001-of-00006.safetensors: 66%|██████▋ | 3.30G/4.98G [00:05<00:01, 937MB/s]
model-00001-of-00006.safetensors: 69%|██████▉ | 3.45G/4.98G [00:05<00:01, 1.01GB/s]
model-00001-of-00006.safetensors: 72%|███████▏ | 3.60G/4.98G [00:05<00:01, 1.10GB/s]
model-00001-of-00006.safetensors: 75%|███████▌ | 3.74G/4.98G [00:05<00:01, 1.16GB/s]
model-00001-of-00006.safetensors: 81%|████████ | 4.03G/4.98G [00:06<00:00, 1.56GB/s]
model-00001-of-00006.safetensors: 84%|████████▍ | 4.20G/4.98G [00:06<00:00, 1.57GB/s]
model-00001-of-00006.safetensors: 88%|████████▊ | 4.38G/4.98G [00:06<00:00, 1.39GB/s]
model-00001-of-00006.safetensors: 91%|█████████ | 4.54G/4.98G [00:06<00:00, 1.42GB/s]
model-00001-of-00006.safetensors: 94%|█████████▍| 4.70G/4.98G [00:06<00:00, 1.20GB/s]
model-00001-of-00006.safetensors: 98%|█████████▊| 4.89G/4.98G [00:06<00:00, 1.33GB/s]
model-00001-of-00006.safetensors: 100%|█████████▉| 4.98G/4.98G [00:07<00:00, 683MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model-00002-of-00006.safetensors: 0%| | 0.00/4.97G [00:00<?, ?B/s]
model-00002-of-00006.safetensors: 0%| | 10.5M/4.97G [00:01<11:20, 7.29MB/s]
model-00002-of-00006.safetensors: 0%| | 21.0M/4.97G [00:01<07:05, 11.6MB/s]
model-00002-of-00006.safetensors: 1%| | 41.9M/4.97G [00:02<02:58, 27.5MB/s]
model-00002-of-00006.safetensors: 1%|▏ | 73.4M/4.97G [00:02<01:26, 56.8MB/s]
model-00002-of-00006.safetensors: 3%|▎ | 136M/4.97G [00:02<00:37, 130MB/s]
model-00002-of-00006.safetensors: 5%|▍ | 231M/4.97G [00:02<00:18, 256MB/s]
model-00002-of-00006.safetensors: 6%|▋ | 315M/4.97G [00:02<00:12, 358MB/s]
model-00002-of-00006.safetensors: 8%|▊ | 377M/4.97G [00:02<00:12, 382MB/s]
model-00002-of-00006.safetensors: 9%|▉ | 472M/4.97G [00:02<00:09, 473MB/s]
model-00002-of-00006.safetensors: 17%|█▋ | 839M/4.97G [00:02<00:03, 1.18GB/s]
model-00002-of-00006.safetensors: 25%|██▌ | 1.25G/4.97G [00:03<00:01, 1.88GB/s]
model-00002-of-00006.safetensors: 30%|██▉ | 1.49G/4.97G [00:03<00:04, 765MB/s]
model-00002-of-00006.safetensors: 34%|███▎ | 1.67G/4.97G [00:04<00:04, 679MB/s]
model-00002-of-00006.safetensors: 36%|███▋ | 1.80G/4.97G [00:04<00:04, 699MB/s]
model-00002-of-00006.safetensors: 39%|███▉ | 1.93G/4.97G [00:04<00:04, 757MB/s]
model-00002-of-00006.safetensors: 42%|████▏ | 2.07G/4.97G [00:04<00:03, 849MB/s]
model-00002-of-00006.safetensors: 44%|████▍ | 2.19G/4.97G [00:04<00:03, 916MB/s]
model-00002-of-00006.safetensors: 52%|█████▏ | 2.57G/4.97G [00:04<00:01, 1.50GB/s]
model-00002-of-00006.safetensors: 56%|█████▌ | 2.77G/4.97G [00:04<00:02, 1.10GB/s]
model-00002-of-00006.safetensors: 59%|█████▉ | 2.93G/4.97G [00:05<00:02, 819MB/s]
model-00002-of-00006.safetensors: 61%|██████▏ | 3.05G/4.97G [00:05<00:02, 712MB/s]
model-00002-of-00006.safetensors: 65%|██████▍ | 3.21G/4.97G [00:05<00:02, 834MB/s]
model-00002-of-00006.safetensors: 68%|██████▊ | 3.36G/4.97G [00:05<00:01, 943MB/s]
model-00002-of-00006.safetensors: 70%|███████ | 3.48G/4.97G [00:05<00:01, 999MB/s]
model-00002-of-00006.safetensors: 74%|███████▍ | 3.67G/4.97G [00:06<00:01, 1.19GB/s]
model-00002-of-00006.safetensors: 77%|███████▋ | 3.82G/4.97G [00:06<00:01, 1.02GB/s]
model-00002-of-00006.safetensors: 79%|███████▉ | 3.94G/4.97G [00:06<00:01, 1.02GB/s]
model-00002-of-00006.safetensors: 82%|████████▏ | 4.07G/4.97G [00:06<00:01, 831MB/s]
model-00002-of-00006.safetensors: 84%|████████▍ | 4.17G/4.97G [00:06<00:01, 757MB/s]
model-00002-of-00006.safetensors: 86%|████████▋ | 4.30G/4.97G [00:06<00:00, 845MB/s]
model-00002-of-00006.safetensors: 89%|████████▉ | 4.45G/4.97G [00:06<00:00, 942MB/s]
model-00002-of-00006.safetensors: 92%|█████████▏| 4.59G/4.97G [00:07<00:00, 1.06GB/s]
model-00002-of-00006.safetensors: 95%|█████████▌| 4.74G/4.97G [00:07<00:00, 1.15GB/s]
model-00002-of-00006.safetensors: 100%|█████████▉| 4.97G/4.97G [00:07<00:00, 951MB/s]
model-00002-of-00006.safetensors: 100%|█████████▉| 4.97G/4.97G [00:07<00:00, 659MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model-00004-of-00006.safetensors: 0%| | 0.00/4.93G [00:00<?, ?B/s]
model-00004-of-00006.safetensors: 0%| | 10.5M/4.93G [00:01<15:08, 5.42MB/s]
model-00004-of-00006.safetensors: 1%| | 41.9M/4.93G [00:02<03:06, 26.2MB/s]
model-00004-of-00006.safetensors: 1%|▏ | 62.9M/4.93G [00:02<02:32, 31.8MB/s]
model-00004-of-00006.safetensors: 2%|▏ | 105M/4.93G [00:02<01:13, 65.3MB/s]
model-00004-of-00006.safetensors: 3%|▎ | 168M/4.93G [00:02<00:37, 126MB/s]
model-00004-of-00006.safetensors: 6%|▌ | 273M/4.93G [00:02<00:19, 238MB/s]
model-00004-of-00006.safetensors: 7%|▋ | 325M/4.93G [00:03<00:16, 273MB/s]
model-00004-of-00006.safetensors: 8%|▊ | 388M/4.93G [00:03<00:13, 331MB/s]
model-00004-of-00006.safetensors: 10%|█ | 514M/4.93G [00:03<00:08, 515MB/s]
model-00004-of-00006.safetensors: 22%|██▏ | 1.08G/4.93G [00:03<00:02, 1.65GB/s]
model-00004-of-00006.safetensors: 27%|██▋ | 1.31G/4.93G [00:03<00:03, 931MB/s]
model-00004-of-00006.safetensors: 30%|███ | 1.49G/4.93G [00:04<00:07, 481MB/s]
model-00004-of-00006.safetensors: 33%|███▎ | 1.61G/4.93G [00:04<00:06, 480MB/s]
model-00004-of-00006.safetensors: 35%|███▌ | 1.73G/4.93G [00:05<00:05, 546MB/s]
model-00004-of-00006.safetensors: 37%|███▋ | 1.85G/4.93G [00:05<00:04, 620MB/s]
model-00004-of-00006.safetensors: 41%|████ | 2.00G/4.93G [00:05<00:03, 758MB/s]
model-00004-of-00006.safetensors: 48%|████▊ | 2.38G/4.93G [00:05<00:02, 1.26GB/s]
model-00004-of-00006.safetensors: 52%|█████▏ | 2.57G/4.93G [00:05<00:03, 762MB/s]
model-00004-of-00006.safetensors: 55%|█████▌ | 2.72G/4.93G [00:06<00:02, 744MB/s]
model-00004-of-00006.safetensors: 58%|█████▊ | 2.84G/4.93G [00:06<00:03, 665MB/s]
model-00004-of-00006.safetensors: 60%|█████▉ | 2.95G/4.93G [00:06<00:02, 673MB/s]
model-00004-of-00006.safetensors: 62%|██████▏ | 3.04G/4.93G [00:06<00:02, 709MB/s]
model-00004-of-00006.safetensors: 64%|██████▎ | 3.14G/4.93G [00:06<00:02, 728MB/s]
model-00004-of-00006.safetensors: 65%|██████▌ | 3.23G/4.93G [00:06<00:02, 768MB/s]
model-00004-of-00006.safetensors: 70%|██████▉ | 3.45G/4.93G [00:06<00:01, 1.06GB/s]
model-00004-of-00006.safetensors: 72%|███████▏ | 3.58G/4.93G [00:07<00:01, 883MB/s]
model-00004-of-00006.safetensors: 76%|███████▌ | 3.75G/4.93G [00:07<00:01, 1.06GB/s]
model-00004-of-00006.safetensors: 79%|███████▊ | 3.88G/4.93G [00:07<00:01, 861MB/s]
model-00004-of-00006.safetensors: 81%|████████ | 3.98G/4.93G [00:07<00:01, 865MB/s]
model-00004-of-00006.safetensors: 83%|████████▎ | 4.09G/4.93G [00:07<00:01, 843MB/s]
model-00004-of-00006.safetensors: 85%|████████▍ | 4.18G/4.93G [00:07<00:00, 842MB/s]
model-00004-of-00006.safetensors: 87%|████████▋ | 4.28G/4.93G [00:08<00:01, 638MB/s]
model-00004-of-00006.safetensors: 91%|█████████ | 4.47G/4.93G [00:08<00:00, 879MB/s]
model-00004-of-00006.safetensors: 94%|█████████▍| 4.66G/4.93G [00:08<00:00, 1.08GB/s]
model-00004-of-00006.safetensors: 99%|█████████▉| 4.89G/4.93G [00:08<00:00, 1.12GB/s]
model-00004-of-00006.safetensors: 100%|█████████▉| 4.93G/4.93G [00:09<00:00, 543MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model-00005-of-00006.safetensors: 0%| | 0.00/4.93G [00:00<?, ?B/s]
model-00005-of-00006.safetensors: 0%| | 10.5M/4.93G [00:01<10:33, 7.78MB/s]
model-00005-of-00006.safetensors: 0%| | 21.0M/4.93G [00:01<05:59, 13.7MB/s]
model-00005-of-00006.safetensors: 1%| | 41.9M/4.93G [00:01<02:43, 29.9MB/s]
model-00005-of-00006.safetensors: 1%|▏ | 62.9M/4.93G [00:02<01:45, 46.4MB/s]
model-00005-of-00006.safetensors: 2%|▏ | 83.9M/4.93G [00:02<01:14, 65.2MB/s]
model-00005-of-00006.safetensors: 3%|▎ | 157M/4.93G [00:02<00:28, 167MB/s]
model-00005-of-00006.safetensors: 6%|▌ | 283M/4.93G [00:02<00:12, 359MB/s]
model-00005-of-00006.safetensors: 7%|▋ | 346M/4.93G [00:02<00:11, 394MB/s]
model-00005-of-00006.safetensors: 8%|▊ | 409M/4.93G [00:02<00:10, 427MB/s]
model-00005-of-00006.safetensors: 12%|█▏ | 577M/4.93G [00:02<00:06, 709MB/s]
model-00005-of-00006.safetensors: 23%|██▎ | 1.11G/4.93G [00:02<00:02, 1.84GB/s]
model-00005-of-00006.safetensors: 27%|██▋ | 1.34G/4.93G [00:03<00:02, 1.55GB/s]
model-00005-of-00006.safetensors: 31%|███ | 1.54G/4.93G [00:03<00:05, 576MB/s]
model-00005-of-00006.safetensors: 34%|███▍ | 1.69G/4.93G [00:04<00:05, 624MB/s]
model-00005-of-00006.safetensors: 38%|███▊ | 1.86G/4.93G [00:04<00:04, 739MB/s]
model-00005-of-00006.safetensors: 40%|████ | 1.99G/4.93G [00:04<00:03, 821MB/s]
model-00005-of-00006.safetensors: 43%|████▎ | 2.13G/4.93G [00:04<00:03, 858MB/s]
model-00005-of-00006.safetensors: 52%|█████▏ | 2.57G/4.93G [00:04<00:01, 1.45GB/s]
model-00005-of-00006.safetensors: 56%|█████▌ | 2.77G/4.93G [00:05<00:02, 849MB/s]
model-00005-of-00006.safetensors: 59%|█████▉ | 2.92G/4.93G [00:05<00:02, 819MB/s]
model-00005-of-00006.safetensors: 62%|██████▏ | 3.04G/4.93G [00:05<00:02, 795MB/s]
model-00005-of-00006.safetensors: 64%|██████▍ | 3.16G/4.93G [00:05<00:02, 788MB/s]
model-00005-of-00006.safetensors: 66%|██████▌ | 3.26G/4.93G [00:05<00:02, 770MB/s]
model-00005-of-00006.safetensors: 70%|███████ | 3.46G/4.93G [00:05<00:01, 990MB/s]
model-00005-of-00006.safetensors: 74%|███████▍ | 3.66G/4.93G [00:06<00:01, 1.18GB/s]
model-00005-of-00006.safetensors: 77%|███████▋ | 3.81G/4.93G [00:06<00:01, 820MB/s]
model-00005-of-00006.safetensors: 79%|███████▉ | 3.92G/4.93G [00:06<00:01, 869MB/s]
model-00005-of-00006.safetensors: 82%|████████▏ | 4.04G/4.93G [00:06<00:01, 805MB/s]
model-00005-of-00006.safetensors: 84%|████████▍ | 4.14G/4.93G [00:06<00:00, 832MB/s]
model-00005-of-00006.safetensors: 86%|████████▌ | 4.25G/4.93G [00:06<00:00, 759MB/s]
model-00005-of-00006.safetensors: 89%|████████▉ | 4.38G/4.93G [00:07<00:00, 886MB/s]
model-00005-of-00006.safetensors: 91%|█████████▏| 4.51G/4.93G [00:07<00:00, 954MB/s]
model-00005-of-00006.safetensors: 95%|█████████▍| 4.68G/4.93G [00:07<00:00, 1.09GB/s]
model-00005-of-00006.safetensors: 100%|█████████▉| 4.92G/4.93G [00:07<00:00, 1.33GB/s]
model-00005-of-00006.safetensors: 100%|█████████▉| 4.93G/4.93G [00:07<00:00, 658MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model-00006-of-00006.safetensors: 0%| | 0.00/1.25G [00:00<?, ?B/s]
model-00006-of-00006.safetensors: 1%| | 10.5M/1.25G [00:01<02:27, 8.39MB/s]
model-00006-of-00006.safetensors: 2%|▏ | 21.0M/1.25G [00:01<01:23, 14.7MB/s]
model-00006-of-00006.safetensors: 3%|▎ | 41.9M/1.25G [00:01<00:44, 27.1MB/s]
model-00006-of-00006.safetensors: 4%|▍ | 52.4M/1.25G [00:02<00:43, 27.7MB/s]
model-00006-of-00006.safetensors: 7%|▋ | 83.9M/1.25G [00:02<00:20, 57.6MB/s]
model-00006-of-00006.safetensors: 10%|█ | 126M/1.25G [00:02<00:10, 105MB/s]
model-00006-of-00006.safetensors: 21%|██ | 262M/1.25G [00:02<00:03, 301MB/s]
model-00006-of-00006.safetensors: 29%|██▉ | 367M/1.25G [00:02<00:02, 409MB/s]
model-00006-of-00006.safetensors: 35%|███▌ | 440M/1.25G [00:02<00:01, 452MB/s]
model-00006-of-00006.safetensors: 51%|█████ | 629M/1.25G [00:02<00:00, 753MB/s]
model-00006-of-00006.safetensors: 95%|█████████▍| 1.18G/1.25G [00:03<00:00, 1.85GB/s]
model-00006-of-00006.safetensors: 100%|█████████▉| 1.25G/1.25G [00:03<00:00, 353MB/s]
khanhnto-khanhnto-v38-mkmlizer:
model.safetensors.index.json: 0%| | 0.00/29.9k [00:00<?, ?B/s]
model.safetensors.index.json: 100%|██████████| 29.9k/29.9k [00:00<00:00, 3.46MB/s]
khanhnto-khanhnto-v38-mkmlizer:
special_tokens_map.json: 0%| | 0.00/548 [00:00<?, ?B/s]
special_tokens_map.json: 100%|██████████| 548/548 [00:00<00:00, 7.46MB/s]
khanhnto-khanhnto-v38-mkmlizer:
tokenizer.json: 0%| | 0.00/1.84M [00:00<?, ?B/s]
tokenizer.json: 100%|██████████| 1.84M/1.84M [00:00<00:00, 5.56MB/s]
tokenizer.json: 100%|██████████| 1.84M/1.84M [00:00<00:00, 5.55MB/s]
khanhnto-khanhnto-v38-mkmlizer:
tokenizer.model: 0%| | 0.00/500k [00:00<?, ?B/s]
tokenizer.model: 100%|██████████| 500k/500k [00:00<00:00, 1.95MB/s]
tokenizer.model: 100%|██████████| 500k/500k [00:00<00:00, 1.95MB/s]
khanhnto-khanhnto-v38-mkmlizer:
tokenizer_config.json: 0%| | 0.00/1.02k [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████| 1.02k/1.02k [00:00<00:00, 11.3MB/s]
khanhnto-khanhnto-v38-mkmlizer: Downloaded to shared memory in 50.199s
khanhnto-khanhnto-v38-mkmlizer: quantizing model to /dev/shm/model_cache
khanhnto-khanhnto-v38-mkmlizer: Saving mkml model at /dev/shm/model_cache
khanhnto-khanhnto-v38-mkmlizer: Reading /tmp/tmpwbl0nlkw/model.safetensors.index.json
khanhnto-khanhnto-v38-mkmlizer:
Profiling: 0%| | 0/363 [00:00<?, ?it/s]
Profiling: 0%| | 1/363 [00:02<12:46, 2.12s/it]
Profiling: 3%|▎ | 12/363 [00:02<00:47, 7.36it/s]
Profiling: 6%|▌ | 22/363 [00:02<00:22, 14.95it/s]
Profiling: 9%|▉ | 32/363 [00:02<00:13, 23.83it/s]
Profiling: 12%|█▏ | 45/363 [00:02<00:08, 37.45it/s]
Profiling: 16%|█▌ | 57/363 [00:02<00:06, 49.86it/s]
Profiling: 19%|█▉ | 69/363 [00:02<00:06, 43.38it/s]
Profiling: 22%|██▏ | 80/363 [00:03<00:05, 52.09it/s]
Profiling: 25%|██▍ | 90/363 [00:03<00:04, 60.34it/s]
Profiling: 28%|██▊ | 103/363 [00:03<00:03, 73.51it/s]
Profiling: 32%|███▏ | 115/363 [00:03<00:02, 83.17it/s]
Profiling: 35%|███▍ | 126/363 [00:03<00:02, 86.53it/s]
Profiling: 38%|███▊ | 138/363 [00:03<00:02, 94.18it/s]
Profiling: 41%|████ | 149/363 [00:03<00:03, 60.61it/s]
Profiling: 45%|████▍ | 162/363 [00:04<00:02, 72.63it/s]
Profiling: 48%|████▊ | 174/363 [00:04<00:02, 81.03it/s]
Profiling: 51%|█████ | 185/363 [00:04<00:02, 83.57it/s]
Profiling: 55%|█████▍ | 198/363 [00:04<00:01, 93.14it/s]
Profiling: 58%|█████▊ | 209/363 [00:04<00:02, 64.69it/s]
Profiling: 60%|██████ | 219/363 [00:04<00:02, 70.82it/s]
Profiling: 63%|██████▎ | 229/363 [00:04<00:01, 76.34it/s]
Profiling: 66%|██████▌ | 239/363 [00:05<00:01, 80.95it/s]
Profiling: 69%|██████▉ | 251/363 [00:05<00:01, 90.57it/s]
Profiling: 73%|███████▎ | 264/363 [00:05<00:01, 96.61it/s]
Profiling: 76%|███████▌ | 275/363 [00:05<00:00, 95.27it/s]
Profiling: 79%|███████▉ | 286/363 [00:05<00:01, 63.22it/s]
Profiling: 82%|████████▏ | 297/363 [00:05<00:00, 72.14it/s]
Profiling: 85%|████████▌ | 309/363 [00:05<00:00, 80.76it/s]
Profiling: 88%|████████▊ | 319/363 [00:05<00:00, 84.09it/s]
Profiling: 91%|█████████ | 329/363 [00:06<00:00, 87.37it/s]
Profiling: 94%|█████████▍| 341/363 [00:06<00:00, 95.82it/s]
Profiling: 97%|█████████▋| 352/363 [00:07<00:00, 18.03it/s]
Profiling: 100%|██████████| 363/363 [00:08<00:00, 44.70it/s]
khanhnto-khanhnto-v38-mkmlizer: quantized model in 28.699s
khanhnto-khanhnto-v38-mkmlizer: Processed model khanhnto/khanhnto in 80.509s
khanhnto-khanhnto-v38-mkmlizer: creating bucket guanaco-mkml-models
khanhnto-khanhnto-v38-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
khanhnto-khanhnto-v38-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/khanhnto-khanhnto-v38
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/khanhnto-khanhnto-v38/config.json
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/khanhnto-khanhnto-v38/added_tokens.json
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/khanhnto-khanhnto-v38/special_tokens_map.json
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/khanhnto-khanhnto-v38/tokenizer_config.json
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/khanhnto-khanhnto-v38/tokenizer.model
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/khanhnto-khanhnto-v38/tokenizer.json
khanhnto-khanhnto-v38-mkmlizer: cp /dev/shm/model_cache/mkml_model.tensors s3://guanaco-mkml-models/khanhnto-khanhnto-v38/mkml_model.tensors
khanhnto-khanhnto-v38-mkmlizer: loading reward model from ChaiML/reward_models_100_170000000_cp_332032
khanhnto-khanhnto-v38-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1067: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
khanhnto-khanhnto-v38-mkmlizer: warnings.warn(
khanhnto-khanhnto-v38-mkmlizer:
config.json: 0%| | 0.00/1.06k [00:00<?, ?B/s]
config.json: 100%|██████████| 1.06k/1.06k [00:00<00:00, 10.9MB/s]
khanhnto-khanhnto-v38-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:690: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
khanhnto-khanhnto-v38-mkmlizer: warnings.warn(
khanhnto-khanhnto-v38-mkmlizer:
tokenizer_config.json: 0%| | 0.00/234 [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████| 234/234 [00:00<00:00, 2.58MB/s]
khanhnto-khanhnto-v38-mkmlizer:
vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s]
vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 2.45MB/s]
vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 2.45MB/s]
khanhnto-khanhnto-v38-mkmlizer:
tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s]
tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 13.9MB/s]
tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 13.8MB/s]
khanhnto-khanhnto-v38-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
khanhnto-khanhnto-v38-mkmlizer: warnings.warn(
khanhnto-khanhnto-v38-mkmlizer:
pytorch_model.bin: 0%| | 0.00/510M [00:00<?, ?B/s]
pytorch_model.bin: 2%|▏ | 10.5M/510M [00:01<01:10, 7.07MB/s]
pytorch_model.bin: 4%|▍ | 21.0M/510M [00:01<00:36, 13.5MB/s]
pytorch_model.bin: 12%|█▏ | 62.9M/510M [00:01<00:08, 52.2MB/s]
pytorch_model.bin: 33%|███▎ | 168M/510M [00:01<00:01, 172MB/s]
pytorch_model.bin: 53%|█████▎ | 273M/510M [00:02<00:00, 300MB/s]
pytorch_model.bin: 98%|█████████▊| 500M/510M [00:17<00:00, 300MB/s]
pytorch_model.bin: 100%|█████████▉| 510M/510M [00:17<00:00, 24.5MB/s]
pytorch_model.bin: 100%|█████████▉| 510M/510M [00:17<00:00, 29.7MB/s]
khanhnto-khanhnto-v38-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
khanhnto-khanhnto-v38-mkmlizer: Saving duration: 0.116s
khanhnto-khanhnto-v38-mkmlizer: Processed model ChaiML/reward_models_100_170000000_cp_332032 in 21.959s
khanhnto-khanhnto-v38-mkmlizer: creating bucket guanaco-reward-models
khanhnto-khanhnto-v38-mkmlizer: Bucket 's3://guanaco-reward-models/' created
khanhnto-khanhnto-v38-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/config.json
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/special_tokens_map.json
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/tokenizer_config.json
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/merges.txt
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/vocab.json
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/tokenizer.json
khanhnto-khanhnto-v38-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/khanhnto-khanhnto-v38_reward/reward.tensors
Job khanhnto-khanhnto-v38-mkmlizer completed after 139.41s with status: succeeded
Stopping job with name khanhnto-khanhnto-v38-mkmlizer
Pipeline stage MKMLizer completed in 145.16s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.12s
Running pipeline stage ISVCDeployer
Creating inference service khanhnto-khanhnto-v38
Waiting for inference service khanhnto-khanhnto-v38 to be ready
Inference service khanhnto-khanhnto-v38 ready after 50.28345513343811s
Pipeline stage ISVCDeployer completed in 58.44s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.326432228088379s
Received healthy response to inference request in 1.043257236480713s
Received healthy response to inference request in 1.5198009014129639s
Received healthy response to inference request in 1.6462793350219727s
Received healthy response to inference request in 2.7234907150268555s
5 requests
0 failed requests
5th percentile: 1.138565969467163
10th percentile: 1.2338747024536132
20th percentile: 1.4244921684265137
30th percentile: 1.5450965881347656
40th percentile: 1.5956879615783692
50th percentile: 1.6462793350219727
60th percentile: 1.918340492248535
70th percentile: 2.1904016494750973
80th percentile: 2.4058439254760744
90th percentile: 2.564667320251465
95th percentile: 2.64407901763916
99th percentile: 2.7076083755493165
mean time: 1.8518520832061767
Pipeline stage StressChecker completed in 10.17s
Running pipeline stage DaemonicModelEvalScorer
Pipeline stage DaemonicModelEvalScorer completed in 0.04s
Running pipeline stage DaemonicSafetyScorer
Pipeline stage DaemonicSafetyScorer completed in 0.06s
Running M-Eval for topic stay_in_character
M-Eval Dataset for topic stay_in_character is loaded
khanhnto-khanhnto_v38 status is now inactive due to auto deactivation removed underperforming models
khanhnto-khanhnto_v38 status is now deployed due to admin request
khanhnto-khanhnto_v38 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of khanhnto-khanhnto_v38
Running pipeline stage ISVCDeleter
Checking if service khanhnto-khanhnto-v38 is running
Tearing down inference service khanhnto-khanhnto-v38
Toredown service khanhnto-khanhnto-v38
Pipeline stage ISVCDeleter completed in 4.59s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key khanhnto-khanhnto-v38/added_tokens.json from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/config.json from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/mkml_model.tensors from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/tokenizer.json from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/tokenizer.model from bucket guanaco-mkml-models
Deleting key khanhnto-khanhnto-v38/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key khanhnto-khanhnto-v38_reward/config.json from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/merges.txt from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/reward.tensors from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key khanhnto-khanhnto-v38_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 3.03s
admin requested tearing down of khanhnto-khanhnto_v49
khanhnto-khanhnto_v38 status is now torndown due to DeploymentManager action
Running pipeline stage ISVCDeleter