Running pipeline stage MKMLizer
Starting job with name thanhdaonguyen-once-upon-a-time-mkmlizer
Waiting for job on thanhdaonguyen-once-upon-a-time-mkmlizer to finish
thanhdaonguyen-once-upon-a-time-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ _______ __ __ _______ _____ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ | | | |/ | | | |_ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ | | <| | | ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ |__|_|__|__|\__|__|_|__|_______| ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ The license key for the current software has been verified as ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ belonging to: ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ Chai Research Corp ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ Expiration: 2024-01-08 23:59:59 ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ║ ║
thanhdaonguyen-once-upon-a-time-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
thanhdaonguyen-once-upon-a-time-mkmlizer: loading model from thanhdaonguyen/once-upon-a-time
thanhdaonguyen-once-upon-a-time-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1067: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
thanhdaonguyen-once-upon-a-time-mkmlizer: warnings.warn(
thanhdaonguyen-once-upon-a-time-mkmlizer:
config.json: 0%| | 0.00/643 [00:00<?, ?B/s]
config.json: 100%|██████████| 643/643 [00:00<00:00, 8.07MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:690: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
thanhdaonguyen-once-upon-a-time-mkmlizer: warnings.warn(
thanhdaonguyen-once-upon-a-time-mkmlizer:
tokenizer_config.json: 0%| | 0.00/828 [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████| 828/828 [00:00<00:00, 6.74MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
tokenizer.model: 0%| | 0.00/500k [00:00<?, ?B/s]
tokenizer.model: 100%|██████████| 500k/500k [00:00<00:00, 16.9MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
tokenizer.json: 0%| | 0.00/1.84M [00:00<?, ?B/s]
tokenizer.json: 100%|██████████| 1.84M/1.84M [00:00<00:00, 41.5MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
added_tokens.json: 0%| | 0.00/21.0 [00:00<?, ?B/s]
added_tokens.json: 100%|██████████| 21.0/21.0 [00:00<00:00, 274kB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
special_tokens_map.json: 0%| | 0.00/438 [00:00<?, ?B/s]
special_tokens_map.json: 100%|██████████| 438/438 [00:00<00:00, 5.01MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 0%| | 0.00/9.95G [00:00<?, ?B/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 0%| | 10.5M/9.95G [00:00<13:03, 12.7MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 0%| | 21.0M/9.95G [00:01<13:56, 11.9MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 0%| | 31.5M/9.95G [00:02<11:59, 13.8MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 0%| | 41.9M/9.95G [00:02<08:00, 20.6MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 1%| | 52.4M/9.95G [00:02<07:02, 23.4MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 1%|▏ | 147M/9.95G [00:02<01:27, 112MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 2%|▏ | 178M/9.95G [00:03<01:13, 133MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 3%|▎ | 283M/9.95G [00:03<00:36, 262MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 9%|▊ | 849M/9.95G [00:03<00:07, 1.18GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 12%|█▏ | 1.24G/9.95G [00:03<00:05, 1.65GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 15%|█▍ | 1.49G/9.95G [00:04<00:15, 542MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 17%|█▋ | 1.67G/9.95G [00:05<00:16, 495MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 18%|█▊ | 1.80G/9.95G [00:05<00:16, 484MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 19%|█▉ | 1.92G/9.95G [00:05<00:15, 514MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 20%|██ | 2.02G/9.95G [00:05<00:14, 557MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 21%|██▏ | 2.12G/9.95G [00:06<00:16, 463MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 22%|██▏ | 2.20G/9.95G [00:06<00:15, 503MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 24%|██▍ | 2.39G/9.95G [00:06<00:10, 706MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 28%|██▊ | 2.81G/9.95G [00:06<00:05, 1.31GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 32%|███▏ | 3.14G/9.95G [00:06<00:04, 1.69GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 34%|███▍ | 3.38G/9.95G [00:06<00:05, 1.13GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 36%|███▌ | 3.57G/9.95G [00:07<00:06, 999MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 37%|███▋ | 3.72G/9.95G [00:07<00:06, 981MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 39%|███▉ | 3.86G/9.95G [00:07<00:06, 980MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 40%|████ | 3.98G/9.95G [00:07<00:07, 821MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 41%|████ | 4.09G/9.95G [00:07<00:07, 794MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 42%|████▏ | 4.19G/9.95G [00:07<00:06, 827MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 44%|████▎ | 4.35G/9.95G [00:08<00:05, 977MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 45%|████▌ | 4.48G/9.95G [00:08<00:05, 1.02GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 47%|████▋ | 4.70G/9.95G [00:08<00:04, 1.29GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 50%|█████ | 4.99G/9.95G [00:08<00:02, 1.70GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 52%|█████▏ | 5.18G/9.95G [00:08<00:02, 1.69GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 54%|█████▍ | 5.37G/9.95G [00:08<00:03, 1.41GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 56%|█████▌ | 5.53G/9.95G [00:08<00:03, 1.13GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 57%|█████▋ | 5.66G/9.95G [00:09<00:04, 969MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 58%|█████▊ | 5.78G/9.95G [00:09<00:04, 983MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 59%|█████▉ | 5.89G/9.95G [00:09<00:04, 957MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 61%|██████ | 6.08G/9.95G [00:09<00:03, 1.14GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 62%|██████▏ | 6.21G/9.95G [00:09<00:03, 1.03GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 64%|██████▍ | 6.35G/9.95G [00:09<00:03, 1.11GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 66%|██████▌ | 6.52G/9.95G [00:09<00:02, 1.22GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 67%|██████▋ | 6.66G/9.95G [00:09<00:02, 1.22GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 68%|██████▊ | 6.79G/9.95G [00:10<00:02, 1.16GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 70%|███████ | 6.98G/9.95G [00:10<00:02, 1.33GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 96%|█████████▌| 9.56G/9.95G [00:12<00:00, 1.72GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00001-of-00003.bin: 98%|█████████▊| 9.78G/9.95G [00:12<00:00, 1.69GB/s][A
pytorch_model-00001-of-00003.bin: 100%|█████████▉| 9.95G/9.95G [00:12<00:00, 786MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
Downloading shards: 33%|███▎ | 1/3 [00:12<00:25, 12.91s/it]
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 0%| | 0.00/9.90G [00:00<?, ?B/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 0%| | 10.5M/9.90G [00:02<37:19, 4.42MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 0%| | 41.9M/9.90G [00:02<09:39, 17.0MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 1%| | 83.9M/9.90G [00:03<04:02, 40.5MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 4%|▍ | 440M/9.90G [00:03<00:30, 313MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 12%|█▏ | 1.15G/9.90G [00:03<00:09, 945MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 14%|█▍ | 1.43G/9.90G [00:05<00:22, 385MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 16%|█▋ | 1.61G/9.90G [00:05<00:18, 439MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 18%|█▊ | 1.80G/9.90G [00:05<00:15, 511MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 20%|█▉ | 1.95G/9.90G [00:05<00:17, 452MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 22%|██▏ | 2.18G/9.90G [00:06<00:12, 608MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 26%|██▌ | 2.57G/9.90G [00:06<00:07, 958MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 28%|██▊ | 2.79G/9.90G [00:06<00:07, 994MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 30%|███ | 2.98G/9.90G [00:06<00:06, 1.08GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 32%|███▏ | 3.20G/9.90G [00:06<00:05, 1.25GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 34%|███▍ | 3.39G/9.90G [00:06<00:04, 1.33GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 36%|███▌ | 3.59G/9.90G [00:06<00:04, 1.43GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 38%|███▊ | 3.76G/9.90G [00:06<00:04, 1.42GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 40%|███▉ | 3.93G/9.90G [00:07<00:04, 1.44GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 42%|████▏ | 4.16G/9.90G [00:07<00:03, 1.45GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 44%|████▎ | 4.32G/9.90G [00:07<00:04, 1.19GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 45%|████▍ | 4.46G/9.90G [00:07<00:04, 1.20GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 46%|████▋ | 4.60G/9.90G [00:07<00:04, 1.25GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 48%|████▊ | 4.74G/9.90G [00:07<00:04, 1.18GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 49%|████▉ | 4.87G/9.90G [00:07<00:04, 1.15GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 51%|█████▏ | 5.10G/9.90G [00:08<00:03, 1.33GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 53%|█████▎ | 5.23G/9.90G [00:08<00:03, 1.26GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 54%|█████▍ | 5.37G/9.90G [00:08<00:03, 1.22GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 56%|█████▌ | 5.51G/9.90G [00:08<00:03, 1.25GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 58%|█████▊ | 5.71G/9.90G [00:08<00:02, 1.44GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 59%|█████▉ | 5.86G/9.90G [00:08<00:03, 1.13GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 61%|██████ | 6.00G/9.90G [00:08<00:03, 1.15GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 62%|██████▏ | 6.19G/9.90G [00:08<00:02, 1.32GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 65%|██████▍ | 6.40G/9.90G [00:09<00:02, 1.50GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 88%|████████▊ | 8.75G/9.90G [00:10<00:00, 1.29GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 90%|████████▉ | 8.88G/9.90G [00:10<00:00, 1.30GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 91%|█████████ | 9.02G/9.90G [00:10<00:00, 1.30GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 93%|█████████▎| 9.19G/9.90G [00:11<00:00, 1.40GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 95%|█████████▌| 9.46G/9.90G [00:11<00:00, 1.77GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00002-of-00003.bin: 98%|█████████▊| 9.73G/9.90G [00:11<00:00, 1.84GB/s][A
pytorch_model-00002-of-00003.bin: 100%|█████████▉| 9.90G/9.90G [00:12<00:00, 815MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
Downloading shards: 67%|██████▋ | 2/3 [00:25<00:12, 12.65s/it]
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 0%| | 0.00/6.18G [00:00<?, ?B/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 0%| | 10.5M/6.18G [00:02<23:09, 4.44MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 0%| | 21.0M/6.18G [00:03<13:44, 7.47MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 3%|▎ | 199M/6.18G [00:03<00:55, 108MB/s] [A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 6%|▋ | 398M/6.18G [00:03<00:23, 248MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 18%|█▊ | 1.11G/6.18G [00:03<00:05, 868MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 22%|██▏ | 1.37G/6.18G [00:04<00:07, 677MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 25%|██▌ | 1.56G/6.18G [00:04<00:06, 686MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 28%|██▊ | 1.71G/6.18G [00:04<00:07, 613MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 30%|██▉ | 1.82G/6.18G [00:04<00:06, 625MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 31%|███ | 1.93G/6.18G [00:04<00:06, 657MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 34%|███▍ | 2.11G/6.18G [00:05<00:05, 785MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 36%|███▌ | 2.22G/6.18G [00:05<00:04, 824MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 39%|███▊ | 2.38G/6.18G [00:05<00:04, 875MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 40%|████ | 2.49G/6.18G [00:05<00:06, 601MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 42%|████▏ | 2.62G/6.18G [00:05<00:05, 708MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 45%|████▍ | 2.78G/6.18G [00:05<00:03, 859MB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 50%|█████ | 3.09G/6.18G [00:06<00:02, 1.32GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 53%|█████▎ | 3.27G/6.18G [00:06<00:02, 1.41GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 56%|█████▌ | 3.47G/6.18G [00:06<00:01, 1.54GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 60%|██████ | 3.71G/6.18G [00:06<00:01, 1.75GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 63%|██████▎ | 3.91G/6.18G [00:06<00:01, 1.60GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 66%|██████▌ | 4.09G/6.18G [00:06<00:01, 1.20GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 69%|██████▊ | 4.24G/6.18G [00:06<00:01, 1.10GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 71%|███████ | 4.37G/6.18G [00:07<00:01, 1.09GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 73%|███████▎ | 4.50G/6.18G [00:07<00:01, 1.07GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 76%|███████▌ | 4.67G/6.18G [00:07<00:01, 1.20GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 78%|███████▊ | 4.83G/6.18G [00:07<00:01, 1.31GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 82%|████████▏ | 5.04G/6.18G [00:07<00:00, 1.51GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 85%|████████▌ | 5.27G/6.18G [00:07<00:00, 1.70GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 89%|████████▉ | 5.49G/6.18G [00:07<00:00, 1.83GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 92%|█████████▏| 5.69G/6.18G [00:07<00:00, 1.67GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model-00003-of-00003.bin: 95%|█████████▌| 5.87G/6.18G [00:07<00:00, 1.58GB/s][A
thanhdaonguyen-once-upon-a-time-mkmlizer:
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:05<00:10, 5.39s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:10<00:05, 5.39s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:14<00:00, 4.45s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:14<00:00, 4.71s/it]
thanhdaonguyen-once-upon-a-time-mkmlizer:
generation_config.json: 0%| | 0.00/154 [00:00<?, ?B/s]
generation_config.json: 100%|██████████| 154/154 [00:00<00:00, 1.89MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: loaded model in 50.046s
thanhdaonguyen-once-upon-a-time-mkmlizer: saved to disk in 115.582s
thanhdaonguyen-once-upon-a-time-mkmlizer: quantizing model to /tmp/model_cache
thanhdaonguyen-once-upon-a-time-mkmlizer: Saving mkml model at /tmp/model_cache
thanhdaonguyen-once-upon-a-time-mkmlizer: Reading /tmp/tmppaiajb2x/model.safetensors.index.json
thanhdaonguyen-once-upon-a-time-mkmlizer:
Profiling: 0%| | 0/363 [00:00<?, ?it/s]
Profiling: 0%| | 1/363 [00:02<16:51, 2.79s/it]
Profiling: 1%| | 3/363 [00:04<07:22, 1.23s/it]
Profiling: 1%| | 4/363 [00:05<07:28, 1.25s/it]
Profiling: 1%|▏ | 5/363 [00:06<07:38, 1.28s/it]
Profiling: 2%|▏ | 7/363 [00:07<04:51, 1.22it/s]
Profiling: 2%|▏ | 8/363 [00:08<04:33, 1.30it/s]
Profiling: 2%|▏ | 9/363 [00:08<04:23, 1.35it/s]
Profiling: 3%|▎ | 10/363 [00:09<04:04, 1.45it/s]
Profiling: 3%|▎ | 12/363 [00:10<03:56, 1.48it/s]
Profiling: 4%|▎ | 13/363 [00:12<04:59, 1.17it/s]
Profiling: 4%|▍ | 14/363 [00:13<05:46, 1.01it/s]
Profiling: 4%|▍ | 16/363 [00:14<04:04, 1.42it/s]
Profiling: 5%|▍ | 17/363 [00:14<03:53, 1.48it/s]
Profiling: 5%|▍ | 18/363 [00:15<03:43, 1.55it/s]
Profiling: 5%|▌ | 19/363 [00:15<03:37, 1.58it/s]
Profiling: 6%|▌ | 21/363 [00:19<06:09, 1.08s/it]
Profiling: 6%|▌ | 22/363 [00:20<06:26, 1.13s/it]
Profiling: 6%|▋ | 23/363 [00:21<06:41, 1.18s/it]
Profiling: 7%|▋ | 25/363 [00:22<04:36, 1.22it/s]
Profiling: 7%|▋ | 26/363 [00:22<04:17, 1.31it/s]
Profiling: 7%|▋ | 27/363 [00:23<03:54, 1.43it/s]
Profiling: 8%|▊ | 28/363 [00:24<03:41, 1.51it/s]
Profiling: 8%|▊ | 29/363 [00:24<03:30, 1.59it/s]
Profiling: 8%|▊ | 30/363 [00:25<03:22, 1.64it/s]
Profiling: 9%|▊ | 31/363 [00:25<03:11, 1.73it/s]
Profiling: 9%|▉ | 32/363 [00:26<03:04, 1.80it/s]
Profiling: 9%|▉ | 33/363 [00:26<02:21, 2.33it/s]
Profiling: 9%|▉ | 34/363 [00:27<03:43, 1.47it/s]
Profiling: 10%|▉ | 35/363 [00:28<04:47, 1.14it/s]
Profiling: 10%|▉ | 36/363 [00:30<05:23, 1.01it/s]
Profiling: 11%|█ | 39/363 [00:31<03:30, 1.54it/s]
Profiling: 11%|█ | 40/363 [00:32<04:03, 1.33it/s]
Profiling: 11%|█▏ | 41/363 [00:33<04:47, 1.12it/s]
Profiling: 12%|█▏ | 43/363 [00:34<03:28, 1.53it/s]
Profiling: 12%|█▏ | 44/363 [00:34<03:24, 1.56it/s]
Profiling: 12%|█▏ | 45/363 [00:35<03:20, 1.59it/s]
Profiling: 13%|█▎ | 46/363 [00:36<03:14, 1.63it/s]
Profiling: 13%|█▎ | 48/363 [00:37<03:30, 1.50it/s]
Profiling: 13%|█▎ | 49/363 [00:38<04:17, 1.22it/s]
Profiling: 14%|█▍ | 50/363 [00:40<04:50, 1.08it/s]
Profiling: 14%|█▍ | 52/363 [00:40<03:21, 1.54it/s]
Profiling: 15%|█▍ | 53/363 [00:41<03:07, 1.66it/s]
Profiling: 15%|█▍ | 54/363 [00:41<02:54, 1.77it/s]
Profiling: 15%|█▌ | 55/363 [00:42<02:44, 1.87it/s]
Profiling: 16%|█▌ | 57/363 [00:43<02:56, 1.73it/s]
Profiling: 16%|█▌ | 58/363 [00:44<03:46, 1.35it/s]
Profiling: 16%|█▋ | 59/363 [00:45<04:32, 1.12it/s]
Profiling: 17%|█▋ | 61/363 [00:46<03:06, 1.62it/s]
Profiling: 17%|█▋ | 62/363 [00:46<02:54, 1.73it/s]
Profiling: 17%|█▋ | 63/363 [00:47<02:43, 1.84it/s]
Profiling: 18%|█▊ | 64/363 [00:47<02:38, 1.89it/s]
Profiling: 18%|█▊ | 65/363 [00:48<02:35, 1.91it/s]
Profiling: 18%|█▊ | 66/363 [00:48<02:31, 1.96it/s]
Profiling: 18%|█▊ | 67/363 [00:49<02:28, 1.99it/s]
Profiling: 19%|█▊ | 68/363 [00:49<01:59, 2.48it/s]
Profiling: 19%|█▉ | 69/363 [00:50<03:14, 1.51it/s]
Profiling: 19%|█▉ | 70/363 [00:51<04:05, 1.19it/s]
Profiling: 20%|█▉ | 71/363 [00:53<04:42, 1.03it/s]
Profiling: 20%|██ | 73/363 [00:53<03:02, 1.59it/s]
Profiling: 20%|██ | 74/363 [00:54<02:49, 1.70it/s]
Profiling: 21%|██ | 75/363 [00:54<02:39, 1.80it/s]
Profiling: 21%|██ | 76/363 [00:55<02:36, 1.83it/s]
Profiling: 21%|██ | 77/363 [00:55<02:34, 1.85it/s]
Profiling: 21%|██▏ | 78/363 [00:56<02:33, 1.86it/s]
Profiling: 22%|██▏ | 80/363 [00:57<02:46, 1.70it/s]
Profiling: 22%|██▏ | 81/363 [00:58<03:27, 1.36it/s]
Profiling: 23%|██▎ | 82/363 [00:59<04:00, 1.17it/s]
Profiling: 23%|██▎ | 84/363 [01:00<02:43, 1.70it/s]
Profiling: 24%|██▎ | 86/363 [01:01<02:44, 1.69it/s]
Profiling: 24%|██▍ | 87/363 [01:02<03:22, 1.36it/s]
Profiling: 24%|██▍ | 88/363 [01:04<03:54, 1.17it/s]
Profiling: 25%|██▍ | 90/363 [01:04<02:46, 1.64it/s]
Profiling: 25%|██▌ | 91/363 [01:04<02:36, 1.74it/s]
Profiling: 25%|██▌ | 92/363 [01:05<02:27, 1.83it/s]
Profiling: 26%|██▌ | 93/363 [01:05<02:20, 1.92it/s]
Profiling: 26%|██▌ | 95/363 [01:07<02:29, 1.79it/s]
Profiling: 26%|██▋ | 96/363 [01:08<03:07, 1.42it/s]
Profiling: 27%|██▋ | 97/363 [01:09<03:43, 1.19it/s]
Profiling: 27%|██▋ | 99/363 [01:10<02:37, 1.68it/s]
Profiling: 28%|██▊ | 100/363 [01:10<02:27, 1.78it/s]
Profiling: 28%|██▊ | 101/363 [01:10<02:19, 1.88it/s]
Profiling: 28%|██▊ | 102/363 [01:11<02:13, 1.95it/s]
Profiling: 28%|██▊ | 103/363 [01:11<01:45, 2.47it/s]
Profiling: 29%|██▊ | 104/363 [01:12<02:45, 1.57it/s]
Profiling: 29%|██▉ | 105/363 [01:13<03:24, 1.26it/s]
Profiling: 29%|██▉ | 106/363 [01:15<03:52, 1.10it/s]
Profiling: 30%|██▉ | 108/363 [01:15<02:31, 1.68it/s]
Profiling: 30%|███ | 109/363 [01:15<02:21, 1.80it/s]
Profiling: 31%|███ | 111/363 [01:17<02:24, 1.74it/s]
Profiling: 31%|███ | 112/363 [01:18<03:00, 1.39it/s]
Profiling: 31%|███ | 113/363 [01:19<03:31, 1.18it/s]
Profiling: 32%|███▏ | 115/363 [01:20<02:26, 1.70it/s]
Profiling: 32%|███▏ | 116/363 [01:20<02:17, 1.80it/s]
Profiling: 32%|███▏ | 117/363 [01:21<02:14, 1.83it/s]
Profiling: 33%|███▎ | 118/363 [01:21<02:09, 1.89it/s]
Profiling: 33%|███▎ | 120/363 [01:22<02:18, 1.76it/s]
Profiling: 33%|███▎ | 121/363 [01:23<02:53, 1.39it/s]
Profiling: 34%|███▎ | 122/363 [01:25<03:24, 1.18it/s]
Profiling: 34%|███▍ | 124/363 [01:25<02:23, 1.66it/s]
Profiling: 34%|███▍ | 125/363 [01:26<02:18, 1.72it/s]
Profiling: 35%|███▍ | 126/363 [01:26<02:13, 1.78it/s]
Profiling: 35%|███▍ | 127/363 [01:27<02:09, 1.83it/s]
Profiling: 36%|███▌ | 129/363 [01:28<02:15, 1.72it/s]
Profiling: 36%|███▌ | 130/363 [01:29<02:50, 1.36it/s]
Profiling: 36%|███▌ | 131/363 [01:30<03:19, 1.16it/s]
Profiling: 37%|███▋ | 133/363 [01:31<02:19, 1.65it/s]
Profiling: 37%|███▋ | 134/363 [01:31<02:11, 1.75it/s]
Profiling: 37%|███▋ | 135/363 [01:32<02:06, 1.80it/s]
Profiling: 37%|███▋ | 136/363 [01:32<02:03, 1.84it/s]
Profiling: 38%|███▊ | 137/363 [01:33<02:01, 1.87it/s]
Profiling: 38%|███▊ | 138/363 [01:33<01:33, 2.40it/s]
Profiling: 38%|███▊ | 139/363 [01:34<02:33, 1.46it/s]
Profiling: 39%|███▊ | 140/363 [01:36<03:05, 1.20it/s]
Profiling: 39%|███▉ | 141/363 [01:37<03:28, 1.06it/s]
Profiling: 39%|███▉ | 143/363 [01:37<02:17, 1.60it/s]
Profiling: 40%|███▉ | 144/363 [01:38<02:09, 1.69it/s]
Profiling: 40%|███▉ | 145/363 [01:38<02:04, 1.76it/s]
Profiling: 40%|████ | 147/363 [01:40<02:06, 1.71it/s]
Profiling: 41%|████ | 148/363 [01:41<02:36, 1.37it/s]
Profiling: 41%|████ | 149/363 [01:42<03:00, 1.19it/s]
Profiling: 42%|████▏ | 151/363 [01:42<02:06, 1.68it/s]
Profiling: 42%|████▏ | 152/363 [01:43<01:57, 1.79it/s]
Profiling: 42%|████▏ | 153/363 [01:43<01:53, 1.84it/s]
Profiling: 42%|████▏ | 154/363 [01:44<01:51, 1.87it/s]
Profiling: 43%|████▎ | 156/363 [01:45<01:55, 1.79it/s]
Profiling: 43%|████▎ | 157/363 [01:46<02:25, 1.42it/s]
Profiling: 44%|████▎ | 158/363 [01:47<02:50, 1.20it/s]
Profiling: 44%|████▍ | 160/363 [01:48<01:59, 1.69it/s]
Profiling: 44%|████▍ | 161/363 [01:48<01:54, 1.76it/s]
Profiling: 45%|████▍ | 162/363 [01:49<01:50, 1.82it/s]
Profiling: 45%|████▍ | 163/363 [01:49<01:47, 1.86it/s]
Profiling: 45%|████▌ | 165/363 [01:51<01:53, 1.75it/s]
Profiling: 46%|████▌ | 166/363 [01:52<02:21, 1.39it/s]
Profiling: 46%|████▌ | 167/363 [01:53<02:45, 1.18it/s]
Profiling: 47%|████▋ | 169/363 [01:54<01:55, 1.67it/s]
Profiling: 47%|████▋ | 170/363 [01:54<01:51, 1.73it/s]
Profiling: 47%|████▋ | 171/363 [01:55<01:47, 1.79it/s]
Profiling: 47%|████▋ | 172/363 [01:55<01:44, 1.84it/s]
Profiling: 48%|████▊ | 173/363 [01:55<01:21, 2.34it/s]
Profiling: 48%|████▊ | 174/363 [01:56<02:03, 1.53it/s]
Profiling: 48%|████▊ | 175/363 [01:58<02:33, 1.22it/s]
Profiling: 48%|████▊ | 176/363 [01:59<02:55, 1.06it/s]
Profiling: 49%|████▉ | 178/363 [01:59<01:56, 1.59it/s]
Profiling: 49%|████▉ | 179/363 [02:00<01:50, 1.67it/s]
Profiling: 50%|████▉ | 180/363 [02:00<01:45, 1.74it/s]
Profiling: 50%|████▉ | 181/363 [02:01<01:41, 1.79it/s]
Profiling: 50%|█████ | 183/363 [02:02<01:45, 1.70it/s]
Profiling: 51%|█████ | 184/363 [02:03<02:10, 1.37it/s]
Profiling: 51%|█████ | 185/363 [02:05<02:29, 1.19it/s]
Profiling: 52%|█████▏ | 187/363 [02:05<01:42, 1.72it/s]
Profiling: 52%|█████▏ | 188/363 [02:05<01:35, 1.83it/s]
Profiling: 52%|█████▏ | 189/363 [02:06<01:33, 1.86it/s]
Profiling: 52%|█████▏ | 190/363 [02:06<01:29, 1.93it/s]
Profiling: 53%|█████▎ | 192/363 [02:08<01:35, 1.79it/s]
Profiling: 53%|█████▎ | 193/363 [02:09<01:59, 1.42it/s]
Profiling: 53%|█████▎ | 194/363 [02:10<02:20, 1.21it/s]
Profiling: 54%|█████▍ | 196/363 [02:11<01:38, 1.70it/s]
Profiling: 54%|█████▍ | 197/363 [02:11<01:32, 1.80it/s]
Profiling: 55%|█████▍ | 198/363 [02:11<01:26, 1.90it/s]
Profiling: 55%|█████▍ | 199/363 [02:12<01:22, 1.98it/s]
Profiling: 55%|█████▌ | 200/363 [02:13<01:55, 1.42it/s]
Profiling: 55%|█████▌ | 201/363 [02:14<02:19, 1.16it/s]
Profiling: 56%|█████▌ | 202/363 [02:15<02:02, 1.32it/s]
Profiling: 56%|█████▌ | 203/363 [02:15<01:50, 1.45it/s]
Profiling: 56%|█████▌ | 204/363 [02:16<01:40, 1.58it/s]
Profiling: 56%|█████▋ | 205/363 [02:16<01:34, 1.67it/s]
Profiling: 57%|█████▋ | 207/363 [02:18<01:38, 1.59it/s]
Profiling: 58%|█████▊ | 210/363 [02:19<01:20, 1.91it/s]
Profiling: 58%|█████▊ | 211/363 [02:20<01:40, 1.51it/s]
Profiling: 58%|█████▊ | 212/363 [02:21<01:58, 1.27it/s]
Profiling: 59%|█████▉ | 214/363 [02:22<01:27, 1.71it/s]
Profiling: 59%|█████▉ | 215/363 [02:22<01:23, 1.76it/s]
Profiling: 60%|█████▉ | 216/363 [02:23<01:21, 1.81it/s]
Profiling: 60%|█████▉ | 217/363 [02:24<01:19, 1.84it/s]
Profiling: 60%|██████ | 219/363 [02:25<01:23, 1.73it/s]
Profiling: 61%|██████ | 220/363 [02:26<01:45, 1.35it/s]
Profiling: 61%|██████ | 221/363 [02:27<02:01, 1.17it/s]
Profiling: 61%|██████▏ | 223/363 [02:28<01:22, 1.69it/s]
Profiling: 62%|██████▏ | 224/363 [02:28<01:17, 1.80it/s]
Profiling: 62%|██████▏ | 225/363 [02:29<01:13, 1.87it/s]
Profiling: 62%|██████▏ | 226/363 [02:29<01:09, 1.96it/s]
Profiling: 63%|██████▎ | 228/363 [02:30<01:15, 1.79it/s]
Profiling: 63%|██████▎ | 229/363 [02:31<01:34, 1.41it/s]
Profiling: 63%|██████▎ | 230/363 [02:33<01:50, 1.21it/s]
Profiling: 64%|██████▍ | 232/363 [02:33<01:15, 1.75it/s]
Profiling: 64%|██████▍ | 233/363 [02:34<01:10, 1.85it/s]
Profiling: 64%|██████▍ | 234/363 [02:34<01:07, 1.91it/s]
Profiling: 65%|██████▍ | 235/363 [02:34<01:05, 1.97it/s]
Profiling: 65%|██████▌ | 236/363 [02:36<01:30, 1.40it/s]
Profiling: 65%|██████▌ | 237/363 [02:36<01:19, 1.58it/s]
Profiling: 66%|██████▌ | 238/363 [02:37<01:11, 1.74it/s]
Profiling: 66%|██████▌ | 239/363 [02:37<01:08, 1.81it/s]
Profiling: 66%|██████▌ | 240/363 [02:38<01:06, 1.86it/s]
Profiling: 66%|██████▋ | 241/363 [02:38<00:50, 2.44it/s]
Profiling: 67%|██████▋ | 242/363 [02:39<01:20, 1.51it/s]
Profiling: 67%|██████▋ | 243/363 [02:40<01:39, 1.20it/s]
Profiling: 68%|██████▊ | 246/363 [02:41<01:09, 1.67it/s]
Profiling: 68%|██████▊ | 247/363 [02:43<01:25, 1.35it/s]
Profiling: 68%|██████▊ | 248/363 [02:44<01:38, 1.17it/s]
Profiling: 69%|██████▉ | 250/363 [02:44<01:09, 1.64it/s]
Profiling: 69%|██████▉ | 251/363 [02:45<01:04, 1.74it/s]
Profiling: 69%|██████▉ | 252/363 [02:45<01:00, 1.83it/s]
Profiling: 70%|██████▉ | 253/363 [02:46<00:57, 1.91it/s]
Profiling: 70%|███████ | 255/363 [02:47<01:01, 1.77it/s]
Profiling: 71%|███████ | 256/363 [02:48<01:17, 1.38it/s]
Profiling: 71%|███████ | 257/363 [02:50<01:30, 1.17it/s]
Profiling: 71%|███████▏ | 259/363 [02:50<01:02, 1.66it/s]
Profiling: 72%|███████▏ | 260/363 [02:51<00:59, 1.73it/s]
Profiling: 72%|███████▏ | 261/363 [02:51<00:56, 1.79it/s]
Profiling: 72%|███████▏ | 262/363 [02:52<00:54, 1.85it/s]
Profiling: 73%|███████▎ | 264/363 [02:53<00:57, 1.73it/s]
Profiling: 73%|███████▎ | 265/363 [02:54<01:10, 1.38it/s]
Profiling: 73%|███████▎ | 266/363 [02:55<01:21, 1.19it/s]
Profiling: 74%|███████▍ | 268/363 [02:56<00:55, 1.70it/s]
Profiling: 74%|███████▍ | 269/363 [02:56<00:51, 1.82it/s]
Profiling: 74%|███████▍ | 270/363 [02:56<00:48, 1.92it/s]
Profiling: 75%|███████▍ | 271/363 [02:57<00:46, 2.00it/s]
Profiling: 75%|███████▍ | 272/363 [02:57<00:44, 2.06it/s]
Profiling: 75%|███████▌ | 273/363 [02:58<00:44, 2.04it/s]
Profiling: 75%|███████▌ | 274/363 [02:58<00:43, 2.03it/s]
Profiling: 76%|███████▌ | 275/363 [02:59<00:43, 2.03it/s]
Profiling: 76%|███████▌ | 276/363 [02:59<00:33, 2.63it/s]
Profiling: 76%|███████▋ | 277/363 [03:00<00:52, 1.63it/s]
Profiling: 77%|███████▋ | 278/363 [03:01<01:05, 1.30it/s]
Profiling: 77%|███████▋ | 279/363 [03:02<01:14, 1.13it/s]
Profiling: 78%|███████▊ | 282/363 [03:04<00:49, 1.65it/s]
Profiling: 78%|███████▊ | 283/363 [03:05<00:58, 1.37it/s]
Profiling: 78%|███████▊ | 284/363 [03:06<01:06, 1.20it/s]
Profiling: 79%|███████▉ | 286/363 [03:06<00:45, 1.70it/s]
Profiling: 79%|███████▉ | 287/363 [03:07<00:43, 1.75it/s]
Profiling: 79%|███████▉ | 288/363 [03:07<00:41, 1.80it/s]
Profiling: 80%|███████▉ | 289/363 [03:08<00:40, 1.83it/s]
Profiling: 80%|████████ | 291/363 [03:09<00:41, 1.72it/s]
Profiling: 80%|████████ | 292/363 [03:10<00:53, 1.34it/s]
Profiling: 81%|████████ | 293/363 [03:12<01:01, 1.13it/s]
Profi
thanhdaonguyen-once-upon-a-time-mkmlizer: ling: 81%|████████▏ | 295/363 [03:12<00:41, 1.64it/s]
Profiling: 82%|████████▏ | 296/363 [03:13<00:38, 1.74it/s]
Profiling: 82%|████████▏ | 297/363 [03:13<00:35, 1.84it/s]
Profiling: 82%|████████▏ | 298/363 [03:14<00:34, 1.86it/s]
Profiling: 83%|████████▎ | 300/363 [03:15<00:36, 1.73it/s]
Profiling: 83%|████████▎ | 301/363 [03:16<00:44, 1.38it/s]
Profiling: 83%|████████▎ | 302/363 [03:17<00:51, 1.19it/s]
Profiling: 84%|████████▎ | 304/363 [03:18<00:35, 1.67it/s]
Profiling: 84%|████████▍ | 305/363 [03:18<00:33, 1.72it/s]
Profiling: 84%|████████▍ | 306/363 [03:19<00:32, 1.78it/s]
Profiling: 85%|████████▍ | 307/363 [03:19<00:29, 1.87it/s]
Profiling: 85%|████████▍ | 308/363 [03:20<00:29, 1.88it/s]
Profiling: 85%|████████▌ | 309/363 [03:20<00:28, 1.89it/s]
Profiling: 85%|████████▌ | 310/363 [03:21<00:29, 1.80it/s]
Profiling: 86%|████████▌ | 311/363 [03:21<00:23, 2.26it/s]
Profiling: 86%|████████▌ | 312/363 [03:22<00:35, 1.44it/s]
Profiling: 86%|████████▌ | 313/363 [03:24<00:43, 1.16it/s]
Profiling: 87%|████████▋ | 314/363 [03:25<00:48, 1.02it/s]
Profiling: 87%|████████▋ | 316/363 [03:26<00:30, 1.54it/s]
Profiling: 88%|████████▊ | 318/363 [03:27<00:29, 1.55it/s]
Profiling: 88%|████████▊ | 319/363 [03:28<00:34, 1.26it/s]
Profiling: 88%|████████▊ | 320/363 [03:29<00:39, 1.10it/s]
Profiling: 89%|████████▊ | 322/363 [03:30<00:26, 1.55it/s]
Profiling: 89%|████████▉ | 323/363 [03:30<00:24, 1.62it/s]
Profiling: 89%|████████▉ | 324/363 [03:31<00:23, 1.68it/s]
Profiling: 90%|████████▉ | 325/363 [03:31<00:21, 1.74it/s]
Profiling: 90%|█████████ | 327/363 [03:33<00:21, 1.69it/s]
Profiling: 90%|█████████ | 328/363 [03:34<00:26, 1.34it/s]
Profiling: 91%|█████████ | 329/363 [03:35<00:29, 1.15it/s]
Profiling: 91%|█████████ | 331/363 [03:36<00:19, 1.64it/s]
Profiling: 91%|█████████▏| 332/363 [03:36<00:17, 1.74it/s]
Profiling: 92%|█████████▏| 333/363 [03:37<00:16, 1.84it/s]
Profiling: 92%|█████████▏| 334/363 [03:37<00:15, 1.85it/s]
Profiling: 93%|█████████▎| 336/363 [03:38<00:15, 1.71it/s]
Profiling: 93%|█████████▎| 337/363 [03:40<00:19, 1.32it/s]
Profiling: 93%|█████████▎| 338/363 [03:41<00:22, 1.12it/s]
Profiling: 94%|█████████▎| 340/363 [03:42<00:14, 1.60it/s]
Profiling: 94%|█████████▍| 341/363 [03:42<00:13, 1.65it/s]
Profiling: 94%|█████████▍| 342/363 [03:43<00:12, 1.71it/s]
Profiling: 94%|█████████▍| 343/363 [03:43<00:11, 1.75it/s]
Profiling: 95%|█████████▍| 344/363 [03:44<00:10, 1.78it/s]
Profiling: 95%|█████████▌| 345/363 [03:44<00:09, 1.86it/s]
Profiling: 95%|█████████▌| 346/363 [03:47<00:21, 1.25s/it]
Profiling: 96%|█████████▌| 348/363 [03:48<00:14, 1.04it/s]
Profiling: 96%|█████████▌| 349/363 [03:50<00:14, 1.02s/it]
Profiling: 96%|█████████▋| 350/363 [03:51<00:13, 1.07s/it]
Profiling: 97%|█████████▋| 352/363 [03:51<00:07, 1.40it/s]
Profiling: 97%|█████████▋| 353/363 [03:52<00:06, 1.49it/s]
Profiling: 98%|█████████▊| 355/363 [03:53<00:05, 1.52it/s]
Profiling: 98%|█████████▊| 356/363 [03:54<00:05, 1.27it/s]
Profiling: 98%|█████████▊| 357/363 [03:55<00:05, 1.13it/s]
Profiling: 99%|█████████▉| 359/363 [03:56<00:02, 1.62it/s]
Profiling: 99%|█████████▉| 360/363 [03:56<00:01, 1.69it/s]
Profiling: 99%|█████████▉| 361/363 [03:57<00:01, 1.76it/s]
Profiling: 100%|█████████▉| 362/363 [03:57<00:00, 1.80it/s]
Profiling: 100%|██████████| 363/363 [03:58<00:00, 1.52it/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: quantized model in 290.866s
thanhdaonguyen-once-upon-a-time-mkmlizer: Processed model thanhdaonguyen/once-upon-a-time in 456.496s
thanhdaonguyen-once-upon-a-time-mkmlizer: creating bucket guanaco-mkml-models
thanhdaonguyen-once-upon-a-time-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
thanhdaonguyen-once-upon-a-time-mkmlizer: uploading /tmp/model_cache to s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/config.json s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/config.json
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/tokenizer_config.json s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/tokenizer_config.json
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/tokenizer.model s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/tokenizer.model
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/added_tokens.json s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/added_tokens.json
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/special_tokens_map.json s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/special_tokens_map.json
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/tokenizer.json s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/tokenizer.json
thanhdaonguyen-once-upon-a-time-mkmlizer: cp /tmp/model_cache/mkml_model.tensors s3://guanaco-mkml-models/thanhdaonguyen-once-upon-a-t-v19/mkml_model.tensors
thanhdaonguyen-once-upon-a-time-mkmlizer: loading reward model from ChaiML/reward_models_100_170000000_cp_498032
thanhdaonguyen-once-upon-a-time-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:1067: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
thanhdaonguyen-once-upon-a-time-mkmlizer: warnings.warn(
thanhdaonguyen-once-upon-a-time-mkmlizer:
config.json: 0%| | 0.00/1.03k [00:00<?, ?B/s]
config.json: 100%|██████████| 1.03k/1.03k [00:00<00:00, 12.0MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:690: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
thanhdaonguyen-once-upon-a-time-mkmlizer: warnings.warn(
thanhdaonguyen-once-upon-a-time-mkmlizer:
tokenizer_config.json: 0%| | 0.00/234 [00:00<?, ?B/s]
tokenizer_config.json: 100%|██████████| 234/234 [00:00<00:00, 3.17MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
vocab.json: 0%| | 0.00/798k [00:00<?, ?B/s]
vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 88.8MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
merges.txt: 0%| | 0.00/456k [00:00<?, ?B/s]
merges.txt: 100%|██████████| 456k/456k [00:00<00:00, 41.8MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
tokenizer.json: 0%| | 0.00/2.11M [00:00<?, ?B/s]
tokenizer.json: 100%|██████████| 2.11M/2.11M [00:00<00:00, 29.9MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer:
special_tokens_map.json: 0%| | 0.00/99.0 [00:00<?, ?B/s]
special_tokens_map.json: 100%|██████████| 99.0/99.0 [00:00<00:00, 1.24MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
thanhdaonguyen-once-upon-a-time-mkmlizer: warnings.warn(
thanhdaonguyen-once-upon-a-time-mkmlizer:
pytorch_model.bin: 0%| | 0.00/510M [00:00<?, ?B/s]
pytorch_model.bin: 1%|▏ | 7.08M/510M [00:00<00:11, 44.4MB/s]
pytorch_model.bin: 16%|█▌ | 80.5M/510M [00:00<00:01, 322MB/s]
pytorch_model.bin: 24%|██▍ | 122M/510M [00:00<00:01, 303MB/s]
pytorch_model.bin: 100%|█████████▉| 510M/510M [00:02<00:00, 243MB/s]
thanhdaonguyen-once-upon-a-time-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
thanhdaonguyen-once-upon-a-time-mkmlizer: Saving duration: 0.094s
thanhdaonguyen-once-upon-a-time-mkmlizer: Processed model ChaiML/reward_models_100_170000000_cp_498032 in 4.694s
thanhdaonguyen-once-upon-a-time-mkmlizer: creating bucket guanaco-reward-models
Job thanhdaonguyen-once-upon-a-time-mkmlizer completed after 500.23s with status: succeeded
Stopping job with name thanhdaonguyen-once-upon-a-time-mkmlizer
Running pipeline stage MKMLKubeTemplater
Running pipeline stage ISVCDeployer
Creating inference service thanhdaonguyen-once-upon-a-t-v19
Waiting for inference service thanhdaonguyen-once-upon-a-t-v19 to be ready
Inference service thanhdaonguyen-once-upon-a-t-v19 ready after 191.6025629043579s
Running pipeline stage StressChecker
Received healthy response to inference request with status code 200 in 2.454556703567505s
Received healthy response to inference request with status code 200 in 1.9068262577056885s
Received healthy response to inference request with status code 200 in 1.8436617851257324s
Received healthy response to inference request with status code 200 in 1.7016546726226807s
Received healthy response to inference request with status code 200 in 1.845677137374878s
Received healthy response to inference request with status code 200 in 1.8270161151885986s
Received healthy response to inference request with status code 200 in 1.8176116943359375s
Received healthy response to inference request with status code 200 in 1.8035285472869873s
Received healthy response to inference request with status code 200 in 1.6945862770080566s
Received healthy response to inference request with status code 200 in 1.8043437004089355s
Received healthy response to inference request with status code 200 in 1.7898280620574951s
Received healthy response to inference request with status code 200 in 1.8144927024841309s
Received healthy response to inference request with status code 200 in 1.8613414764404297s
Received healthy response to inference request with status code 200 in 1.8275642395019531s
Received healthy response to inference request with status code 200 in 1.8393409252166748s
Received healthy response to inference request with status code 200 in 1.8422765731811523s
Received healthy response to inference request with status code 200 in 1.8170888423919678s
Received healthy response to inference request with status code 200 in 1.8413503170013428s
Received healthy response to inference request with status code 200 in 1.8476135730743408s
Received healthy response to inference request with status code 200 in 2.009742259979248s
Received healthy response to inference request with status code 200 in 1.8593475818634033s
Received healthy response to inference request with status code 200 in 1.8475041389465332s
Received healthy response to inference request with status code 200 in 1.825324296951294s
Received healthy response to inference request with status code 200 in 1.7538461685180664s
Received healthy response to inference request with status code 200 in 1.8337323665618896s
Received healthy response to inference request with status code 200 in 1.8372058868408203s
Received healthy response to inference request with status code 200 in 1.8441991806030273s
Received healthy response to inference request with status code 200 in 2.0531108379364014s
Received healthy response to inference request with status code 200 in 1.8361790180206299s
Received healthy response to inference request with status code 200 in 1.8213648796081543s
Received healthy response to inference request with status code 200 in 1.825716495513916s
Received healthy response to inference request with status code 200 in 1.8484609127044678s
Received healthy response to inference request with status code 200 in 1.8100838661193848s
Received healthy response to inference request with status code 200 in 1.867431402206421s
Received healthy response to inference request with status code 200 in 1.842439889907837s
Received healthy response to inference request with status code 200 in 1.8381457328796387s
Received healthy response to inference request with status code 200 in 1.8457427024841309s
Received healthy response to inference request with status code 200 in 1.840120792388916s
Received healthy response to inference request with status code 200 in 1.844956636428833s
Received healthy response to inference request with status code 200 in 2.0686488151550293s
Received healthy response to inference request with status code 200 in 1.7595794200897217s
Received healthy response to inference request with status code 200 in 0.8910303115844727s
Received healthy response to inference request with status code 200 in 1.8024108409881592s
Received healthy response to inference request with status code 200 in 1.814594030380249s
Received healthy response to inference request with status code 200 in 1.424208164215088s
Received healthy response to inference request with status code 200 in 1.2648637294769287s
Received healthy response to inference request with status code 200 in 1.816805124282837s
Received healthy response to inference request with status code 200 in 0.781275749206543s
Received healthy response to inference request with status code 200 in 1.7249372005462646s
Received healthy response to inference request with status code 200 in 0.7982988357543945s
Received healthy response to inference request with status code 200 in 0.7959401607513428s
Received healthy response to inference request with status code 200 in 0.741218090057373s
Received healthy response to inference request with status code 200 in 1.8296501636505127s
Received healthy response to inference request with status code 200 in 0.9206328392028809s
Received healthy response to inference request with status code 200 in 0.7953472137451172s
Received healthy response to inference request with status code 200 in 1.0683391094207764s
Received healthy response to inference request with status code 200 in 1.8126826286315918s
Received healthy response to inference request with status code 200 in 1.812828540802002s
Received healthy response to inference request with status code 200 in 1.8131980895996094s
Received healthy response to inference request with status code 200 in 0.9137001037597656s
Received healthy response to inference request with status code 200 in 1.2336351871490479s
Received healthy response to inference request with status code 200 in 1.7115776538848877s
Received healthy response to inference request with status code 200 in 1.238374948501587s
Received healthy response to inference request with status code 200 in 0.740257740020752s
Received healthy response to inference request with status code 200 in 1.800154209136963s
Received healthy response to inference request with status code 200 in 1.7844021320343018s
Received healthy response to inference request with status code 200 in 1.233445405960083s
Received healthy response to inference request with status code 200 in 0.8212718963623047s
Received healthy response to inference request with status code 200 in 1.471064805984497s
Received healthy response to inference request with status code 200 in 0.8352720737457275s
Received healthy response to inference request with status code 200 in 1.907135248184204s
Received healthy response to inference request with status code 200 in 0.8011832237243652s
Received healthy response to inference request with status code 200 in 0.9156303405761719s
Received healthy response to inference request with status code 200 in 1.789670467376709s
Received healthy response to inference request with status code 200 in 1.465212345123291s
Received healthy response to inference request with status code 200 in 1.8670248985290527s
Received healthy response to inference request with status code 200 in 1.8026113510131836s
Received healthy response to inference request with status code 200 in 1.5986707210540771s
Received healthy response to inference request with status code 200 in 1.42649245262146s
Received healthy response to inference request with status code 200 in 1.8168971538543701s
Received healthy response to inference request with status code 200 in 0.8081827163696289s
Received healthy response to inference request with status code 200 in 1.8077640533447266s
Received healthy response to inference request with status code 200 in 1.6595587730407715s
Received healthy response to inference request with status code 200 in 1.8006656169891357s
Received healthy response to inference request with status code 200 in 1.7969532012939453s
Received healthy response to inference request with status code 200 in 1.4144997596740723s
Received healthy response to inference request with status code 200 in 1.7964248657226562s
Received healthy response to inference request with status code 200 in 1.7996704578399658s
Received healthy response to inference request with status code 200 in 1.8197259902954102s
Received healthy response to inference request with status code 200 in 1.3253085613250732s
Received healthy response to inference request with status code 200 in 0.9738376140594482s
Received healthy response to inference request with status code 200 in 0.7888283729553223s
Received healthy response to inference request with status code 200 in 1.1450035572052002s
Received healthy response to inference request with status code 200 in 2.4525914192199707s
Received healthy response to inference request with status code 200 in 0.7304091453552246s
Received healthy response to inference request with status code 200 in 0.7790801525115967s
Received healthy response to inference request with status code 200 in 1.8338096141815186s
Received healthy response to inference request with status code 200 in 1.7952549457550049s
Received healthy response to inference request with status code 200 in 1.8090705871582031s
Received healthy response to inference request with status code 200 in 1.0417463779449463s
100 requests
0 failed requests
5th percentile: 0.7884507417678833
10th percentile: 0.8074827671051025
20th percentile: 1.1296706676483155
30th percentile: 1.4693090677261353
40th percentile: 1.787563133239746
50th percentile: 1.8030699491500854
60th percentile: 1.815478467941284
70th percentile: 1.8281900167465208
80th percentile: 1.8423092365264893
90th percentile: 1.859546971321106
95th percentile: 1.9122655987739559
99th percentile: 2.452611072063446
mean time: 1.58553573846817
Running pipeline stage SafetyScorer
thanhdaonguyen-once-upon-a-t_v19 status is now inactive due to auto deactivation removed underperforming models
thanhdaonguyen-once-upon-a-t_v19 status is now deployed due to admin request
thanhdaonguyen-once-upon-a-t_v19 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of thanhdaonguyen-once-upon-a-t_v19
Running pipeline stage ISVCDeleter
Checking if service thanhdaonguyen-once-upon-a-t-v19 is running
Tearing down inference service thanhdaonguyen-once-upon-a-t-v19
Toredown service thanhdaonguyen-once-upon-a-t-v19
Pipeline stage ISVCDeleter completed in 6.07s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key thanhdaonguyen-once-upon-a-t-v19/added_tokens.json from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/config.json from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/mkml_model.tensors from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/tokenizer.json from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/tokenizer.model from bucket guanaco-mkml-models
Deleting key thanhdaonguyen-once-upon-a-t-v19/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/config.json from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/merges.txt from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/reward.tensors from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key thanhdaonguyen-once-upon-a-t-v19_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 2.94s
thanhdaonguyen-once-upon-a-t_v19 status is now torndown due to DeploymentManager action