juvi21-uwu-8b

submission_id: juvi21-uwu-8b_v3
developer_uid: juvi21
formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.21, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|end_header_id|>,', '<|eot_id|>,', '\n\n{user_name}', 'You:', '\n\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64, 'reward_max_token_input': 256}
model_name: uwu-8b
model_repo: juvi21/UwU-8B
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: Jellywibble/gpt2_xl_pairwise_89m_step_347634
status: torndown
timestamp: 2024-08-03T14:49:29+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name juvi21-uwu-8b-v3-mkmlizer
Waiting for job on juvi21-uwu-8b-v3-mkmlizer to finish
juvi21-uwu-8b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
juvi21-uwu-8b-v3-mkmlizer: ║     _____            __           __                                ║
juvi21-uwu-8b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
juvi21-uwu-8b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
juvi21-uwu-8b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
juvi21-uwu-8b-v3-mkmlizer: ║       /___/                                                         ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Version: 0.9.9                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
juvi21-uwu-8b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
juvi21-uwu-8b-v3-mkmlizer: ║  belonging to:                                                      ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
juvi21-uwu-8b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
juvi21-uwu-8b-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
juvi21-uwu-8b-v3-mkmlizer: Downloaded to shared memory in 22.600s
juvi21-uwu-8b-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpzmmvb5uo, device:0
juvi21-uwu-8b-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
juvi21-uwu-8b-v3-mkmlizer: quantized model in 26.467s
juvi21-uwu-8b-v3-mkmlizer: Processed model juvi21/UwU-8B in 49.067s
juvi21-uwu-8b-v3-mkmlizer: creating bucket guanaco-mkml-models
juvi21-uwu-8b-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
juvi21-uwu-8b-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/juvi21-uwu-8b-v3
juvi21-uwu-8b-v3-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/291 [00:04<11:32,  2.40s/it]
Loading 0:   2%|▏         | 6/291 [00:04<03:04,  1.54it/s]
Loading 0:   4%|▍         | 11/291 [00:05<01:21,  3.44it/s]
Loading 0:   5%|▌         | 15/291 [00:05<00:51,  5.34it/s]
Loading 0:   8%|▊         | 23/291 [00:05<00:25, 10.47it/s]
Loading 0:  10%|▉         | 29/291 [00:05<00:18, 14.10it/s]
Loading 0:  12%|█▏        | 34/291 [00:05<00:14, 17.75it/s]
Loading 0:  14%|█▍        | 41/291 [00:05<00:10, 24.16it/s]
Loading 0:  16%|█▌        | 46/291 [00:05<00:08, 28.06it/s]
Loading 0:  18%|█▊        | 51/291 [00:05<00:08, 28.02it/s]
Loading 0:  20%|██        | 59/291 [00:06<00:06, 36.68it/s]
Loading 0:  22%|██▏       | 65/291 [00:06<00:08, 27.79it/s]
Loading 0:  24%|██▍       | 70/291 [00:06<00:07, 30.65it/s]
Loading 0:  26%|██▋       | 77/291 [00:06<00:05, 37.26it/s]
Loading 0:  29%|██▊       | 83/291 [00:06<00:05, 37.34it/s]
Loading 0:  30%|███       | 88/291 [00:06<00:05, 38.81it/s]
Loading 0:  33%|███▎      | 95/291 [00:07<00:04, 44.11it/s]
Loading 0:  35%|███▍      | 101/291 [00:07<00:04, 42.65it/s]
Loading 0:  36%|███▋      | 106/291 [00:07<00:04, 43.08it/s]
Loading 0:  39%|███▉      | 113/291 [00:07<00:03, 47.91it/s]
Loading 0:  41%|████      | 119/291 [00:07<00:03, 44.61it/s]
Loading 0:  43%|████▎     | 124/291 [00:07<00:03, 44.11it/s]
Loading 0:  45%|████▌     | 131/291 [00:07<00:03, 49.26it/s]
Loading 0:  47%|████▋     | 137/291 [00:07<00:03, 44.13it/s]
Loading 0:  49%|████▉     | 142/291 [00:08<00:03, 43.78it/s]
Loading 0:  51%|█████     | 149/291 [00:08<00:02, 49.16it/s]
Loading 0:  53%|█████▎    | 155/291 [00:08<00:03, 45.21it/s]
Loading 0:  55%|█████▍    | 160/291 [00:08<00:03, 41.45it/s]
Loading 0:  57%|█████▋    | 166/291 [00:08<00:03, 32.52it/s]
Loading 0:  58%|█████▊    | 170/291 [00:08<00:03, 33.90it/s]
Loading 0:  60%|██████    | 176/291 [00:08<00:03, 38.17it/s]
Loading 0:  62%|██████▏   | 181/291 [00:09<00:02, 39.68it/s]
Loading 0:  64%|██████▍   | 186/291 [00:09<00:02, 36.21it/s]
Loading 0:  67%|██████▋   | 194/291 [00:09<00:02, 45.00it/s]
Loading 0:  69%|██████▊   | 200/291 [00:09<00:02, 43.46it/s]
Loading 0:  70%|███████   | 205/291 [00:09<00:01, 43.79it/s]
Loading 0:  73%|███████▎  | 212/291 [00:09<00:01, 48.68it/s]
Loading 0:  75%|███████▍  | 218/291 [00:09<00:01, 44.30it/s]
Loading 0:  77%|███████▋  | 223/291 [00:10<00:01, 44.44it/s]
Loading 0:  79%|███████▉  | 230/291 [00:10<00:01, 49.81it/s]
Loading 0:  81%|████████  | 236/291 [00:10<00:01, 46.09it/s]
Loading 0:  83%|████████▎ | 241/291 [00:10<00:01, 46.09it/s]
Loading 0:  85%|████████▌ | 248/291 [00:10<00:00, 50.80it/s]
Loading 0:  87%|████████▋ | 254/291 [00:10<00:00, 46.87it/s]
Loading 0:  89%|████████▉ | 259/291 [00:10<00:00, 45.79it/s]
Loading 0:  91%|█████████▏| 266/291 [00:11<00:00, 33.97it/s]
Loading 0:  93%|█████████▎| 272/291 [00:11<00:00, 35.45it/s]
Loading 0:  95%|█████████▍| 276/291 [00:11<00:00, 36.06it/s]
Loading 0:  98%|█████████▊| 284/291 [00:11<00:00, 45.26it/s]
Loading 0: 100%|█████████▉| 290/291 [00:11<00:00, 43.79it/s]
                                                            
ERROR "cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/config.json": QuotaExceeded:  status code: 403, request id: tx0000058a7373eb19c96be-0066ae43b6-a438a3bd5-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/special_tokens_map.json": QuotaExceeded:  status code: 403, request id: tx00000cb61473911942294-0066ae43b6-a438a3bd5-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer_config.json": QuotaExceeded:  status code: 403, request id: tx00000b5f9e32df97efd25-0066ae43b6-a438a3bd5-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer.json": QuotaExceeded:  status code: 403, request id: tx00000fa1d56344f9b9b5d-0066ae43b7-a438a3bd5-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/juvi21-uwu-8b-v3/flywheel_model.0.safetensors": MultipartUpload: upload multipart failed upload id: 2~FxjyJcP6DJ1oD21dHKq1_EW1ExtzHOp caused by: QuotaExceeded:  status code: 403, request id: tx0000020cdde9ad46c0d1a-0066ae43b7-a41e5e51a-default, host id:
Job juvi21-uwu-8b-v3-mkmlizer completed after 73.43s with status: failed
Stopping job with name juvi21-uwu-8b-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name juvi21-uwu-8b-v3-mkmlizer
Waiting for job on juvi21-uwu-8b-v3-mkmlizer to finish
juvi21-uwu-8b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
juvi21-uwu-8b-v3-mkmlizer: ║     _____            __           __                                ║
juvi21-uwu-8b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
juvi21-uwu-8b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
juvi21-uwu-8b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
juvi21-uwu-8b-v3-mkmlizer: ║       /___/                                                         ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Version: 0.9.9                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
juvi21-uwu-8b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
juvi21-uwu-8b-v3-mkmlizer: ║  belonging to:                                                      ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
juvi21-uwu-8b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
juvi21-uwu-8b-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
juvi21-uwu-8b-v3-mkmlizer: Downloaded to shared memory in 21.486s
juvi21-uwu-8b-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmprl_3vba4, device:0
juvi21-uwu-8b-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
juvi21-uwu-8b-v3-mkmlizer: quantized model in 26.389s
juvi21-uwu-8b-v3-mkmlizer: Processed model juvi21/UwU-8B in 47.875s
juvi21-uwu-8b-v3-mkmlizer: creating bucket guanaco-mkml-models
juvi21-uwu-8b-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
juvi21-uwu-8b-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/juvi21-uwu-8b-v3
juvi21-uwu-8b-v3-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/291 [00:04<11:29,  2.38s/it]
Loading 0:   2%|▏         | 6/291 [00:04<03:03,  1.55it/s]
Loading 0:   4%|▍         | 13/291 [00:05<01:05,  4.25it/s]
Loading 0:   7%|▋         | 19/291 [00:05<00:37,  7.21it/s]
Loading 0:   8%|▊         | 24/291 [00:05<00:27,  9.80it/s]
Loading 0:  11%|█▏        | 33/291 [00:05<00:16, 15.89it/s]
Loading 0:  14%|█▍        | 41/291 [00:05<00:11, 22.65it/s]
Loading 0:  16%|█▌        | 47/291 [00:05<00:09, 26.25it/s]
Loading 0:  18%|█▊        | 53/291 [00:05<00:07, 30.23it/s]
Loading 0:  20%|██        | 59/291 [00:05<00:06, 34.50it/s]
Loading 0:  22%|██▏       | 65/291 [00:06<00:08, 26.99it/s]
Loading 0:  24%|██▍       | 70/291 [00:06<00:07, 29.92it/s]
Loading 0:  26%|██▋       | 77/291 [00:06<00:05, 36.91it/s]
Loading 0:  29%|██▊       | 83/291 [00:06<00:05, 37.61it/s]
Loading 0:  30%|███       | 88/291 [00:06<00:05, 38.81it/s]
Loading 0:  33%|███▎      | 95/291 [00:06<00:04, 44.16it/s]
Loading 0:  35%|███▍      | 101/291 [00:07<00:04, 42.49it/s]
Loading 0:  36%|███▋      | 106/291 [00:07<00:04, 42.40it/s]
Loading 0:  38%|███▊      | 112/291 [00:07<00:03, 46.41it/s]
Loading 0:  40%|████      | 117/291 [00:07<00:03, 46.68it/s]
Loading 0:  42%|████▏     | 123/291 [00:07<00:04, 41.77it/s]
Loading 0:  45%|████▌     | 131/291 [00:07<00:03, 49.58it/s]
Loading 0:  47%|████▋     | 137/291 [00:07<00:03, 44.98it/s]
Loading 0:  49%|████▉     | 142/291 [00:07<00:03, 44.54it/s]
Loading 0:  51%|█████     | 149/291 [00:08<00:02, 49.44it/s]
Loading 0:  53%|█████▎    | 155/291 [00:08<00:02, 45.73it/s]
Loading 0:  55%|█████▍    | 160/291 [00:08<00:02, 44.98it/s]
Loading 0:  57%|█████▋    | 166/291 [00:08<00:03, 34.41it/s]
Loading 0:  58%|█████▊    | 170/291 [00:08<00:03, 34.88it/s]
Loading 0:  60%|██████    | 176/291 [00:08<00:02, 39.74it/s]
Loading 0:  63%|██████▎   | 182/291 [00:08<00:02, 39.69it/s]
Loading 0:  64%|██████▍   | 187/291 [00:09<00:02, 39.94it/s]
Loading 0:  67%|██████▋   | 194/291 [00:09<00:02, 46.05it/s]
Loading 0:  68%|██████▊   | 199/291 [00:09<00:01, 47.03it/s]
Loading 0:  70%|███████   | 204/291 [00:09<00:02, 40.03it/s]
Loading 0:  73%|███████▎  | 212/291 [00:09<00:01, 48.26it/s]
Loading 0:  75%|███████▍  | 218/291 [00:09<00:01, 43.49it/s]
Loading 0:  77%|███████▋  | 223/291 [00:09<00:01, 43.13it/s]
Loading 0:  79%|███████▉  | 230/291 [00:09<00:01, 48.45it/s]
Loading 0:  81%|████████  | 236/291 [00:10<00:01, 45.09it/s]
Loading 0:  83%|████████▎ | 241/291 [00:10<00:01, 44.64it/s]
Loading 0:  85%|████████▌ | 248/291 [00:10<00:00, 49.54it/s]
Loading 0:  87%|████████▋ | 254/291 [00:10<00:00, 45.06it/s]
Loading 0:  89%|████████▉ | 259/291 [00:10<00:00, 43.81it/s]
Loading 0:  91%|█████████ | 265/291 [00:10<00:00, 47.41it/s]
Loading 0:  93%|█████████▎| 270/291 [00:11<00:00, 32.56it/s]
Loading 0:  95%|█████████▍| 275/291 [00:11<00:00, 35.46it/s]
Loading 0:  97%|█████████▋| 281/291 [00:11<00:00, 36.25it/s]
Loading 0:  98%|█████████▊| 286/291 [00:11<00:00, 37.42it/s]
                                                            
ERROR "cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/config.json": QuotaExceeded:  status code: 403, request id: tx000000783a33708929c41-0066ae43fd-a43888a4c-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/special_tokens_map.json": QuotaExceeded:  status code: 403, request id: tx00000f940d4b6b99af3b4-0066ae43fd-a43888a4c-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer_config.json": QuotaExceeded:  status code: 403, request id: tx000002244428e55ca8d7d-0066ae43fd-a43888a4c-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer.json": QuotaExceeded:  status code: 403, request id: tx0000022e30841c2da8e83-0066ae43fd-a41e5e51a-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/juvi21-uwu-8b-v3/flywheel_model.0.safetensors": MultipartUpload: upload multipart failed upload id: 2~DyIL-91Su6wxiCeCi00N7WH8trYL0Ew caused by: QuotaExceeded:  status code: 403, request id: tx00000c8cc100cf736a936-0066ae43fe-a41e5e51a-default, host id:
Job juvi21-uwu-8b-v3-mkmlizer completed after 63.13s with status: failed
Stopping job with name juvi21-uwu-8b-v3-mkmlizer
%s, retrying in %s seconds...
Starting job with name juvi21-uwu-8b-v3-mkmlizer
Waiting for job on juvi21-uwu-8b-v3-mkmlizer to finish
juvi21-uwu-8b-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
juvi21-uwu-8b-v3-mkmlizer: ║     _____            __           __                                ║
juvi21-uwu-8b-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
juvi21-uwu-8b-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
juvi21-uwu-8b-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
juvi21-uwu-8b-v3-mkmlizer: ║       /___/                                                         ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Version: 0.9.9                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
juvi21-uwu-8b-v3-mkmlizer: ║  https://mk1.ai                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
juvi21-uwu-8b-v3-mkmlizer: ║  belonging to:                                                      ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ║  Chai Research Corp.                                                ║
juvi21-uwu-8b-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
juvi21-uwu-8b-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
juvi21-uwu-8b-v3-mkmlizer: ║                                                                     ║
juvi21-uwu-8b-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
juvi21-uwu-8b-v3-mkmlizer: Downloaded to shared memory in 21.544s
juvi21-uwu-8b-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp8bewx1re, device:0
juvi21-uwu-8b-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
juvi21-uwu-8b-v3-mkmlizer: quantized model in 26.512s
juvi21-uwu-8b-v3-mkmlizer: Processed model juvi21/UwU-8B in 48.056s
juvi21-uwu-8b-v3-mkmlizer: creating bucket guanaco-mkml-models
juvi21-uwu-8b-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
juvi21-uwu-8b-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/juvi21-uwu-8b-v3
juvi21-uwu-8b-v3-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/291 [00:04<11:33,  2.40s/it]
Loading 0:   2%|▏         | 6/291 [00:04<03:04,  1.55it/s]
Loading 0:   4%|▍         | 13/291 [00:05<01:05,  4.23it/s]
Loading 0:   7%|▋         | 19/291 [00:05<00:37,  7.18it/s]
Loading 0:   8%|▊         | 24/291 [00:05<00:27,  9.74it/s]
Loading 0:  11%|█         | 32/291 [00:05<00:16, 15.89it/s]
Loading 0:  13%|█▎        | 38/291 [00:05<00:12, 19.90it/s]
Loading 0:  15%|█▌        | 44/291 [00:05<00:10, 24.61it/s]
Loading 0:  17%|█▋        | 50/291 [00:05<00:08, 29.91it/s]
Loading 0:  19%|█▉        | 56/291 [00:05<00:07, 32.27it/s]
Loading 0:  21%|██        | 61/291 [00:06<00:08, 25.68it/s]
Loading 0:  23%|██▎       | 68/291 [00:06<00:06, 32.46it/s]
Loading 0:  25%|██▌       | 74/291 [00:06<00:06, 33.70it/s]
Loading 0:  27%|██▋       | 79/291 [00:06<00:05, 35.84it/s]
Loading 0:  30%|██▉       | 86/291 [00:06<00:04, 42.28it/s]
Loading 0:  32%|███▏      | 92/291 [00:06<00:04, 41.60it/s]
Loading 0:  33%|███▎      | 97/291 [00:06<00:04, 41.48it/s]
Loading 0:  36%|███▌      | 104/291 [00:07<00:03, 47.10it/s]
Loading 0:  38%|███▊      | 110/291 [00:07<00:04, 42.97it/s]
Loading 0:  40%|███▉      | 115/291 [00:07<00:04, 43.14it/s]
Loading 0:  42%|████▏     | 122/291 [00:07<00:03, 48.48it/s]
Loading 0:  44%|████▍     | 128/291 [00:07<00:03, 46.01it/s]
Loading 0:  46%|████▌     | 133/291 [00:07<00:03, 45.70it/s]
Loading 0:  48%|████▊     | 139/291 [00:07<00:03, 48.94it/s]
Loading 0:  50%|████▉     | 145/291 [00:07<00:02, 49.94it/s]
Loading 0:  52%|█████▏    | 151/291 [00:08<00:03, 43.99it/s]
Loading 0:  54%|█████▍    | 158/291 [00:08<00:02, 48.87it/s]
Loading 0:  56%|█████▋    | 164/291 [00:08<00:02, 43.98it/s]
Loading 0:  58%|█████▊    | 169/291 [00:08<00:03, 31.11it/s]
Loading 0:  60%|██████    | 176/291 [00:08<00:03, 37.48it/s]
Loading 0:  63%|██████▎   | 182/291 [00:08<00:02, 37.92it/s]
Loading 0:  64%|██████▍   | 187/291 [00:09<00:02, 39.29it/s]
Loading 0:  66%|██████▋   | 193/291 [00:09<00:02, 43.73it/s]
Loading 0:  68%|██████▊   | 198/291 [00:09<00:02, 43.60it/s]
Loading 0:  70%|██████▉   | 203/291 [00:09<00:01, 44.96it/s]
Loading 0:  72%|███████▏  | 209/291 [00:09<00:01, 42.58it/s]
Loading 0:  74%|███████▎  | 214/291 [00:09<00:01, 42.64it/s]
Loading 0:  76%|███████▌  | 220/291 [00:09<00:01, 46.13it/s]
Loading 0:  77%|███████▋  | 225/291 [00:09<00:01, 46.49it/s]
Loading 0:  79%|███████▉  | 230/291 [00:10<00:01, 47.13it/s]
Loading 0:  81%|████████  | 236/291 [00:10<00:01, 43.27it/s]
Loading 0:  83%|████████▎ | 241/291 [00:10<00:01, 42.88it/s]
Loading 0:  85%|████████▌ | 248/291 [00:10<00:00, 47.85it/s]
Loading 0:  87%|████████▋ | 254/291 [00:10<00:00, 44.36it/s]
Loading 0:  89%|████████▉ | 259/291 [00:10<00:00, 42.95it/s]
Loading 0:  91%|█████████▏| 266/291 [00:10<00:00, 32.94it/s]
Loading 0:  93%|█████████▎| 272/291 [00:11<00:00, 34.19it/s]
Loading 0:  95%|█████████▍| 276/291 [00:11<00:00, 34.01it/s]
Loading 0:  97%|█████████▋| 283/291 [00:11<00:00, 41.56it/s]
Loading 0:  99%|█████████▉| 288/291 [00:11<00:00, 42.88it/s]
                                                            
ERROR "cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/config.json": QuotaExceeded:  status code: 403, request id: tx000005101a48841acc2a5-0066ae443c-a41e57da0-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/special_tokens_map.json": QuotaExceeded:  status code: 403, request id: tx00000fc913e0f54c1b42f-0066ae443c-a41e57da0-default, host id:
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer_config.json": QuotaExceeded:  status code: 403, request id: tx00000b5b5ce220f0bc367-0066ae443c-a41e57da0-default, host id:
juvi21-uwu-8b-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/juvi21-uwu-8b-v3/tokenizer.json
juvi21-uwu-8b-v3-mkmlizer: ERROR "cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/juvi21-uwu-8b-v3/flywheel_model.0.safetensors": MultipartUpload: upload multipart failed upload id: 2~xCLVoirUv8wQ4KN9jWPaNv9QuwpP8Nc caused by: QuotaExceeded:  status code: 403, request id: tx00000f5e1a67e0b7068c4-0066ae443d-a41e57da0-default, host id:
Job juvi21-uwu-8b-v3-mkmlizer completed after 63.72s with status: failed
Stopping job with name juvi21-uwu-8b-v3-mkmlizer
MKMLizerError('')
juvi21-uwu-8b_v3 status is now failed due to DeploymentManager action
admin requested tearing down of juvi21-uwu-8b_v3
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.20s
Running pipeline stage MKMLModelDeleter
Skipping deletion as no model was successfully uploaded
Pipeline stage MKMLModelDeleter completed in 0.17s
juvi21-uwu-8b_v3 status is now torndown due to DeploymentManager action