Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-10-14dpo-v19-mkmlizer
Waiting for job on rica40325-10-14dpo-v19-mkmlizer to finish
rica40325-10-14dpo-v19-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-10-14dpo-v19-mkmlizer: ║ _____ __ __ ║
rica40325-10-14dpo-v19-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
rica40325-10-14dpo-v19-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
rica40325-10-14dpo-v19-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
rica40325-10-14dpo-v19-mkmlizer: ║ /___/ ║
rica40325-10-14dpo-v19-mkmlizer: ║ ║
rica40325-10-14dpo-v19-mkmlizer: ║ Version: 0.11.12 ║
rica40325-10-14dpo-v19-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
rica40325-10-14dpo-v19-mkmlizer: ║ https://mk1.ai ║
rica40325-10-14dpo-v19-mkmlizer: ║ ║
rica40325-10-14dpo-v19-mkmlizer: ║ The license key for the current software has been verified as ║
rica40325-10-14dpo-v19-mkmlizer: ║ belonging to: ║
rica40325-10-14dpo-v19-mkmlizer: ║ ║
rica40325-10-14dpo-v19-mkmlizer: ║ Chai Research Corp. ║
rica40325-10-14dpo-v19-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rica40325-10-14dpo-v19-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rica40325-10-14dpo-v19-mkmlizer: ║ ║
rica40325-10-14dpo-v19-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rica40325-10-14dpo-v19-mkmlizer: Downloaded to shared memory in 62.898s
rica40325-10-14dpo-v19-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpfgrmijqx, device:0
rica40325-10-14dpo-v19-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rica40325-10-14dpo-v19-mkmlizer: quantized model in 42.074s
rica40325-10-14dpo-v19-mkmlizer: Processed model rica40325/10_14dpo in 104.973s
rica40325-10-14dpo-v19-mkmlizer: creating bucket guanaco-mkml-models
rica40325-10-14dpo-v19-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rica40325-10-14dpo-v19-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-10-14dpo-v19
rica40325-10-14dpo-v19-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-10-14dpo-v19/config.json
rica40325-10-14dpo-v19-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-10-14dpo-v19/special_tokens_map.json
rica40325-10-14dpo-v19-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-10-14dpo-v19/tokenizer_config.json
rica40325-10-14dpo-v19-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rica40325-10-14dpo-v19/tokenizer.json
rica40325-10-14dpo-v19-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:15, 23.40it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:12, 29.35it/s]
Loading 0: 4%|▍ | 14/363 [00:00<00:13, 25.80it/s]
Loading 0: 6%|▌ | 20/363 [00:00<00:09, 35.02it/s]
Loading 0: 7%|▋ | 24/363 [00:00<00:15, 22.59it/s]
Loading 0: 7%|▋ | 27/363 [00:01<00:15, 21.89it/s]
Loading 0: 9%|▉ | 32/363 [00:01<00:14, 23.07it/s]
Loading 0: 11%|█ | 39/363 [00:01<00:10, 29.64it/s]
Loading 0: 12%|█▏ | 43/363 [00:01<00:11, 28.93it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:10, 31.37it/s]
Loading 0: 14%|█▍ | 52/363 [00:01<00:10, 30.08it/s]
Loading 0: 15%|█▌ | 56/363 [00:02<00:10, 29.79it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:11, 25.39it/s]
Loading 0: 18%|█▊ | 64/363 [00:02<00:13, 22.56it/s]
Loading 0: 20%|█▉ | 71/363 [00:02<00:09, 29.37it/s]
Loading 0: 21%|██ | 75/363 [00:02<00:09, 29.05it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:09, 28.53it/s]
Loading 0: 23%|██▎ | 84/363 [00:03<00:09, 30.82it/s]
Loading 0: 24%|██▍ | 88/363 [00:03<00:09, 29.79it/s]
Loading 0: 26%|██▌ | 93/363 [00:03<00:08, 32.17it/s]
Loading 0: 27%|██▋ | 97/363 [00:03<00:08, 30.62it/s]
Loading 0: 28%|██▊ | 101/363 [00:03<00:10, 23.85it/s]
Loading 0: 29%|██▊ | 104/363 [00:03<00:12, 21.57it/s]
Loading 0: 31%|███ | 111/363 [00:04<00:08, 28.23it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:08, 27.79it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:07, 30.40it/s]
Loading 0: 34%|███▍ | 124/363 [00:04<00:08, 29.73it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 31.96it/s]
Loading 0: 37%|███▋ | 133/363 [00:04<00:07, 30.61it/s]
Loading 0: 38%|███▊ | 137/363 [00:04<00:07, 30.44it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:09, 23.91it/s]
Loading 0: 40%|███▉ | 145/363 [00:05<00:09, 22.99it/s]
Loading 0: 41%|████ | 149/363 [00:05<00:09, 22.30it/s]
Loading 0: 43%|████▎ | 156/363 [00:05<00:07, 28.58it/s]
Loading 0: 44%|████▍ | 160/363 [00:05<00:07, 28.29it/s]
Loading 0: 45%|████▌ | 165/363 [00:05<00:06, 31.05it/s]
Loading 0: 47%|████▋ | 169/363 [00:06<00:06, 30.00it/s]
Loading 0: 48%|████▊ | 174/363 [00:06<00:05, 32.32it/s]
Loading 0: 49%|████▉ | 178/363 [00:06<00:06, 30.63it/s]
Loading 0: 50%|█████ | 182/363 [00:06<00:08, 22.33it/s]
Loading 0: 51%|█████ | 185/363 [00:06<00:08, 20.28it/s]
Loading 0: 53%|█████▎ | 192/363 [00:07<00:06, 27.23it/s]
Loading 0: 54%|█████▍ | 196/363 [00:07<00:06, 26.88it/s]
Loading 0: 55%|█████▌ | 201/363 [00:07<00:05, 29.62it/s]
Loading 0: 56%|█████▋ | 205/363 [00:07<00:05, 28.81it/s]
Loading 0: 58%|█████▊ | 210/363 [00:07<00:04, 30.75it/s]
Loading 0: 59%|█████▉ | 214/363 [00:07<00:05, 29.32it/s]
Loading 0: 60%|██████ | 218/363 [00:07<00:04, 29.16it/s]
Loading 0: 61%|██████▏ | 223/363 [00:08<00:05, 24.90it/s]
Loading 0: 62%|██████▏ | 226/363 [00:08<00:05, 23.39it/s]
Loading 0: 63%|██████▎ | 230/363 [00:08<00:05, 22.20it/s]
Loading 0: 65%|██████▌ | 237/363 [00:08<00:04, 29.06it/s]
Loading 0: 66%|██████▋ | 241/363 [00:08<00:04, 28.67it/s]
Loading 0: 68%|██████▊ | 246/363 [00:08<00:03, 30.63it/s]
Loading 0: 69%|██████▉ | 250/363 [00:09<00:03, 29.12it/s]
Loading 0: 70%|███████ | 255/363 [00:09<00:03, 31.20it/s]
Loading 0: 71%|███████▏ | 259/363 [00:09<00:03, 30.05it/s]
Loading 0: 72%|███████▏ | 263/363 [00:09<00:04, 24.05it/s]
Loading 0: 73%|███████▎ | 266/363 [00:09<00:04, 21.56it/s]
Loading 0: 75%|███████▌ | 273/363 [00:09<00:03, 27.13it/s]
Loading 0: 76%|███████▌ | 276/363 [00:10<00:03, 25.58it/s]
Loading 0: 78%|███████▊ | 282/363 [00:10<00:02, 30.25it/s]
Loading 0: 79%|███████▉ | 286/363 [00:10<00:02, 29.59it/s]
Loading 0: 80%|████████ | 291/363 [00:10<00:02, 31.80it/s]
Loading 0: 81%|████████▏ | 295/363 [00:10<00:02, 29.70it/s]
Loading 0: 82%|████████▏ | 299/363 [00:10<00:02, 30.00it/s]
Loading 0: 84%|████████▎ | 304/363 [00:11<00:02, 24.94it/s]
Loading 0: 85%|████████▍ | 307/363 [00:11<00:02, 23.59it/s]
Loading 0: 86%|████████▌ | 311/363 [00:11<00:02, 22.93it/s]
Loading 0: 88%|████████▊ | 318/363 [00:11<00:01, 29.50it/s]
Loading 0: 89%|████████▊ | 322/363 [00:11<00:01, 28.66it/s]
Loading 0: 90%|█████████ | 327/363 [00:11<00:01, 31.14it/s]
Loading 0: 91%|█████████ | 331/363 [00:11<00:01, 30.10it/s]
Loading 0: 93%|█████████▎| 336/363 [00:12<00:00, 31.99it/s]
Loading 0: 94%|█████████▎| 340/363 [00:12<00:00, 30.22it/s]
Loading 0: 95%|█████████▍| 344/363 [00:19<00:09, 1.99it/s]
Loading 0: 96%|█████████▌| 348/363 [00:19<00:05, 2.67it/s]
Loading 0: 97%|█████████▋| 353/363 [00:19<00:02, 3.88it/s]
Loading 0: 98%|█████████▊| 357/363 [00:19<00:01, 5.04it/s]
Job rica40325-10-14dpo-v19-mkmlizer completed after 134.81s with status: succeeded
Stopping job with name rica40325-10-14dpo-v19-mkmlizer
Pipeline stage MKMLizer completed in 135.29s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=DeploymentError('while parsing a flow mapping\n in "<unicode string>", line 1, column 1867:\n ... name\': \'TOP_K\', \'value\': \'80\'}, {\'name\': \'STOPPING_WORDS\', \'valu ... \n ^\nexpected \',\' or \'}\', but got \'<scalar>\'\n in "<unicode string>", line 1, column 1908:\n ... \'STOPPING_WORDS\', \'value\': \'["\\\'\\\\n\\\',", "\\\'</s>\\\',", "\\\'Bot:\\\', ... \n ^')
Shutdown handler de-registered
rica40325-10-14dpo_v19 status is now failed due to DeploymentManager action