billxbf-chai-nemo13b-sf_55377

submission_id: billxbf-chai-nemo13b-sf_55377_v2
developer_uid: billxbf
status: failed
model_repo: billxbf/chai-nemo13b-sft-orpo-merge_v2
generation_params: {'temperature': 1.1, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ["'Me:',", "'</s>',", "'####'", "'You:',", "'Bot:',", "'User:',", '\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2025-02-09T21:12:35+00:00
model_name: billxbf-chai-nemo13b-sf_55377_v2
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name billxbf-chai-nemo13b-sf-55377-v2-mkmlizer
Waiting for job on billxbf-chai-nemo13b-sf-55377-v2-mkmlizer to finish
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║     _____            __           __                                ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║       /___/                                                         ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║                                                                     ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  https://mk1.ai                                                     ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║                                                                     ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  belonging to:                                                      ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║                                                                     ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  Chai Research Corp.                                                ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ║                                                                     ║
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: Downloaded to shared memory in 50.743s
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp44zuwnlw, device:0
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: quantized model in 35.544s
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: Processed model billxbf/chai-nemo13b-sft-orpo-merge_v2 in 86.288s
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: creating bucket guanaco-mkml-models
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2/config.json
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2/special_tokens_map.json
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2/tokenizer_config.json
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2/tokenizer.json
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/billxbf-chai-nemo13b-sf-55377-v2/flywheel_model.0.safetensors
billxbf-chai-nemo13b-sf-55377-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:42,  3.11s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:58,  1.20it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:46,  3.28it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:12,  4.80it/s]
Loading 0:   6%|▋         | 23/363 [00:06<00:43,  7.85it/s]
Loading 0:   8%|▊         | 29/363 [00:06<00:29, 11.27it/s]
Loading 0:   9%|▉         | 34/363 [00:06<00:22, 14.62it/s]
Loading 0:  11%|█         | 40/363 [00:07<00:19, 16.24it/s]
Loading 0:  12%|█▏        | 44/363 [00:07<00:16, 18.93it/s]
Loading 0:  14%|█▍        | 50/363 [00:07<00:12, 24.42it/s]
Loading 0:  15%|█▌        | 56/363 [00:07<00:10, 27.91it/s]
Loading 0:  17%|█▋        | 61/363 [00:07<00:09, 30.67it/s]
Loading 0:  19%|█▊        | 68/363 [00:07<00:07, 37.17it/s]
Loading 0:  20%|██        | 74/363 [00:08<00:07, 37.89it/s]
Loading 0:  22%|██▏       | 79/363 [00:08<00:07, 38.59it/s]
Loading 0:  24%|██▎       | 86/363 [00:08<00:06, 43.85it/s]
Loading 0:  25%|██▌       | 92/363 [00:08<00:06, 42.41it/s]
Loading 0:  27%|██▋       | 97/363 [00:08<00:06, 41.52it/s]
Loading 0:  29%|██▊       | 104/363 [00:08<00:05, 46.13it/s]
Loading 0:  30%|███       | 110/363 [00:08<00:05, 43.53it/s]
Loading 0:  32%|███▏      | 115/363 [00:08<00:05, 42.04it/s]
Loading 0:  33%|███▎      | 121/363 [00:09<00:07, 31.01it/s]
Loading 0:  34%|███▍      | 125/363 [00:09<00:07, 32.25it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:06, 37.12it/s]
Loading 0:  38%|███▊      | 137/363 [00:09<00:06, 37.59it/s]
Loading 0:  39%|███▉      | 142/363 [00:09<00:05, 38.12it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:04, 43.96it/s]
Loading 0:  43%|████▎     | 155/363 [00:10<00:04, 42.04it/s]
Loading 0:  44%|████▍     | 160/363 [00:10<00:04, 41.47it/s]
Loading 0:  46%|████▌     | 166/363 [00:10<00:04, 45.60it/s]
Loading 0:  47%|████▋     | 171/363 [00:10<00:04, 45.76it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:04, 45.55it/s]
Loading 0:  50%|█████     | 182/363 [00:10<00:04, 42.87it/s]
Loading 0:  52%|█████▏    | 187/363 [00:10<00:04, 41.49it/s]
Loading 0:  53%|█████▎    | 194/363 [00:10<00:03, 46.28it/s]
Loading 0:  55%|█████▌    | 200/363 [00:11<00:03, 43.62it/s]
Loading 0:  56%|█████▋    | 205/363 [00:11<00:05, 28.08it/s]
Loading 0:  58%|█████▊    | 212/363 [00:11<00:04, 34.39it/s]
Loading 0:  60%|██████    | 218/363 [00:11<00:04, 35.71it/s]
Loading 0:  61%|██████▏   | 223/363 [00:11<00:03, 36.55it/s]
Loading 0:  63%|██████▎   | 229/363 [00:11<00:03, 41.03it/s]
Loading 0:  64%|██████▍   | 234/363 [00:12<00:03, 41.23it/s]
Loading 0:  66%|██████▌   | 239/363 [00:12<00:02, 42.65it/s]
Loading 0:  67%|██████▋   | 245/363 [00:12<00:02, 41.36it/s]
Loading 0:  69%|██████▉   | 250/363 [00:12<00:02, 41.19it/s]
Loading 0:  71%|███████   | 256/363 [00:12<00:02, 42.94it/s]
Loading 0:  72%|███████▏  | 261/363 [00:12<00:02, 44.09it/s]
Loading 0:  73%|███████▎  | 266/363 [00:12<00:02, 42.84it/s]
Loading 0:  75%|███████▍  | 272/363 [00:12<00:02, 41.68it/s]
Loading 0:  76%|███████▋  | 277/363 [00:13<00:02, 42.35it/s]
Loading 0:  78%|███████▊  | 283/363 [00:13<00:02, 30.40it/s]
Loading 0:  79%|███████▉  | 287/363 [00:13<00:02, 31.84it/s]
Loading 0:  81%|████████  | 293/363 [00:13<00:01, 36.73it/s]
Loading 0:  82%|████████▏ | 298/363 [00:13<00:01, 39.56it/s]
Loading 0:  83%|████████▎ | 303/363 [00:13<00:01, 35.22it/s]
Loading 0:  86%|████████▌ | 311/363 [00:13<00:01, 43.62it/s]
Loading 0:  87%|████████▋ | 317/363 [00:14<00:01, 42.56it/s]
Loading 0:  89%|████████▊ | 322/363 [00:14<00:00, 43.07it/s]
Loading 0:  90%|█████████ | 328/363 [00:14<00:00, 46.16it/s]
Loading 0:  92%|█████████▏| 333/363 [00:14<00:00, 45.62it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:00, 46.28it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 43.68it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 42.59it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 47.49it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 44.74it/s]
                                                            
Job billxbf-chai-nemo13b-sf-55377-v2-mkmlizer completed after 114.49s with status: succeeded
Stopping job with name billxbf-chai-nemo13b-sf-55377-v2-mkmlizer
Pipeline stage MKMLizer completed in 114.92s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=DeploymentError('while parsing a flow mapping\n  in "<unicode string>", line 1, column 1885:\n     ... name\': \'TOP_K\', \'value\': \'80\'}, {\'name\': \'STOPPING_WORDS\', \'valu ... \n                                         ^\nexpected \',\' or \'}\', but got \'<scalar>\'\n  in "<unicode string>", line 1, column 1933:\n     ... NG_WORDS\', \'value\': \'["\\\\n", "\\\'</s>\\\',", "\\\'Bot:\\\',", "\\\'User:\\ ... \n                                         ^')
Shutdown handler de-registered
billxbf-chai-nemo13b-sf_55377_v2 status is now failed due to DeploymentManager action