sao10k-12b-spicie-tyr

submission_id: sao10k-12b-spicie-tyr_v2
developer_uid: sao10k
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 0.75, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '\n\n', '\nYou:', '[/INST]', '<|im_end|>', '</s>'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}
model_name: tyr-spice
model_repo: Sao10K/12B-Spicie-Tyr
status: deployed
timestamp: 2024-10-22T17:26:08+00:00
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name sao10k-12b-spicie-tyr-v2-mkmlizer
Waiting for job on sao10k-12b-spicie-tyr-v2-mkmlizer to finish
sao10k-12b-spicie-tyr-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-12b-spicie-tyr-v2-mkmlizer: ║     _____            __           __                                ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║       /___/                                                         ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║                                                                     ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  https://mk1.ai                                                     ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║                                                                     ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  belonging to:                                                      ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║                                                                     ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  Chai Research Corp.                                                ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ║                                                                     ║
sao10k-12b-spicie-tyr-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sao10k-12b-spicie-tyr-v2-mkmlizer: Downloaded to shared memory in 44.850s
sao10k-12b-spicie-tyr-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpph_twmz3, device:0
sao10k-12b-spicie-tyr-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-12b-spicie-tyr-v2-mkmlizer: quantized model in 35.588s
sao10k-12b-spicie-tyr-v2-mkmlizer: Processed model Sao10K/12B-Spicie-Tyr in 80.438s
sao10k-12b-spicie-tyr-v2-mkmlizer: creating bucket guanaco-mkml-models
sao10k-12b-spicie-tyr-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-12b-spicie-tyr-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2
sao10k-12b-spicie-tyr-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2/config.json
sao10k-12b-spicie-tyr-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2/special_tokens_map.json
sao10k-12b-spicie-tyr-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2/tokenizer_config.json
sao10k-12b-spicie-tyr-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2/tokenizer.json
sao10k-12b-spicie-tyr-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/sao10k-12b-spicie-tyr-v2/flywheel_model.0.safetensors
sao10k-12b-spicie-tyr-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 2/363 [00:06<18:10,  3.02s/it]
Loading 0:   2%|▏         | 6/363 [00:06<04:50,  1.23it/s]
Loading 0:   4%|▎         | 13/363 [00:06<01:43,  3.37it/s]
Loading 0:   5%|▍         | 17/363 [00:06<01:10,  4.94it/s]
Loading 0:   6%|▋         | 23/363 [00:06<00:42,  8.09it/s]
Loading 0:   8%|▊         | 29/363 [00:06<00:28, 11.78it/s]
Loading 0:   9%|▉         | 34/363 [00:06<00:21, 15.22it/s]
Loading 0:  11%|█▏        | 41/363 [00:06<00:15, 21.44it/s]
Loading 0:  13%|█▎        | 47/363 [00:07<00:12, 25.77it/s]
Loading 0:  14%|█▍        | 52/363 [00:07<00:10, 29.07it/s]
Loading 0:  16%|█▌        | 58/363 [00:07<00:12, 25.14it/s]
Loading 0:  17%|█▋        | 62/363 [00:07<00:11, 26.81it/s]
Loading 0:  18%|█▊        | 67/363 [00:07<00:09, 30.31it/s]
Loading 0:  20%|█▉        | 71/363 [00:07<00:09, 31.58it/s]
Loading 0:  21%|██        | 77/363 [00:07<00:07, 37.51it/s]
Loading 0:  23%|██▎       | 83/363 [00:08<00:07, 39.86it/s]
Loading 0:  24%|██▍       | 88/363 [00:08<00:06, 41.22it/s]
Loading 0:  26%|██▌       | 95/363 [00:08<00:05, 47.30it/s]
Loading 0:  28%|██▊       | 101/363 [00:08<00:05, 45.92it/s]
Loading 0:  29%|██▉       | 106/363 [00:08<00:05, 44.96it/s]
Loading 0:  31%|███       | 112/363 [00:08<00:05, 48.70it/s]
Loading 0:  33%|███▎      | 118/363 [00:08<00:05, 48.46it/s]
Loading 0:  34%|███▍      | 123/363 [00:08<00:05, 41.75it/s]
Loading 0:  36%|███▌      | 131/363 [00:09<00:04, 48.72it/s]
Loading 0:  38%|███▊      | 137/363 [00:09<00:04, 47.48it/s]
Loading 0:  39%|███▉      | 142/363 [00:09<00:04, 46.03it/s]
Loading 0:  41%|████      | 149/363 [00:09<00:04, 49.95it/s]
Loading 0:  43%|████▎     | 155/363 [00:09<00:04, 44.86it/s]
Loading 0:  44%|████▍     | 160/363 [00:09<00:06, 29.16it/s]
Loading 0:  46%|████▌     | 166/363 [00:10<00:05, 33.35it/s]
Loading 0:  47%|████▋     | 171/363 [00:10<00:05, 35.34it/s]
Loading 0:  48%|████▊     | 176/363 [00:10<00:04, 38.22it/s]
Loading 0:  50%|█████     | 182/363 [00:10<00:04, 39.72it/s]
Loading 0:  52%|█████▏    | 187/363 [00:10<00:04, 40.32it/s]
Loading 0:  53%|█████▎    | 194/363 [00:10<00:03, 46.09it/s]
Loading 0:  55%|█████▌    | 200/363 [00:10<00:03, 45.88it/s]
Loading 0:  56%|█████▋    | 205/363 [00:10<00:03, 43.52it/s]
Loading 0:  58%|█████▊    | 212/363 [00:10<00:03, 47.67it/s]
Loading 0:  60%|█████▉    | 217/363 [00:11<00:03, 47.90it/s]
Loading 0:  61%|██████    | 222/363 [00:11<00:03, 39.73it/s]
Loading 0:  63%|██████▎   | 230/363 [00:11<00:02, 47.58it/s]
Loading 0:  65%|██████▌   | 236/363 [00:11<00:02, 45.38it/s]
Loading 0:  66%|██████▋   | 241/363 [00:11<00:02, 44.66it/s]
Loading 0:  68%|██████▊   | 248/363 [00:11<00:02, 48.67it/s]
Loading 0:  70%|██████▉   | 254/363 [00:11<00:02, 47.56it/s]
Loading 0:  71%|███████▏  | 259/363 [00:12<00:03, 30.30it/s]
Loading 0:  73%|███████▎  | 265/363 [00:12<00:02, 35.47it/s]
Loading 0:  74%|███████▍  | 270/363 [00:12<00:02, 35.65it/s]
Loading 0:  76%|███████▌  | 275/363 [00:12<00:02, 36.67it/s]
Loading 0:  77%|███████▋  | 281/363 [00:12<00:02, 38.80it/s]
Loading 0:  79%|███████▉  | 286/363 [00:12<00:01, 39.81it/s]
Loading 0:  81%|████████  | 293/363 [00:12<00:01, 44.87it/s]
Loading 0:  82%|████████▏ | 299/363 [00:13<00:01, 44.50it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:01, 43.82it/s]
Loading 0:  86%|████████▌ | 311/363 [00:13<00:01, 47.87it/s]
Loading 0:  87%|████████▋ | 316/363 [00:13<00:00, 47.87it/s]
Loading 0:  88%|████████▊ | 321/363 [00:13<00:01, 37.52it/s]
Loading 0:  90%|█████████ | 328/363 [00:13<00:00, 43.98it/s]
Loading 0:  92%|█████████▏| 333/363 [00:13<00:00, 44.16it/s]
Loading 0:  93%|█████████▎| 338/363 [00:13<00:00, 45.38it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 45.27it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 44.54it/s]
Loading 0:  98%|█████████▊| 355/363 [00:14<00:00, 33.00it/s]
Loading 0:  99%|█████████▉| 359/363 [00:14<00:00, 33.53it/s]
                                                            
Job sao10k-12b-spicie-tyr-v2-mkmlizer completed after 104.02s with status: succeeded
Stopping job with name sao10k-12b-spicie-tyr-v2-mkmlizer
Pipeline stage MKMLizer completed in 104.54s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service sao10k-12b-spicie-tyr-v2
Waiting for inference service sao10k-12b-spicie-tyr-v2 to be ready
Inference service sao10k-12b-spicie-tyr-v2 ready after 140.48224329948425s
Pipeline stage MKMLDeployer completed in 141.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.061652898788452s
Received healthy response to inference request in 1.721250295639038s
Received healthy response to inference request in 1.6166300773620605s
Received healthy response to inference request in 1.9745841026306152s
Received healthy response to inference request in 1.7803542613983154s
5 requests
0 failed requests
5th percentile: 1.637554121017456
10th percentile: 1.6584781646728515
20th percentile: 1.7003262519836426
30th percentile: 1.7330710887908936
40th percentile: 1.7567126750946045
50th percentile: 1.7803542613983154
60th percentile: 1.8580461978912353
70th percentile: 1.9357381343841553
80th percentile: 1.9919978618621825
90th percentile: 2.0268253803253176
95th percentile: 2.044239139556885
99th percentile: 2.0581701469421385
mean time: 1.8308943271636964
Pipeline stage StressChecker completed in 10.55s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.91s
Shutdown handler de-registered
sao10k-12b-spicie-tyr_v2 status is now deployed due to DeploymentManager action