hastagaras-sbr-3b2

submission_id: hastagaras-sbr-3b2_v1
developer_uid: Hastagaras
formatter: {'memory_template': '### Instruction:\n{memory}\n', 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}', 'truncate_by_message': False}
generation_params: {'temperature': 0.95, 'top_p': 0.95, 'min_p': 0.08, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
model_name: hastagaras-sbr-3b2_v1
model_repo: Hastagaras/sbr-3b2
status: torndown
timestamp: 2024-09-29T09:08:36+00:00
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name hastagaras-sbr-3b2-v1-mkmlizer
Waiting for job on hastagaras-sbr-3b2-v1-mkmlizer to finish
hastagaras-sbr-3b2-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
hastagaras-sbr-3b2-v1-mkmlizer: ║     _____            __           __                                ║
hastagaras-sbr-3b2-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
hastagaras-sbr-3b2-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
hastagaras-sbr-3b2-v1-mkmlizer: ║       /___/                                                         ║
hastagaras-sbr-3b2-v1-mkmlizer: ║                                                                     ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  https://mk1.ai                                                     ║
hastagaras-sbr-3b2-v1-mkmlizer: ║                                                                     ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  belonging to:                                                      ║
hastagaras-sbr-3b2-v1-mkmlizer: ║                                                                     ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  Chai Research Corp.                                                ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
hastagaras-sbr-3b2-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
hastagaras-sbr-3b2-v1-mkmlizer: ║                                                                     ║
hastagaras-sbr-3b2-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
hastagaras-sbr-3b2-v1-mkmlizer: Downloaded to shared memory in 16.328s
hastagaras-sbr-3b2-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpnv0pgrld, device:0
hastagaras-sbr-3b2-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
hastagaras-sbr-3b2-v1-mkmlizer: quantized model in 12.271s
hastagaras-sbr-3b2-v1-mkmlizer: Processed model Hastagaras/sbr-3b2 in 28.600s
hastagaras-sbr-3b2-v1-mkmlizer: creating bucket guanaco-mkml-models
hastagaras-sbr-3b2-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
hastagaras-sbr-3b2-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1
hastagaras-sbr-3b2-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1/config.json
hastagaras-sbr-3b2-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1/special_tokens_map.json
hastagaras-sbr-3b2-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1/tokenizer_config.json
hastagaras-sbr-3b2-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1/tokenizer.json
hastagaras-sbr-3b2-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/hastagaras-sbr-3b2-v1/flywheel_model.0.safetensors
hastagaras-sbr-3b2-v1-mkmlizer: 
Loading 0:   0%|          | 0/254 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/254 [00:00<00:06, 41.12it/s]
Loading 0:   6%|▌         | 14/254 [00:00<00:04, 52.46it/s]
Loading 0:   9%|▉         | 23/254 [00:00<00:04, 56.86it/s]
Loading 0:  13%|█▎        | 32/254 [00:00<00:03, 58.59it/s]
Loading 0:  16%|█▌        | 41/254 [00:00<00:03, 60.83it/s]
Loading 0:  20%|█▉        | 50/254 [00:00<00:03, 62.90it/s]
Loading 0:  23%|██▎       | 59/254 [00:00<00:03, 63.07it/s]
Loading 0:  27%|██▋       | 68/254 [00:01<00:03, 60.67it/s]
Loading 0:  30%|███       | 77/254 [00:01<00:03, 57.52it/s]
Loading 0:  33%|███▎      | 85/254 [00:01<00:02, 62.17it/s]
Loading 0:  36%|███▌      | 92/254 [00:01<00:02, 55.49it/s]
Loading 0:  39%|███▊      | 98/254 [00:01<00:02, 55.67it/s]
Loading 0:  41%|████      | 104/254 [00:01<00:03, 48.69it/s]
Loading 0:  44%|████▍     | 113/254 [00:02<00:02, 52.90it/s]
Loading 0:  47%|████▋     | 120/254 [00:02<00:02, 54.00it/s]
Loading 0:  50%|█████     | 128/254 [00:02<00:02, 55.79it/s]
Loading 0:  54%|█████▍    | 137/254 [00:02<00:02, 55.39it/s]
Loading 0:  57%|█████▋    | 146/254 [00:02<00:01, 58.28it/s]
Loading 0:  61%|██████    | 155/254 [00:02<00:01, 59.73it/s]
Loading 0:  65%|██████▍   | 164/254 [00:02<00:01, 60.33it/s]
Loading 0:  68%|██████▊   | 173/254 [00:02<00:01, 62.52it/s]
Loading 0:  72%|███████▏  | 182/254 [00:03<00:01, 64.29it/s]
Loading 0:  74%|███████▍  | 189/254 [00:03<00:01, 48.84it/s]
Loading 0:  78%|███████▊  | 199/254 [00:03<00:01, 53.78it/s]
Loading 0:  82%|████████▏ | 208/254 [00:03<00:00, 57.37it/s]
Loading 0:  85%|████████▌ | 217/254 [00:03<00:00, 60.17it/s]
Loading 0:  89%|████████▉ | 226/254 [00:03<00:00, 60.64it/s]
Loading 0:  93%|█████████▎| 235/254 [00:04<00:00, 62.09it/s]
Loading 0:  96%|█████████▌| 244/254 [00:04<00:00, 63.92it/s]
Loading 0: 100%|█████████▉| 253/254 [00:04<00:00, 63.78it/s]
                                                            
Job hastagaras-sbr-3b2-v1-mkmlizer completed after 51.94s with status: succeeded
Stopping job with name hastagaras-sbr-3b2-v1-mkmlizer
Pipeline stage MKMLizer completed in 52.34s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.07s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service hastagaras-sbr-3b2-v1
Waiting for inference service hastagaras-sbr-3b2-v1 to be ready
Inference service hastagaras-sbr-3b2-v1 ready after 220.62018275260925s
Pipeline stage MKMLDeployer completed in 220.93s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.08692522048950195
10th percentile: 0.08778753280639648
20th percentile: 0.08951215744018555
30th percentile: 0.09077539443969726
40th percentile: 0.09157724380493164
50th percentile: 0.09237909317016602
60th percentile: 0.09389762878417969
70th percentile: 0.09541616439819336
80th percentile: 0.1057785987854004
90th percentile: 0.12498493194580079
95th percentile: 0.13458809852600095
99th percentile: 0.14227063179016114
mean time: 0.10183663368225097
%s, retrying in %s seconds...
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.09713525772094726
10th percentile: 0.09749355316162109
20th percentile: 0.09821014404296875
30th percentile: 0.09869351387023925
40th percentile: 0.09894366264343261
50th percentile: 0.09919381141662598
60th percentile: 0.1004112720489502
70th percentile: 0.10162873268127441
80th percentile: 0.1035116195678711
90th percentile: 0.10605993270874023
95th percentile: 0.1073340892791748
99th percentile: 0.10835341453552245
mean time: 0.10107698440551757
%s, retrying in %s seconds...
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
{"detail":"'message'"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.10400280952453614
10th percentile: 0.10424108505249023
20th percentile: 0.10471763610839843
30th percentile: 0.10695185661315917
40th percentile: 0.11094374656677246
50th percentile: 0.11493563652038574
60th percentile: 0.11532101631164551
70th percentile: 0.11570639610290527
80th percentile: 0.11855239868164062
90th percentile: 0.12385902404785157
95th percentile: 0.12651233673095702
99th percentile: 0.1286349868774414
mean time: 0.11374416351318359
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
hastagaras-sbr-3b2_v1 status is now failed due to DeploymentManager action
hastagaras-sbr-3b2_v1 status is now torndown due to DeploymentManager action