rirv938-testing-model-175

submission_id: rirv938-testing-model-175_v1
developer_uid: rirv938
status: failed
model_repo: rirv938/testing_model_175
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2024-12-21T03:20:43+00:00
model_name: rirv938-testing-model-175_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-testing-model-175-v1-mkmlizer
Waiting for job on rirv938-testing-model-175-v1-mkmlizer to finish
rirv938-testing-model-175-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-testing-model-175-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-testing-model-175-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-testing-model-175-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-testing-model-175-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-testing-model-175-v1-mkmlizer: ║       /___/                                                         ║
rirv938-testing-model-175-v1-mkmlizer: ║                                                                     ║
rirv938-testing-model-175-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
rirv938-testing-model-175-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-testing-model-175-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-testing-model-175-v1-mkmlizer: ║                                                                     ║
rirv938-testing-model-175-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-testing-model-175-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-testing-model-175-v1-mkmlizer: ║                                                                     ║
rirv938-testing-model-175-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-testing-model-175-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-testing-model-175-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
rirv938-testing-model-175-v1-mkmlizer: ║                                                                     ║
rirv938-testing-model-175-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-testing-model-175-v1-mkmlizer: Downloaded to shared memory in 110.975s
rirv938-testing-model-175-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpecmf7_ty, device:0
rirv938-testing-model-175-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-testing-model-175-v1-mkmlizer: quantized model in 42.215s
rirv938-testing-model-175-v1-mkmlizer: Processed model rirv938/testing_model_175 in 153.190s
rirv938-testing-model-175-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-testing-model-175-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-testing-model-175-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-testing-model-175-v1
rirv938-testing-model-175-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-testing-model-175-v1/config.json
rirv938-testing-model-175-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-testing-model-175-v1/special_tokens_map.json
rirv938-testing-model-175-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-testing-model-175-v1/tokenizer.json
rirv938-testing-model-175-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-testing-model-175-v1/flywheel_model.0.safetensors
rirv938-testing-model-175-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:16, 21.14it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:13, 26.99it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:15, 23.19it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:10, 32.31it/s]
Loading 0:   7%|▋         | 24/363 [00:01<00:15, 21.63it/s]
Loading 0:   7%|▋         | 27/363 [00:01<00:16, 20.41it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:13, 23.76it/s]
Loading 0:   9%|▉         | 34/363 [00:01<00:13, 23.72it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:11, 27.36it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:12, 26.33it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:11, 28.32it/s]
Loading 0:  14%|█▍        | 51/363 [00:02<00:12, 25.76it/s]
Loading 0:  15%|█▌        | 56/363 [00:02<00:10, 28.68it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:12, 24.18it/s]
Loading 0:  18%|█▊        | 64/363 [00:02<00:14, 20.98it/s]
Loading 0:  19%|█▉        | 69/363 [00:02<00:11, 26.26it/s]
Loading 0:  20%|██        | 73/363 [00:02<00:11, 25.68it/s]
Loading 0:  21%|██        | 77/363 [00:03<00:12, 23.67it/s]
Loading 0:  23%|██▎       | 82/363 [00:03<00:09, 28.57it/s]
Loading 0:  24%|██▎       | 86/363 [00:03<00:10, 25.45it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:08, 31.97it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 30.08it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:10, 23.90it/s]
Loading 0:  29%|██▊       | 104/363 [00:04<00:12, 21.17it/s]
Loading 0:  31%|███       | 111/363 [00:04<00:09, 27.62it/s]
Loading 0:  32%|███▏      | 115/363 [00:04<00:09, 27.14it/s]
Loading 0:  33%|███▎      | 120/363 [00:04<00:08, 29.91it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:08, 28.64it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:07, 30.99it/s]
Loading 0:  37%|███▋      | 133/363 [00:05<00:07, 29.49it/s]
Loading 0:  38%|███▊      | 137/363 [00:05<00:07, 29.30it/s]
Loading 0:  39%|███▉      | 142/363 [00:05<00:08, 25.03it/s]
Loading 0:  40%|███▉      | 145/363 [00:05<00:09, 23.25it/s]
Loading 0:  41%|████      | 148/363 [00:05<00:08, 24.38it/s]
Loading 0:  42%|████▏     | 151/363 [00:05<00:08, 24.13it/s]
Loading 0:  42%|████▏     | 154/363 [00:05<00:08, 25.13it/s]
Loading 0:  44%|████▎     | 158/363 [00:06<00:09, 22.33it/s]
Loading 0:  45%|████▍     | 163/363 [00:06<00:07, 27.27it/s]
Loading 0:  46%|████▌     | 167/363 [00:06<00:08, 23.82it/s]
Loading 0:  47%|████▋     | 172/363 [00:06<00:06, 28.77it/s]
Loading 0:  48%|████▊     | 176/363 [00:06<00:07, 24.99it/s]
Loading 0:  50%|█████     | 182/363 [00:07<00:07, 23.72it/s]
Loading 0:  51%|█████     | 185/363 [00:07<00:08, 20.76it/s]
Loading 0:  52%|█████▏    | 190/363 [00:07<00:06, 25.81it/s]
Loading 0:  53%|█████▎    | 194/363 [00:07<00:07, 23.57it/s]
Loading 0:  55%|█████▌    | 201/363 [00:07<00:05, 29.62it/s]
Loading 0:  56%|█████▋    | 205/363 [00:07<00:05, 27.69it/s]
Loading 0:  58%|█████▊    | 210/363 [00:08<00:05, 29.32it/s]
Loading 0:  59%|█████▉    | 214/363 [00:08<00:05, 27.93it/s]
Loading 0:  60%|██████    | 218/363 [00:08<00:05, 27.56it/s]
Loading 0:  61%|██████    | 222/363 [00:08<00:04, 29.70it/s]
Loading 0:  62%|██████▏   | 226/363 [00:08<00:06, 20.86it/s]
Loading 0:  63%|██████▎   | 230/363 [00:09<00:06, 20.57it/s]
Loading 0:  65%|██████▌   | 237/363 [00:09<00:04, 27.32it/s]
Loading 0:  66%|██████▋   | 241/363 [00:09<00:04, 27.14it/s]
Loading 0:  68%|██████▊   | 246/363 [00:09<00:03, 29.65it/s]
Loading 0:  69%|██████▉   | 250/363 [00:09<00:03, 28.70it/s]
Loading 0:  70%|███████   | 255/363 [00:09<00:03, 30.59it/s]
Loading 0:  71%|███████▏  | 259/363 [00:09<00:03, 28.38it/s]
Loading 0:  72%|███████▏  | 263/363 [00:10<00:04, 23.04it/s]
Loading 0:  73%|███████▎  | 266/363 [00:10<00:04, 20.60it/s]
Loading 0:  75%|███████▌  | 273/363 [00:10<00:03, 27.52it/s]
Loading 0:  76%|███████▋  | 277/363 [00:10<00:03, 26.92it/s]
Loading 0:  78%|███████▊  | 282/363 [00:10<00:02, 29.44it/s]
Loading 0:  79%|███████▉  | 286/363 [00:10<00:02, 28.33it/s]
Loading 0:  80%|████████  | 291/363 [00:11<00:02, 30.94it/s]
Loading 0:  81%|████████▏ | 295/363 [00:11<00:02, 29.48it/s]
Loading 0:  82%|████████▏ | 299/363 [00:11<00:02, 29.77it/s]
Loading 0:  84%|████████▎ | 304/363 [00:11<00:02, 25.74it/s]
Loading 0:  85%|████████▍ | 307/363 [00:11<00:02, 24.02it/s]
Loading 0:  86%|████████▌ | 311/363 [00:11<00:02, 23.06it/s]
Loading 0:  88%|████████▊ | 318/363 [00:12<00:01, 29.96it/s]
Loading 0:  89%|████████▊ | 322/363 [00:12<00:01, 28.93it/s]
Loading 0:  90%|█████████ | 327/363 [00:12<00:01, 31.75it/s]
Loading 0:  91%|█████████ | 331/363 [00:12<00:01, 30.29it/s]
Loading 0:  93%|█████████▎| 336/363 [00:12<00:00, 32.24it/s]
Loading 0:  94%|█████████▎| 340/363 [00:12<00:00, 30.17it/s]
Loading 0:  95%|█████████▍| 344/363 [00:19<00:09,  1.97it/s]
Loading 0:  96%|█████████▌| 348/363 [00:20<00:05,  2.64it/s]
Loading 0:  97%|█████████▋| 353/363 [00:20<00:02,  3.83it/s]
Loading 0:  98%|█████████▊| 357/363 [00:20<00:01,  4.93it/s]
Loading 0: 100%|█████████▉| 362/363 [00:20<00:00,  6.99it/s]
                                                            
Job rirv938-testing-model-175-v1-mkmlizer completed after 176.11s with status: succeeded
Stopping job with name rirv938-testing-model-175-v1-mkmlizer
Pipeline stage MKMLizer completed in 176.60s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-testing-model-175-v1
Waiting for inference service rirv938-testing-model-175-v1 to be ready
Inference service rirv938-testing-model-175-v1 ready after 252.05664801597595s
Pipeline stage MKMLDeployer completed in 252.89s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.6444785594940186s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
4 failed requests
5th percentile: 5.36655626296997
10th percentile: 9.088633966445922
20th percentile: 16.53278937339783
30th percentile: 20.27730760574341
40th percentile: 20.322188663482667
50th percentile: 20.367069721221924
60th percentile: 20.36948690414429
70th percentile: 20.37190408706665
80th percentile: 20.4138813495636
90th percentile: 20.49541869163513
95th percentile: 20.536187362670898
99th percentile: 20.568802299499513
mean time: 16.643296813964845
%s, retrying in %s seconds...
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Failed to get response for submission function_lijit_2024-11-26: ('http://blend-jifom-2024-12-20-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"detail":"Internal server error"}')
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.3203013420105
10th percentile: 20.324738359451295
20th percentile: 20.333612394332885
30th percentile: 20.338920879364014
40th percentile: 20.340663814544676
50th percentile: 20.342406749725342
60th percentile: 20.373171615600587
70th percentile: 20.40393648147583
80th percentile: 20.809687089920043
90th percentile: 21.59042344093323
95th percentile: 21.98079161643982
99th percentile: 22.293086156845092
mean time: 20.75735983848572
%s, retrying in %s seconds...
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.23699116706848
10th percentile: 20.24540686607361
20th percentile: 20.262238264083862
30th percentile: 20.274253559112548
40th percentile: 20.28145275115967
50th percentile: 20.288651943206787
60th percentile: 20.32509469985962
70th percentile: 20.36153745651245
80th percentile: 20.398450565338134
90th percentile: 20.43583402633667
95th percentile: 20.454525756835938
99th percentile: 20.46947914123535
mean time: 20.32817153930664
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
rirv938-testing-model-175_v1 status is now failed due to DeploymentManager action