delta-vector-odin-9b

submission_id: delta-vector-odin-9b_v1
developer_uid: Delta-Vector
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
model_name: delta-vector-odin-9b_v1
model_repo: Delta-Vector/Odin-9B
status: torndown
timestamp: 2024-10-24T13:50:09+00:00
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name delta-vector-odin-9b-v1-mkmlizer
Waiting for job on delta-vector-odin-9b-v1-mkmlizer to finish
delta-vector-odin-9b-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
delta-vector-odin-9b-v1-mkmlizer: ║     _____            __           __                                ║
delta-vector-odin-9b-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
delta-vector-odin-9b-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
delta-vector-odin-9b-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
delta-vector-odin-9b-v1-mkmlizer: ║       /___/                                                         ║
delta-vector-odin-9b-v1-mkmlizer: ║                                                                     ║
delta-vector-odin-9b-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
delta-vector-odin-9b-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
delta-vector-odin-9b-v1-mkmlizer: ║  https://mk1.ai                                                     ║
delta-vector-odin-9b-v1-mkmlizer: ║                                                                     ║
delta-vector-odin-9b-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
delta-vector-odin-9b-v1-mkmlizer: ║  belonging to:                                                      ║
delta-vector-odin-9b-v1-mkmlizer: ║                                                                     ║
delta-vector-odin-9b-v1-mkmlizer: ║  Chai Research Corp.                                                ║
delta-vector-odin-9b-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
delta-vector-odin-9b-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
delta-vector-odin-9b-v1-mkmlizer: ║                                                                     ║
delta-vector-odin-9b-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
delta-vector-odin-9b-v1-mkmlizer: quantized model in 35.248s
delta-vector-odin-9b-v1-mkmlizer: Processed model Delta-Vector/Odin-9B in 77.772s
delta-vector-odin-9b-v1-mkmlizer: creating bucket guanaco-mkml-models
delta-vector-odin-9b-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
delta-vector-odin-9b-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/delta-vector-odin-9b-v1
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/delta-vector-odin-9b-v1/special_tokens_map.json
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/delta-vector-odin-9b-v1/config.json
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/delta-vector-odin-9b-v1/tokenizer_config.json
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/delta-vector-odin-9b-v1/tokenizer.model
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/delta-vector-odin-9b-v1/tokenizer.json
delta-vector-odin-9b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/delta-vector-odin-9b-v1/flywheel_model.0.safetensors
delta-vector-odin-9b-v1-mkmlizer: 
Loading 0:   0%|          | 0/464 [00:00<?, ?it/s]
Loading 0:   3%|▎         | 12/464 [00:00<00:05, 79.99it/s]
Loading 0:   5%|▍         | 23/464 [00:00<00:06, 73.48it/s]
Loading 0:   7%|▋         | 34/464 [00:00<00:05, 80.08it/s]
Loading 0:  10%|▉         | 45/464 [00:00<00:05, 83.14it/s]
Loading 0:  12%|█▏        | 56/464 [00:00<00:04, 84.79it/s]
Loading 0:  14%|█▍        | 67/464 [00:00<00:04, 87.16it/s]
Loading 0:  17%|█▋        | 78/464 [00:00<00:04, 89.65it/s]
Loading 0:  19%|█▉        | 88/464 [00:01<00:06, 59.47it/s]
Loading 0:  21%|██        | 98/464 [00:01<00:05, 65.87it/s]
Loading 0:  23%|██▎       | 108/464 [00:01<00:04, 71.82it/s]
Loading 0:  25%|██▌       | 117/464 [00:01<00:04, 73.64it/s]
Loading 0:  28%|██▊       | 128/464 [00:01<00:04, 77.15it/s]
Loading 0:  30%|██▉       | 139/464 [00:01<00:04, 79.75it/s]
Loading 0:  32%|███▏      | 150/464 [00:01<00:03, 82.83it/s]
Loading 0:  35%|███▍      | 161/464 [00:02<00:03, 85.52it/s]
Loading 0:  37%|███▋      | 172/464 [00:02<00:03, 86.90it/s]
Loading 0:  39%|███▉      | 183/464 [00:02<00:03, 86.49it/s]
Loading 0:  42%|████▏     | 194/464 [00:02<00:03, 87.74it/s]
Loading 0:  44%|████▍     | 205/464 [00:02<00:02, 92.64it/s]
Loading 0:  47%|████▋     | 216/464 [00:02<00:02, 92.10it/s]
Loading 0:  49%|████▊     | 226/464 [00:02<00:03, 67.89it/s]
Loading 0:  52%|█████▏    | 242/464 [00:02<00:02, 87.03it/s]
Loading 0:  55%|█████▍    | 253/464 [00:03<00:02, 87.02it/s]
Loading 0:  57%|█████▋    | 263/464 [00:03<00:02, 85.82it/s]
Loading 0:  59%|█████▉    | 273/464 [00:03<00:02, 85.78it/s]
Loading 0:  61%|██████    | 283/464 [00:03<00:02, 85.63it/s]
Loading 0:  63%|██████▎   | 292/464 [00:03<00:02, 82.70it/s]
Loading 0:  65%|██████▍   | 301/464 [00:03<00:01, 83.10it/s]
Loading 0:  67%|██████▋   | 311/464 [00:03<00:01, 86.28it/s]
Loading 0:  69%|██████▉   | 320/464 [00:03<00:01, 87.29it/s]
Loading 0:  71%|███████▏  | 331/464 [00:04<00:01, 88.31it/s]
Loading 0:  74%|███████▎  | 342/464 [00:04<00:01, 89.38it/s]
Loading 0:  76%|███████▌  | 353/464 [00:04<00:01, 87.72it/s]
Loading 0:  78%|███████▊  | 362/464 [00:04<00:01, 57.32it/s]
Loading 0:  81%|████████  | 375/464 [00:04<00:01, 68.76it/s]
Loading 0:  83%|████████▎ | 386/464 [00:04<00:01, 74.12it/s]
Loading 0:  86%|████████▌ | 397/464 [00:04<00:00, 79.04it/s]
Loading 0:  88%|████████▊ | 408/464 [00:05<00:00, 82.01it/s]
Loading 0:  90%|█████████ | 419/464 [00:05<00:00, 82.12it/s]
Loading 0:  93%|█████████▎| 430/464 [00:05<00:00, 85.42it/s]
Loading 0:  95%|█████████▌| 441/464 [00:05<00:00, 87.91it/s]
Loading 0:  97%|█████████▋| 452/464 [00:05<00:00, 88.57it/s]
Loading 0: 100%|█████████▉| 463/464 [00:05<00:00, 87.69it/s]
                                                            
Job delta-vector-odin-9b-v1-mkmlizer completed after 104.29s with status: succeeded
Stopping job with name delta-vector-odin-9b-v1-mkmlizer
Pipeline stage MKMLizer completed in 104.86s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service delta-vector-odin-9b-v1
Waiting for inference service delta-vector-odin-9b-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service delta-vector-odin-9b-v1 ready after 130.57322359085083s
Pipeline stage MKMLDeployer completed in 131.23s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.185838556289674
10th percentile: 20.18757743835449
20th percentile: 20.19105520248413
30th percentile: 20.199364709854127
40th percentile: 20.212505960464476
50th percentile: 20.22564721107483
60th percentile: 20.230942964553833
70th percentile: 20.236238718032837
80th percentile: 20.25626425743103
90th percentile: 20.29101958274841
95th percentile: 20.308397245407104
99th percentile: 20.322299375534058
mean time: 20.233440494537355
%s, retrying in %s seconds...
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.159937715530397
Retrying (%r) after connection broken by '%r': %s
10th percentile: 20.16157455444336
20th percentile: 20.164848232269286
30th percentile: 20.167220640182496
40th percentile: 20.168691778182982
50th percentile: 20.17016291618347
60th percentile: 20.17815718650818
70th percentile: 20.186151456832885
80th percentile: 20.198821783065796
90th percentile: 20.21616816520691
95th percentile: 20.224841356277466
99th percentile: 20.231779909133913
mean time: 20.183722400665282
%s, retrying in %s seconds...
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 20.140118217468263
10th percentile: 20.14317650794983
20th percentile: 20.149293088912962
30th percentile: 20.158210945129394
40th percentile: 20.169930076599123
50th percentile: 20.181649208068848
60th percentile: 20.204113388061522
70th percentile: 20.2265775680542
80th percentile: 20.305293321609497
90th percentile: 20.440260648727417
95th percentile: 20.507744312286377
99th percentile: 20.561731243133544
mean time: 20.256819629669188
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
delta-vector-odin-9b_v1 status is now failed due to DeploymentManager action
delta-vector-odin-9b_v1 status is now torndown due to DeploymentManager action