submission_id: sao10k-14b-qwen2-5-kunou-v1_v1
developer_uid: sy2x20000202
status: failed
model_repo: Sao10K/14B-Qwen2.5-Kunou-v1
generation_params: {'temperature': 1.1, 'top_p': 1.0, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': False}
timestamp: 2025-02-04T01:32:16+00:00
model_name: sao10k-14b-qwen2-5-kunou-v1_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer
Waiting for job on sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer to finish
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ _____ __ __ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ /___/ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ Version: 0.11.12 ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ https://mk1.ai ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ The license key for the current software has been verified as ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ belonging to: ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ Chai Research Corp. ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ║ ║
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: Downloaded to shared memory in 50.638s
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpwjtymb4g, device:0
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: quantized model in 35.333s
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: Processed model Sao10K/14B-Qwen2.5-Kunou-v1 in 85.971s
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: creating bucket guanaco-mkml-models
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/config.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/added_tokens.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/special_tokens_map.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/tokenizer_config.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/merges.txt
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/vocab.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/tokenizer.json
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/flywheel_model.1.safetensors
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/sao10k-14b-qwen2-5-kunou-v1-v1/flywheel_model.0.safetensors
sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer: Loading 0: 0%| | 0/579 [00:00<?, ?it/s] Loading 0: 0%| | 2/579 [00:06<33:16, 3.46s/it] Loading 0: 3%|▎ | 16/579 [00:07<03:01, 3.11it/s] Loading 0: 5%|▌ | 29/579 [00:07<01:22, 6.66it/s] Loading 0: 7%|▋ | 42/579 [00:07<00:47, 11.41it/s] Loading 0: 11%|█ | 62/579 [00:07<00:24, 21.22it/s] Loading 0: 13%|█▎ | 77/579 [00:07<00:16, 29.72it/s] Loading 0: 16%|█▌ | 91/579 [00:07<00:12, 39.42it/s] Loading 0: 19%|█▉ | 110/579 [00:07<00:08, 56.27it/s] Loading 0: 22%|██▏ | 126/579 [00:07<00:06, 66.69it/s] Loading 0: 25%|██▍ | 144/579 [00:07<00:05, 84.45it/s] Loading 0: 28%|██▊ | 160/579 [00:08<00:07, 58.31it/s] Loading 0: 30%|██▉ | 173/579 [00:08<00:06, 67.49it/s] Loading 0: 32%|███▏ | 186/579 [00:08<00:05, 77.17it/s] Loading 0: 35%|███▌ | 204/579 [00:08<00:03, 96.17it/s] Loading 0: 38%|███▊ | 220/579 [00:08<00:03, 108.06it/s] Loading 0: 41%|████ | 235/579 [00:08<00:03, 111.16it/s] Loading 0: 44%|████▍ | 254/579 [00:09<00:02, 129.30it/s] Loading 0: 47%|████▋ | 270/579 [00:09<00:02, 123.88it/s] Loading 0: 50%|█████ | 290/579 [00:09<00:02, 142.32it/s] Loading 0: 53%|█████▎ | 306/579 [00:09<00:02, 131.96it/s] Loading 0: 56%|█████▋ | 327/579 [00:09<00:01, 151.37it/s] Loading 0: 59%|█████▉ | 344/579 [00:09<00:01, 141.15it/s] Loading 0: 63%|██████▎ | 362/579 [00:09<00:01, 150.53it/s] Loading 0: 65%|██████▌ | 378/579 [00:10<00:02, 78.88it/s] Loading 0: 68%|██████▊ | 391/579 [00:10<00:02, 85.96it/s] Loading 0: 71%|███████ | 410/579 [00:10<00:01, 105.18it/s] Loading 0: 73%|███████▎ | 425/579 [00:10<00:01, 110.12it/s] Loading 0: 76%|███████▌ | 439/579 [00:10<00:01, 116.23it/s] Loading 0: 79%|███████▉ | 460/579 [00:10<00:00, 131.73it/s] Loading 0: 82%|████████▏ | 475/579 [00:10<00:00, 128.55it/s] Loading 0: 84%|████████▍ | 489/579 [00:24<00:24, 3.71it/s] Loading 0: 88%|████████▊ | 508/579 [00:24<00:12, 5.56it/s] Loading 0: 90%|█████████ | 523/579 [00:25<00:07, 7.56it/s] Loading 0: 94%|█████████▍| 544/579 [00:25<00:03, 11.48it/s] Loading 0: 97%|█████████▋| 560/579 [00:25<00:01, 15.41it/s] Loading 0: 100%|█████████▉| 578/579 [00:25<00:00, 21.50it/s]
Job sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer completed after 114.69s with status: succeeded
Stopping job with name sao10k-14b-qwen2-5-kunou-v1-v1-mkmlizer
Pipeline stage MKMLizer completed in 115.21s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service sao10k-14b-qwen2-5-kunou-v1-v1
Waiting for inference service sao10k-14b-qwen2-5-kunou-v1-v1 to be ready
Inference service sao10k-14b-qwen2-5-kunou-v1-v1 ready after 180.8996546268463s
Pipeline stage MKMLDeployer completed in 181.35s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 12.195538473129272
10th percentile: 12.199869298934937
20th percentile: 12.208530950546265
30th percentile: 12.21427936553955
40th percentile: 12.217114543914795
50th percentile: 12.219949722290039
60th percentile: 12.232406997680664
70th percentile: 12.24486427307129
80th percentile: 13.82505660057068
90th percentile: 16.972983980178835
95th percentile: 18.54694766998291
99th percentile: 19.806118621826172
mean time: 13.799204683303833
%s, retrying in %s seconds...
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 12.192867851257324
10th percentile: 12.193566036224365
20th percentile: 12.194962406158448
30th percentile: 12.197382402420043
40th percentile: 12.200826025009155
50th percentile: 12.204269647598267
60th percentile: 12.205036973953247
70th percentile: 12.205804300308227
80th percentile: 12.206424617767334
90th percentile: 12.206897926330566
95th percentile: 12.207134580612182
99th percentile: 12.207323904037475
mean time: 12.201131820678711
%s, retrying in %s seconds...
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='sao10k-14b-qwen2-5-kunou-v1-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 12.19498438835144
10th percentile: 12.195876741409302
20th percentile: 12.197661447525025
30th percentile: 12.200070858001709
40th percentile: 12.203104972839355
50th percentile: 12.206139087677002
60th percentile: 12.212771701812745
70th percentile: 12.219404315948486
80th percentile: 12.225062799453735
90th percentile: 12.22974715232849
95th percentile: 12.23208932876587
99th percentile: 12.233963069915772
mean time: 12.211187410354615
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
sao10k-14b-qwen2-5-kunou-v1_v1 status is now failed due to DeploymentManager action