Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-lexical-nemov-5966-v10-mkmlizer
Waiting for job on zonemercy-lexical-nemov-5966-v10-mkmlizer to finish
zonemercy-lexical-nemov-5966-v10-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ _____ __ __ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ /___/ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ Version: 0.10.1 ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ https://mk1.ai ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ belonging to: ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ Chai Research Corp. ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ║ ║
zonemercy-lexical-nemov-5966-v10-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-lexical-nemov-5966-v10-mkmlizer: Downloaded to shared memory in 97.921s
zonemercy-lexical-nemov-5966-v10-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpbiqxepbz, device:0
zonemercy-lexical-nemov-5966-v10-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-lexical-nemov-5966-v10-mkmlizer: quantized model in 42.159s
zonemercy-lexical-nemov-5966-v10-mkmlizer: Processed model zonemercy/Lexical-Nemov8-1k1e5 in 140.081s
zonemercy-lexical-nemov-5966-v10-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-lexical-nemov-5966-v10-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-lexical-nemov-5966-v10-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10
zonemercy-lexical-nemov-5966-v10-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10/config.json
zonemercy-lexical-nemov-5966-v10-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10/special_tokens_map.json
zonemercy-lexical-nemov-5966-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10/tokenizer_config.json
zonemercy-lexical-nemov-5966-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10/tokenizer.json
zonemercy-lexical-nemov-5966-v10-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-lexical-nemov-5966-v10/flywheel_model.0.safetensors
zonemercy-lexical-nemov-5966-v10-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:16, 22.21it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:12, 28.62it/s]
Loading 0: 4%|▍ | 14/363 [00:00<00:13, 25.46it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:09, 36.76it/s]
Loading 0: 7%|▋ | 26/363 [00:01<00:15, 21.29it/s]
Loading 0: 9%|▊ | 31/363 [00:01<00:12, 25.87it/s]
Loading 0: 10%|▉ | 35/363 [00:01<00:12, 26.90it/s]
Loading 0: 11%|█ | 39/363 [00:01<00:11, 28.29it/s]
Loading 0: 12%|█▏ | 43/363 [00:01<00:11, 27.78it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:10, 30.57it/s]
Loading 0: 14%|█▍ | 52/363 [00:01<00:10, 29.01it/s]
Loading 0: 15%|█▌ | 56/363 [00:02<00:10, 29.86it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:11, 25.89it/s]
Loading 0: 18%|█▊ | 64/363 [00:02<00:13, 22.20it/s]
Loading 0: 20%|█▉ | 71/363 [00:02<00:10, 29.16it/s]
Loading 0: 21%|██ | 75/363 [00:02<00:09, 28.99it/s]
Loading 0: 22%|██▏ | 79/363 [00:02<00:10, 28.18it/s]
Loading 0: 23%|██▎ | 84/363 [00:03<00:09, 30.87it/s]
Loading 0: 24%|██▍ | 88/363 [00:03<00:09, 29.10it/s]
Loading 0: 25%|██▌ | 92/363 [00:03<00:08, 31.30it/s]
Loading 0: 26%|██▋ | 96/363 [00:03<00:10, 26.26it/s]
Loading 0: 28%|██▊ | 101/363 [00:03<00:11, 23.56it/s]
Loading 0: 29%|██▊ | 104/363 [00:03<00:12, 21.23it/s]
Loading 0: 31%|███ | 111/363 [00:04<00:08, 28.38it/s]
Loading 0: 32%|███▏ | 115/363 [00:04<00:08, 27.77it/s]
Loading 0: 33%|███▎ | 120/363 [00:04<00:07, 30.50it/s]
Loading 0: 34%|███▍ | 124/363 [00:04<00:08, 29.06it/s]
Loading 0: 36%|███▌ | 129/363 [00:04<00:07, 31.93it/s]
Loading 0: 37%|███▋ | 133/363 [00:04<00:07, 30.61it/s]
Loading 0: 38%|███▊ | 137/363 [00:04<00:07, 30.61it/s]
Loading 0: 39%|███▉ | 142/363 [00:05<00:08, 26.01it/s]
Loading 0: 40%|███▉ | 145/363 [00:05<00:08, 24.35it/s]
Loading 0: 41%|████ | 149/363 [00:05<00:09, 22.66it/s]
Loading 0: 43%|████▎ | 156/363 [00:05<00:06, 29.99it/s]
Loading 0: 44%|████▍ | 160/363 [00:05<00:06, 29.68it/s]
Loading 0: 45%|████▌ | 165/363 [00:05<00:06, 31.79it/s]
Loading 0: 47%|████▋ | 169/363 [00:06<00:06, 30.41it/s]
Loading 0: 48%|████▊ | 174/363 [00:06<00:05, 32.02it/s]
Loading 0: 49%|████▉ | 178/363 [00:06<00:06, 28.51it/s]
Loading 0: 50%|████▉ | 181/363 [00:06<00:06, 27.80it/s]
Loading 0: 51%|█████ | 184/363 [00:06<00:08, 20.43it/s]
Loading 0: 52%|█████▏ | 187/363 [00:06<00:08, 21.71it/s]
Loading 0: 53%|█████▎ | 192/363 [00:07<00:06, 26.07it/s]
Loading 0: 54%|█████▎ | 195/363 [00:07<00:06, 24.72it/s]
Loading 0: 55%|█████▌ | 201/363 [00:07<00:05, 30.39it/s]
Loading 0: 56%|█████▋ | 205/363 [00:07<00:05, 28.76it/s]
Loading 0: 58%|█████▊ | 210/363 [00:07<00:04, 31.25it/s]
Loading 0: 59%|█████▉ | 214/363 [00:07<00:05, 29.33it/s]
Loading 0: 60%|██████ | 218/363 [00:07<00:04, 29.11it/s]
Loading 0: 61%|██████ | 222/363 [00:07<00:04, 30.93it/s]
Loading 0: 62%|██████▏ | 226/363 [00:08<00:06, 20.63it/s]
Loading 0: 63%|██████▎ | 230/363 [00:08<00:06, 20.82it/s]
Loading 0: 65%|██████▌ | 237/363 [00:08<00:04, 27.32it/s]
Loading 0: 66%|██████▋ | 241/363 [00:08<00:04, 26.90it/s]
Loading 0: 68%|██████▊ | 246/363 [00:08<00:04, 29.19it/s]
Loading 0: 69%|██████▉ | 250/363 [00:09<00:04, 27.65it/s]
Loading 0: 70%|███████ | 255/363 [00:09<00:03, 29.09it/s]
Loading 0: 71%|███████▏ | 259/363 [00:09<00:03, 27.86it/s]
Loading 0: 72%|███████▏ | 263/363 [00:09<00:04, 22.59it/s]
Loading 0: 73%|███████▎ | 266/363 [00:09<00:04, 19.90it/s]
Loading 0: 75%|███████▍ | 271/363 [00:10<00:03, 25.24it/s]
Loading 0: 76%|███████▌ | 275/363 [00:10<00:03, 23.15it/s]
Loading 0: 77%|███████▋ | 280/363 [00:10<00:03, 27.63it/s]
Loading 0: 78%|███████▊ | 284/363 [00:10<00:03, 24.66it/s]
Loading 0: 80%|███████▉ | 289/363 [00:10<00:02, 29.31it/s]
Loading 0: 81%|████████ | 293/363 [00:10<00:02, 25.17it/s]
Loading 0: 82%|████████▏ | 298/363 [00:10<00:02, 29.72it/s]
Loading 0: 83%|████████▎ | 303/363 [00:11<00:01, 31.34it/s]
Loading 0: 85%|████████▍ | 307/363 [00:11<00:02, 21.06it/s]
Loading 0: 86%|████████▌ | 311/363 [00:11<00:02, 20.75it/s]
Loading 0: 88%|████████▊ | 318/363 [00:11<00:01, 27.17it/s]
Loading 0: 89%|████████▊ | 322/363 [00:11<00:01, 26.48it/s]
Loading 0: 90%|█████████ | 327/363 [00:12<00:01, 29.08it/s]
Loading 0: 91%|█████████ | 331/363 [00:12<00:01, 28.25it/s]
Loading 0: 93%|█████████▎| 336/363 [00:12<00:00, 31.32it/s]
Loading 0: 94%|█████████▎| 340/363 [00:12<00:00, 30.02it/s]
Loading 0: 95%|█████████▍| 344/363 [00:19<00:10, 1.88it/s]
Loading 0: 96%|█████████▌| 348/363 [00:20<00:05, 2.53it/s]
Loading 0: 97%|█████████▋| 353/363 [00:20<00:02, 3.68it/s]
Loading 0: 98%|█████████▊| 357/363 [00:20<00:01, 4.79it/s]
Job zonemercy-lexical-nemov-5966-v10-mkmlizer completed after 165.54s with status: succeeded
Stopping job with name zonemercy-lexical-nemov-5966-v10-mkmlizer
Pipeline stage MKMLizer completed in 166.63s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-lexical-nemov-5966-v10
Waiting for inference service zonemercy-lexical-nemov-5966-v10 to be ready
Inference service zonemercy-lexical-nemov-5966-v10 ready after 221.49981713294983s
Pipeline stage MKMLDeployer completed in 221.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0271823406219482s
Received healthy response to inference request in 1.9992289543151855s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.0890915393829346s
Received healthy response to inference request in 1.458916187286377s
Received healthy response to inference request in 1.6730072498321533s
5 requests
0 failed requests
5th percentile: 1.5017343997955321
10th percentile: 1.5445526123046875
20th percentile: 1.6301890373229981
30th percentile: 1.7382515907287597
40th percentile: 1.8687402725219726
50th percentile: 1.9992289543151855
60th percentile: 2.0104103088378906
70th percentile: 2.0215916633605957
80th percentile: 2.0395641803741453
90th percentile: 2.06432785987854
95th percentile: 2.0767096996307375
99th percentile: 2.086615171432495
mean time: 1.8494852542877198
Pipeline stage StressChecker completed in 10.96s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.60s
Shutdown handler de-registered
zonemercy-lexical-nemov_5966_v10 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-lexical-nemov-5966-v10-profiler
Waiting for inference service zonemercy-lexical-nemov-5966-v10-profiler to be ready
Inference service zonemercy-lexical-nemov-5966-v10-profiler ready after 210.58290195465088s
Pipeline stage MKMLProfilerDeployer completed in 210.93s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/zonemercy-lexical-nec3725bd01983d78d6b1496b71af07969-deplozv8ks:/code/chaiverse_profiler_1726851774 --namespace tenant-chaiml-guanaco
kubectl exec -it zonemercy-lexical-nec3725bd01983d78d6b1496b71af07969-deplozv8ks --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726851774 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726851774/summary.json'
kubectl exec -it zonemercy-lexical-nec3725bd01983d78d6b1496b71af07969-deplozv8ks --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726851774/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1156.94s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service zonemercy-lexical-nemov-5966-v10-profiler is running
Tearing down inference service zonemercy-lexical-nemov-5966-v10-profiler
Service zonemercy-lexical-nemov-5966-v10-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.33s
Shutdown handler de-registered
zonemercy-lexical-nemov_5966_v10 status is now inactive due to auto deactivation removed underperforming models
zonemercy-lexical-nemov_5966_v10 status is now torndown due to DeploymentManager action