Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer
Waiting for job on sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer to finish
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ _____ __ __ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ /___/ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ Version: 0.11.12 ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ https://mk1.ai ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ The license key for the current software has been verified as ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ belonging to: ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ Chai Research Corp. ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ║ ║
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: Downloaded to shared memory in 31.491s
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpk54jnq5t, device:0
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: quantized model in 35.275s
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: Processed model Sao10K/MN-12B-Lyra-v4a1-Old in 66.766s
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: creating bucket guanaco-mkml-models
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6/config.json
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6/special_tokens_map.json
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6/tokenizer_config.json
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6/tokenizer.json
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/sao10k-mn-12b-lyra-v4a1-old-v6/flywheel_model.0.safetensors
sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:06<18:33, 3.08s/it]
Loading 0: 2%|▏ | 6/363 [00:06<04:54, 1.21it/s]
Loading 0: 4%|▎ | 13/363 [00:06<01:44, 3.33it/s]
Loading 0: 5%|▍ | 18/363 [00:06<01:05, 5.31it/s]
Loading 0: 7%|▋ | 24/363 [00:06<00:41, 8.08it/s]
Loading 0: 9%|▉ | 32/363 [00:06<00:25, 13.21it/s]
Loading 0: 10%|█ | 38/363 [00:06<00:19, 16.80it/s]
Loading 0: 12%|█▏ | 43/363 [00:07<00:15, 20.15it/s]
Loading 0: 14%|█▍ | 50/363 [00:07<00:11, 26.33it/s]
Loading 0: 15%|█▌ | 56/363 [00:07<00:10, 29.30it/s]
Loading 0: 17%|█▋ | 61/363 [00:07<00:14, 21.02it/s]
Loading 0: 18%|█▊ | 67/363 [00:07<00:11, 26.22it/s]
Loading 0: 20%|█▉ | 72/363 [00:07<00:09, 29.49it/s]
Loading 0: 21%|██ | 77/363 [00:08<00:08, 33.29it/s]
Loading 0: 23%|██▎ | 83/363 [00:08<00:07, 35.37it/s]
Loading 0: 24%|██▍ | 88/363 [00:08<00:07, 37.43it/s]
Loading 0: 26%|██▌ | 94/363 [00:08<00:06, 42.02it/s]
Loading 0: 27%|██▋ | 99/363 [00:08<00:06, 43.33it/s]
Loading 0: 29%|██▊ | 104/363 [00:08<00:05, 44.70it/s]
Loading 0: 30%|███ | 110/363 [00:08<00:05, 42.74it/s]
Loading 0: 32%|███▏ | 115/363 [00:08<00:05, 41.92it/s]
Loading 0: 34%|███▎ | 122/363 [00:09<00:05, 47.03it/s]
Loading 0: 35%|███▌ | 128/363 [00:09<00:05, 44.64it/s]
Loading 0: 37%|███▋ | 133/363 [00:09<00:05, 44.11it/s]
Loading 0: 38%|███▊ | 139/363 [00:09<00:04, 47.42it/s]
Loading 0: 40%|███▉ | 144/363 [00:09<00:04, 47.48it/s]
Loading 0: 41%|████ | 149/363 [00:09<00:04, 47.40it/s]
Loading 0: 43%|████▎ | 155/363 [00:09<00:04, 44.75it/s]
Loading 0: 44%|████▍ | 160/363 [00:10<00:07, 25.79it/s]
Loading 0: 46%|████▌ | 166/363 [00:10<00:06, 31.17it/s]
Loading 0: 47%|████▋ | 171/363 [00:10<00:05, 33.84it/s]
Loading 0: 48%|████▊ | 176/363 [00:10<00:05, 36.61it/s]
Loading 0: 50%|█████ | 182/363 [00:10<00:04, 37.63it/s]
Loading 0: 52%|█████▏ | 187/363 [00:10<00:04, 38.58it/s]
Loading 0: 53%|█████▎ | 193/363 [00:10<00:03, 42.68it/s]
Loading 0: 55%|█████▍ | 198/363 [00:11<00:03, 43.98it/s]
Loading 0: 56%|█████▌ | 203/363 [00:11<00:03, 44.61it/s]
Loading 0: 58%|█████▊ | 209/363 [00:11<00:03, 42.71it/s]
Loading 0: 59%|█████▉ | 214/363 [00:11<00:03, 42.13it/s]
Loading 0: 61%|██████ | 220/363 [00:11<00:03, 46.35it/s]
Loading 0: 62%|██████▏ | 225/363 [00:11<00:02, 46.61it/s]
Loading 0: 63%|██████▎ | 230/363 [00:11<00:02, 46.68it/s]
Loading 0: 65%|██████▌ | 236/363 [00:11<00:02, 43.22it/s]
Loading 0: 66%|██████▋ | 241/363 [00:11<00:02, 42.72it/s]
Loading 0: 68%|██████▊ | 247/363 [00:12<00:02, 47.10it/s]
Loading 0: 69%|██████▉ | 252/363 [00:12<00:02, 45.98it/s]
Loading 0: 71%|███████ | 257/363 [00:12<00:04, 25.53it/s]
Loading 0: 72%|███████▏ | 263/363 [00:12<00:03, 28.62it/s]
Loading 0: 74%|███████▎ | 267/363 [00:12<00:03, 29.20it/s]
Loading 0: 75%|███████▌ | 274/363 [00:13<00:02, 37.25it/s]
Loading 0: 77%|███████▋ | 279/363 [00:13<00:02, 39.50it/s]
Loading 0: 78%|███████▊ | 284/363 [00:13<00:01, 40.95it/s]
Loading 0: 80%|███████▉ | 290/363 [00:13<00:01, 40.54it/s]
Loading 0: 81%|████████▏ | 295/363 [00:13<00:01, 40.31it/s]
Loading 0: 83%|████████▎ | 301/363 [00:13<00:01, 44.96it/s]
Loading 0: 84%|████████▍ | 306/363 [00:13<00:01, 45.20it/s]
Loading 0: 86%|████████▌ | 311/363 [00:13<00:01, 40.78it/s]
Loading 0: 87%|████████▋ | 317/363 [00:14<00:01, 40.59it/s]
Loading 0: 89%|████████▊ | 322/363 [00:14<00:01, 40.57it/s]
Loading 0: 90%|█████████ | 328/363 [00:14<00:00, 44.86it/s]
Loading 0: 92%|█████████▏| 333/363 [00:14<00:00, 45.72it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:00, 45.26it/s]
Loading 0: 94%|█████████▍| 343/363 [00:14<00:00, 44.44it/s]
Loading 0: 96%|█████████▌| 348/363 [00:14<00:00, 36.92it/s]
Loading 0: 98%|█████████▊| 355/363 [00:15<00:00, 27.16it/s]
Loading 0: 99%|█████████▉| 359/363 [00:15<00:00, 28.95it/s]
Job sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer completed after 93.79s with status: succeeded
Stopping job with name sao10k-mn-12b-lyra-v4a1-old-v6-mkmlizer
Pipeline stage MKMLizer completed in 94.21s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service sao10k-mn-12b-lyra-v4a1-old-v6
Waiting for inference service sao10k-mn-12b-lyra-v4a1-old-v6 to be ready
Inference service sao10k-mn-12b-lyra-v4a1-old-v6 ready after 160.974426984787s
Pipeline stage MKMLDeployer completed in 161.35s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.018420934677124s
Received healthy response to inference request in 2.084780216217041s
Received healthy response to inference request in 1.8015673160552979s
Received healthy response to inference request in 1.6317853927612305s
5 requests
1 failed requests
5th percentile: 1.665741777420044
10th percentile: 1.6996981620788574
20th percentile: 1.7676109313964843
30th percentile: 1.844938039779663
40th percentile: 1.9316794872283936
50th percentile: 2.018420934677124
60th percentile: 2.044964647293091
70th percentile: 2.071508359909058
80th percentile: 5.690303707122806
90th percentile: 12.901350688934329
95th percentile: 16.506874179840086
99th percentile: 19.391292972564695
mean time: 5.529790306091309
%s, retrying in %s seconds...
Received healthy response to inference request in 2.144949436187744s
Received healthy response to inference request in 1.7445158958435059s
Received healthy response to inference request in 1.8631749153137207s
Received healthy response to inference request in 1.6464800834655762s
Received healthy response to inference request in 1.3971774578094482s
5 requests
0 failed requests
5th percentile: 1.4470379829406739
10th percentile: 1.4968985080718995
20th percentile: 1.5966195583343505
30th percentile: 1.6660872459411622
40th percentile: 1.705301570892334
50th percentile: 1.7445158958435059
60th percentile: 1.7919795036315918
70th percentile: 1.8394431114196776
80th percentile: 1.9195298194885255
90th percentile: 2.032239627838135
95th percentile: 2.0885945320129395
99th percentile: 2.133678455352783
mean time: 1.7592595577239991
Pipeline stage StressChecker completed in 38.56s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.59s
Shutdown handler de-registered
sao10k-mn-12b-lyra-v4a1-old_v6 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3311.17s
Shutdown handler de-registered
sao10k-mn-12b-lyra-v4a1-old_v6 status is now inactive due to auto deactivation removed underperforming models
sao10k-mn-12b-lyra-v4a1-old_v6 status is now torndown due to DeploymentManager action