Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-72217-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-72217-v1-mkmlizer to finish
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-72217-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-72217-v1-mkmlizer: Downloaded to shared memory in 32.703s
alexdaoud-trainer-bagir-72217-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmppq9vqyjm, device:0
alexdaoud-trainer-bagir-72217-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-72217-v1-mkmlizer: quantized model in 86.637s
alexdaoud-trainer-bagir-72217-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-26 in 119.340s
alexdaoud-trainer-bagir-72217-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-72217-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-72217-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-72217-v1
alexdaoud-trainer-bagir-72217-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-72217-v1/config.json
alexdaoud-trainer-bagir-72217-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-72217-v1/special_tokens_map.json
alexdaoud-trainer-bagir-72217-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-72217-v1/tokenizer_config.json
alexdaoud-trainer-bagir-72217-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-72217-v1/tokenizer.json
alexdaoud-trainer-bagir-72217-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:56, 5.05it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:32, 3.10it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:04, 2.31it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:03, 4.43it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:02, 4.51it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:54, 5.14it/s]
Loading 0: 4%|▍ | 12/291 [00:02<01:05, 4.27it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:26, 3.21it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:49, 2.54it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:02, 4.37it/s]
Loading 0: 6%|▌ | 18/291 [00:04<00:59, 4.60it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:52, 5.18it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:02, 4.30it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:23, 3.24it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:44, 2.55it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:00, 4.35it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:57, 4.59it/s]
Loading 0: 10%|▉ | 28/291 [00:07<00:50, 5.20it/s]
Loading 0: 10%|█ | 30/291 [00:07<01:00, 4.32it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:19, 3.26it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:41, 2.56it/s]
Loading 0: 12%|█▏ | 35/291 [00:09<00:58, 4.36it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:55, 4.61it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:49, 5.14it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:58, 4.29it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:17, 3.25it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:38, 2.55it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:56, 4.34it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:53, 4.59it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:47, 5.12it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:56, 4.30it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:14, 3.25it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:34, 2.55it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:54, 4.34it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:51, 4.59it/s]
Loading 0: 19%|█▉ | 55/291 [00:13<00:45, 5.18it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:54, 4.32it/s]
Loading 0: 20%|█▉ | 58/291 [00:15<01:11, 3.26it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:30, 2.57it/s]
Loading 0: 21%|██▏ | 62/291 [00:16<00:52, 4.36it/s]
Loading 0: 22%|██▏ | 63/291 [00:16<00:49, 4.61it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:43, 5.19it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:52, 4.32it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:08, 3.25it/s]
Loading 0: 23%|██▎ | 68/291 [00:18<01:26, 2.57it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:50, 4.36it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:47, 4.60it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:42, 5.13it/s]
Loading 0: 26%|██▌ | 75/291 [00:19<00:50, 4.29it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:06, 3.24it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:23, 2.56it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:47, 4.40it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:45, 4.64it/s]
Loading 0: 28%|██▊ | 82/291 [00:20<00:39, 5.25it/s]
Loading 0: 29%|██▊ | 83/291 [00:21<00:38, 5.38it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<00:59, 3.49it/s]
Loading 0: 29%|██▉ | 85/291 [00:22<01:15, 2.73it/s]
Loading 0: 30%|██▉ | 86/291 [00:22<01:31, 2.24it/s]
Loading 0: 31%|███ | 89/291 [00:23<00:49, 4.12it/s]
Loading 0: 31%|███ | 90/291 [00:23<00:45, 4.40it/s]
Loading 0: 31%|███▏ | 91/291 [00:23<00:40, 5.00it/s]
Loading 0: 32%|███▏ | 93/291 [00:23<00:46, 4.21it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:01, 3.19it/s]
Loading 0: 33%|███▎ | 95/291 [00:25<01:18, 2.51it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:44, 4.30it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:42, 4.55it/s]
Loading 0: 34%|███▍ | 100/291 [00:25<00:37, 5.16it/s]
Loading 0: 35%|███▌ | 102/291 [00:26<00:43, 4.30it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:58, 3.24it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:13, 2.54it/s]
Loading 0: 37%|███▋ | 107/291 [00:27<00:42, 4.33it/s]
Loading 0: 37%|███▋ | 108/291 [00:27<00:39, 4.58it/s]
Loading 0: 37%|███▋ | 109/291 [00:28<00:35, 5.17it/s]
Loading 0: 38%|███▊ | 111/291 [00:28<00:41, 4.31it/s]
Loading 0: 38%|███▊ | 112/291 [00:29<00:55, 3.25it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:09, 2.57it/s]
Loading 0: 40%|███▉ | 116/291 [00:30<00:39, 4.43it/s]
Loading 0: 40%|████ | 117/291 [00:30<00:37, 4.66it/s]
Loading 0: 41%|████ | 119/291 [00:30<00:26, 6.41it/s]
Loading 0: 42%|████▏ | 121/291 [00:31<00:51, 3.33it/s]
Loading 0: 42%|████▏ | 122/291 [00:32<01:02, 2.71it/s]
Loading 0: 43%|████▎ | 125/291 [00:32<00:38, 4.30it/s]
Loading 0: 43%|████▎ | 126/291 [00:32<00:36, 4.53it/s]
Loading 0: 44%|████▎ | 127/291 [00:32<00:32, 5.09it/s]
Loading 0: 44%|████▍ | 129/291 [00:33<00:37, 4.31it/s]
Loading 0: 45%|████▍ | 130/291 [00:33<00:49, 3.28it/s]
Loading 0: 45%|████▌ | 131/291 [00:34<01:01, 2.60it/s]
Loading 0: 46%|████▌ | 134/291 [00:34<00:35, 4.41it/s]
Loading 0: 46%|████▋ | 135/291 [00:34<00:33, 4.65it/s]
Loading 0: 47%|████▋ | 136/291 [00:34<00:29, 5.23it/s]
Loading 0: 47%|████▋ | 138/291 [00:35<00:35, 4.35it/s]
Loading 0: 48%|████▊ | 139/291 [00:36<00:46, 3.27it/s]
Loading 0: 48%|████▊ | 140/291 [00:36<00:59, 2.56it/s]
Loading 0: 49%|████▉ | 143/291 [00:36<00:34, 4.34it/s]
Loading 0: 49%|████▉ | 144/291 [00:37<00:32, 4.58it/s]
Loading 0: 50%|████▉ | 145/291 [00:37<00:28, 5.06it/s]
Loading 0: 51%|█████ | 147/291 [00:37<00:33, 4.27it/s]
Loading 0: 51%|█████ | 148/291 [00:38<00:44, 3.23it/s]
Loading 0: 51%|█████ | 149/291 [00:39<00:55, 2.56it/s]
Loading 0: 52%|█████▏ | 152/291 [00:39<00:32, 4.34it/s]
Loading 0: 53%|█████▎ | 153/291 [00:39<00:30, 4.58it/s]
Loading 0: 53%|█████▎ | 154/291 [00:39<00:26, 5.16it/s]
Loading 0: 54%|█████▎ | 156/291 [00:40<00:31, 4.31it/s]
Loading 0: 54%|█████▍ | 157/291 [00:40<00:41, 3.25it/s]
Loading 0: 54%|█████▍ | 158/291 [00:41<00:51, 2.56it/s]
Loading 0: 55%|█████▌ | 161/291 [00:41<00:29, 4.41it/s]
Loading 0: 56%|█████▌ | 162/291 [00:41<00:27, 4.66it/s]
Loading 0: 56%|█████▋ | 164/291 [00:41<00:19, 6.40it/s]
Loading 0: 57%|█████▋ | 166/291 [00:43<00:37, 3.33it/s]
Loading 0: 57%|█████▋ | 167/291 [00:43<00:45, 2.72it/s]
Loading 0: 58%|█████▊ | 170/291 [00:43<00:27, 4.32it/s]
Loading 0: 59%|█████▉ | 171/291 [00:44<00:26, 4.55it/s]
Loading 0: 59%|█████▉ | 172/291 [00:44<00:23, 5.11it/s]
Loading 0: 59%|█████▉ | 173/291 [00:44<00:33, 3.54it/s]
Loading 0: 60%|██████ | 175/291 [00:44<00:24, 4.79it/s]
Loading 0: 60%|██████ | 176/291 [00:45<00:22, 5.01it/s]
Loading 0: 61%|██████ | 177/291 [00:45<00:20, 5.67it/s]
Loading 0: 62%|██████▏ | 179/291 [00:45<00:25, 4.47it/s]
Loading 0: 62%|██████▏ | 180/291 [00:46<00:33, 3.28it/s]
Loading 0: 62%|██████▏ | 181/291 [00:46<00:42, 2.57it/s]
Loading 0: 63%|██████▎ | 184/291 [00:47<00:24, 4.41it/s]
Loading 0: 64%|██████▎ | 185/291 [00:47<00:22, 4.66it/s]
Loading 0: 64%|██████▍ | 186/291 [00:47<00:20, 5.24it/s]
Loading 0: 64%|██████▍ | 187/291 [00:47<00:19, 5.47it/s]
Loading 0: 65%|██████▍ | 188/291 [00:48<00:29, 3.50it/s]
Loading 0: 65%|██████▍ | 189/291 [00:48<00:38, 2.62it/s]
Loading 0: 66%|██████▌ | 192/291 [00:49<00:27, 3.54it/s]
Loading 0: 66%|██████▋ | 193/291 [00:50<00:33, 2.91it/s]
Loading 0: 67%|██████▋ | 194/291 [00:50<00:39, 2.43it/s]
Loading 0: 68%|██████▊ | 197/291 [00:50<00:22, 4.10it/s]
Loading 0: 68%|██████▊ | 198/291 [00:51<00:21, 4.36it/s]
Loading 0: 68%|██████▊ | 199/291 [00:51<00:18, 4.95it/s]
Loading 0: 69%|██████▉ | 201/291 [00:51<00:21, 4.21it/s]
Loading 0: 69%|██████▉ | 202/291 [00:52<00:27, 3.23it/s]
Loading 0: 70%|██████▉ | 203/291 [00:52<00:34, 2.56it/s]
Loading 0: 71%|███████ | 206/291 [00:53<00:19, 4.33it/s]
Loading 0: 71%|███████ | 207/291 [00:53<00:18, 4.57it/s]
Loading 0: 71%|███████▏ | 208/291 [00:53<00:16, 5.15it/s]
Loading 0: 72%|███████▏ | 210/291 [00:54<00:18, 4.31it/s]
Loading 0: 73%|███████▎ | 211/291 [00:54<00:24, 3.25it/s]
Loading 0: 73%|███████▎ | 212/291 [00:55<00:30, 2.58it/s]
Loading 0: 74%|███████▍ | 215/291 [00:55<00:17, 4.38it/s]
Loading 0: 74%|███████▍ | 216/291 [00:55<00:16, 4.62it/s]
Loading 0: 75%|███████▍ | 217/291 [00:55<00:14, 5.18it/s]
Loading 0: 75%|███████▌ | 219/291 [00:56<00:16, 4.34it/s]
Loading 0: 76%|███████▌ | 220/291 [00:56<00:21, 3.27it/s]
Loading 0: 76%|███████▌ | 221/291 [00:57<00:27, 2.57it/s]
Loading 0: 77%|███████▋ | 224/291 [00:57<00:15, 4.38it/s]
Loading 0: 77%|███████▋ | 225/291 [00:57<00:14, 4.62it/s]
Loading 0: 78%|███████▊ | 226/291 [00:58<00:12, 5.18it/s]
Loading 0: 78%|███████▊ | 228/291 [00:58<00:14, 4.31it/s]
Loading 0: 79%|███████▊ | 229/291 [00:59<00:19, 3.26it/s]
Loading 0: 79%|███████▉ | 230/291 [00:59<00:23, 2.59it/s]
Loading 0: 80%|████████ | 233/291 [01:00<00:13, 4.45it/s]
Loading 0: 80%|████████ | 234/291 [01:00<00:12, 4.69it/s]
Loading 0: 81%|████████ | 235/291 [01:00<00:10, 5.27it/s]
Loading 0: 81%|████████▏ | 237/291 [01:00<00:12, 4.36it/s]
Loading 0: 82%|████████▏ | 238/291 [01:01<00:16, 3.27it/s]
Loading 0: 82%|████████▏ | 239/291 [01:02<00:20, 2.59it/s]
Loading 0: 83%|████████▎ | 242/291 [01:02<00:11, 4.44it/s]
Loading 0: 84%|████████▎ | 243/291 [01:02<00:10, 4.67it/s]
Loading 0: 84%|████████▍ | 244/291 [01:02<00:08, 5.26it/s]
Loading 0: 85%|████████▍ | 246/291 [01:03<00:10, 4.35it/s]
Loading 0: 85%|████████▍ | 247/291 [01:03<00:13, 3.28it/s]
Loading 0: 85%|████████▌ | 248/291 [01:04<00:16, 2.60it/s]
Loading 0: 86%|████████▋ | 251/291 [01:04<00:09, 4.40it/s]
Loading 0: 87%|████████▋ | 252/291 [01:04<00:08, 4.64it/s]
Loading 0: 87%|████████▋ | 253/291 [01:04<00:07, 5.20it/s]
Loading 0: 88%|████████▊ | 255/291 [01:05<00:08, 4.32it/s]
Loading 0: 88%|████████▊ | 256/291 [01:06<00:10, 3.26it/s]
Loading 0: 88%|████████▊ | 257/291 [01:06<00:13, 2.59it/s]
Loading 0: 89%|████████▉ | 260/291 [01:06<00:07, 4.40it/s]
Loading 0: 90%|████████▉ | 261/291 [01:07<00:06, 4.64it/s]
Loading 0: 90%|█████████ | 262/291 [01:07<00:05, 5.21it/s]
Loading 0: 91%|█████████ | 264/291 [01:07<00:06, 4.34it/s]
Loading 0: 91%|█████████ | 265/291 [01:08<00:07, 3.26it/s]
Loading 0: 91%|█████████▏| 266/291 [01:09<00:09, 2.58it/s]
Loading 0: 92%|█████████▏| 269/291 [01:09<00:04, 4.44it/s]
Loading 0: 93%|█████████▎| 270/291 [01:09<00:04, 4.66it/s]
Loading 0: 93%|█████████▎| 271/291 [01:09<00:03, 5.22it/s]
Loading 0: 94%|█████████▍| 273/291 [01:10<00:04, 4.35it/s]
Loading 0: 94%|█████████▍| 274/291 [01:10<00:05, 3.28it/s]
Loading 0: 95%|█████████▍| 275/291 [01:11<00:06, 2.58it/s]
Loading 0: 96%|█████████▌| 278/291 [01:11<00:02, 4.43it/s]
Loading 0: 96%|█████████▌| 279/291 [01:11<00:02, 4.67it/s]
Loading 0: 96%|█████████▌| 280/291 [01:11<00:02, 5.20it/s]
Loading 0: 97%|█████████▋| 281/291 [01:12<00:02, 3.50it/s]
Loading 0: 97%|█████████▋| 282/291 [01:13<00:03, 2.65it/s]
Loading 0: 98%|█████████▊| 284/291 [01:13<00:01, 3.81it/s]
Loading 0: 98%|█████████▊| 285/291 [01:13<00:01, 4.16it/s]
Loading 0: 98%|█████████▊| 286/291 [01:13<00:01, 4.83it/s]
Loading 0: 99%|█████████▊| 287/291 [01:13<00:00, 4.85it/s]
Loading 0: 99%|█████████▉| 288/291 [01:14<00:00, 3.24it/s]
Job alexdaoud-trainer-bagir-72217-v1-mkmlizer completed after 145.66s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-72217-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.54s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-72217-v1
Waiting for inference service alexdaoud-trainer-bagir-72217-v1 to be ready
Inference service alexdaoud-trainer-bagir-72217-v1 ready after 291.59617710113525s
Pipeline stage MKMLDeployer completed in 292.15s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 7.2416768074035645s
Received healthy response to inference request in 3.440035343170166s
Received healthy response to inference request in 2.9745256900787354s
Received healthy response to inference request in 6.083128213882446s
Received healthy response to inference request in 2.5806944370269775s
5 requests
0 failed requests
5th percentile: 2.659460687637329
10th percentile: 2.7382269382476805
20th percentile: 2.895759439468384
30th percentile: 3.0676276206970217
40th percentile: 3.253831481933594
50th percentile: 3.440035343170166
60th percentile: 4.497272491455078
70th percentile: 5.55450963973999
80th percentile: 6.31483793258667
90th percentile: 6.778257369995117
95th percentile: 7.009967088699341
99th percentile: 7.19533486366272
mean time: 4.464012098312378
%s, retrying in %s seconds...
Received healthy response to inference request in 5.144574165344238s
Received healthy response to inference request in 3.154587507247925s
Received healthy response to inference request in 3.5878794193267822s
Received healthy response to inference request in 4.126107454299927s
Received healthy response to inference request in 4.943214654922485s
5 requests
0 failed requests
5th percentile: 3.241245889663696
10th percentile: 3.3279042720794676
20th percentile: 3.501221036911011
30th percentile: 3.695525026321411
40th percentile: 3.9108162403106688
50th percentile: 4.126107454299927
60th percentile: 4.45295033454895
70th percentile: 4.779793214797974
80th percentile: 4.983486557006836
90th percentile: 5.064030361175537
95th percentile: 5.104302263259887
99th percentile: 5.136519784927368
mean time: 4.191272640228272
%s, retrying in %s seconds...
Received healthy response to inference request in 3.0363430976867676s
Received healthy response to inference request in 4.891549825668335s
Received healthy response to inference request in 3.6572558879852295s
Received healthy response to inference request in 2.906693458557129s
Received healthy response to inference request in 3.298729419708252s
5 requests
0 failed requests
5th percentile: 2.9326233863830566
10th percentile: 2.9585533142089844
20th percentile: 3.01041316986084
30th percentile: 3.0888203620910644
40th percentile: 3.1937748908996584
50th percentile: 3.298729419708252
60th percentile: 3.442140007019043
70th percentile: 3.585550594329834
80th percentile: 3.9041146755218508
90th percentile: 4.397832250595092
95th percentile: 4.644691038131714
99th percentile: 4.842178068161011
mean time: 3.5581143379211424
Pipeline stage StressChecker completed in 65.62s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.69s
Shutdown handler de-registered
alexdaoud-trainer-bagir_72217_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-72217-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-72217-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-72217-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 3.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-72217-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-72217-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-72217-v1-profiler ready after 250.53241300582886s
Pipeline stage MKMLProfilerDeployer completed in 250.85s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deplovbx26:/code/chaiverse_profiler_1735417849 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deplovbx26 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1735417849 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1735417849/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deplovbx26:/code/chaiverse_profiler_1735420647 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deplovbx26:/code/chaiverse_profiler_1735420648 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deplovbx26 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1735420648 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1735420648/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-72217-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-72217-v1-profiler
Service alexdaoud-trainer-bagir-72217-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.24s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-72217-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 3.29s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-72217-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-72217-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-72217-v1-profiler ready after 230.5001769065857s
Pipeline stage MKMLProfilerDeployer completed in 231.05s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deploszgzf:/code/chaiverse_profiler_1735421445 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deploszgzf --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1735421445 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1735421445/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deploszgzf:/code/chaiverse_profiler_1735424221 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba2950dd44779fbfc247542ff4e753aee4-deploszgzf --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1735424221 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1735424221/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-72217-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-72217-v1-profiler
Service alexdaoud-trainer-bagir-72217-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.19s
Shutdown handler de-registered
alexdaoud-trainer-bagir_72217_v1 status is now inactive due to auto deactivation removed underperforming models
alexdaoud-trainer-bagir_72217_v1 status is now torndown due to DeploymentManager action
Shutdown handler not registered because Python interpreter is not running in the main thread
Deleting key chaiml-test-configurabl-44178-v5/config.json from bucket guanaco-mkml-models
alexdaoud-trainer-bagir_72217_v1 status is now torndown due to DeploymentManager action
run pipeline %s
alexdaoud-trainer-bagir_72217_v1 status is now torndown due to DeploymentManager action