Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-8476-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-8476-v1-mkmlizer to finish
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8476-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-8476-v1-mkmlizer: Downloaded to shared memory in 33.468s
alexdaoud-trainer-bagir-8476-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmprrc3upty, device:0
alexdaoud-trainer-bagir-8476-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-8476-v1-mkmlizer: quantized model in 87.511s
alexdaoud-trainer-bagir-8476-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-11 in 120.979s
alexdaoud-trainer-bagir-8476-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-8476-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-8476-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1
alexdaoud-trainer-bagir-8476-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1/config.json
alexdaoud-trainer-bagir-8476-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1/special_tokens_map.json
alexdaoud-trainer-bagir-8476-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1/tokenizer_config.json
alexdaoud-trainer-bagir-8476-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1/tokenizer.json
alexdaoud-trainer-bagir-8476-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8476-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-8476-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:57, 5.00it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:33, 3.06it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:05, 2.28it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:04, 4.36it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:03, 4.43it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:56, 4.99it/s]
Loading 0: 4%|▍ | 12/291 [00:03<01:07, 4.11it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:29, 3.11it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:52, 2.46it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:04, 4.24it/s]
Loading 0: 6%|▌ | 18/291 [00:04<01:00, 4.48it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:53, 5.05it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:04, 4.21it/s]
Loading 0: 8%|▊ | 22/291 [00:06<01:24, 3.18it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:46, 2.51it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:01, 4.28it/s]
Loading 0: 9%|▉ | 27/291 [00:07<00:58, 4.51it/s]
Loading 0: 10%|▉ | 28/291 [00:07<00:52, 5.03it/s]
Loading 0: 10%|█ | 30/291 [00:07<01:01, 4.23it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:21, 3.20it/s]
Loading 0: 11%|█ | 32/291 [00:09<01:42, 2.53it/s]
Loading 0: 12%|█▏ | 35/291 [00:09<00:58, 4.36it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:54, 4.67it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:48, 5.20it/s]
Loading 0: 13%|█▎ | 39/291 [00:10<00:58, 4.32it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:17, 3.24it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:39, 2.52it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:56, 4.34it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:53, 4.58it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:47, 5.18it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:56, 4.29it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:15, 3.22it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:34, 2.54it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:54, 4.34it/s]
Loading 0: 19%|█▊ | 54/291 [00:14<00:51, 4.57it/s]
Loading 0: 19%|█▉ | 55/291 [00:14<00:45, 5.17it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:54, 4.28it/s]
Loading 0: 20%|█▉ | 58/291 [00:15<01:12, 3.22it/s]
Loading 0: 20%|██ | 59/291 [00:16<01:31, 2.52it/s]
Loading 0: 21%|██▏ | 62/291 [00:16<00:53, 4.29it/s]
Loading 0: 22%|██▏ | 63/291 [00:16<00:50, 4.53it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:44, 5.10it/s]
Loading 0: 23%|██▎ | 66/291 [00:17<00:52, 4.26it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:09, 3.22it/s]
Loading 0: 23%|██▎ | 68/291 [00:18<01:27, 2.55it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:50, 4.38it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:47, 4.62it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:41, 5.20it/s]
Loading 0: 26%|██▌ | 75/291 [00:19<00:50, 4.31it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:06, 3.26it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:23, 2.55it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:48, 4.39it/s]
Loading 0: 28%|██▊ | 81/291 [00:21<00:45, 4.62it/s]
Loading 0: 28%|██▊ | 82/291 [00:21<00:40, 5.22it/s]
Loading 0: 29%|██▊ | 83/291 [00:21<00:42, 4.91it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<01:02, 3.33it/s]
Loading 0: 29%|██▉ | 85/291 [00:22<01:17, 2.65it/s]
Loading 0: 30%|██▉ | 86/291 [00:23<01:33, 2.19it/s]
Loading 0: 31%|███ | 89/291 [00:23<00:50, 4.04it/s]
Loading 0: 31%|███ | 90/291 [00:23<00:46, 4.32it/s]
Loading 0: 32%|███▏ | 92/291 [00:23<00:32, 6.08it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:01, 3.22it/s]
Loading 0: 33%|███▎ | 95/291 [00:25<01:14, 2.64it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:45, 4.26it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:42, 4.49it/s]
Loading 0: 34%|███▍ | 100/291 [00:25<00:38, 5.01it/s]
Loading 0: 35%|███▌ | 102/291 [00:26<00:44, 4.24it/s]
Loading 0: 35%|███▌ | 103/291 [00:27<00:58, 3.24it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:12, 2.56it/s]
Loading 0: 37%|███▋ | 107/291 [00:28<00:42, 4.33it/s]
Loading 0: 37%|███▋ | 108/291 [00:28<00:40, 4.57it/s]
Loading 0: 37%|███▋ | 109/291 [00:28<00:35, 5.16it/s]
Loading 0: 38%|███▊ | 111/291 [00:28<00:41, 4.29it/s]
Loading 0: 38%|███▊ | 112/291 [00:29<00:54, 3.26it/s]
Loading 0: 39%|███▉ | 113/291 [00:30<01:09, 2.57it/s]
Loading 0: 40%|███▉ | 116/291 [00:30<00:40, 4.33it/s]
Loading 0: 40%|████ | 117/291 [00:30<00:38, 4.57it/s]
Loading 0: 41%|████ | 118/291 [00:30<00:33, 5.14it/s]
Loading 0: 41%|████ | 120/291 [00:31<00:39, 4.28it/s]
Loading 0: 42%|████▏ | 121/291 [00:31<00:52, 3.23it/s]
Loading 0: 42%|████▏ | 122/291 [00:32<01:06, 2.56it/s]
Loading 0: 43%|████▎ | 125/291 [00:32<00:37, 4.38it/s]
Loading 0: 43%|████▎ | 126/291 [00:32<00:35, 4.61it/s]
Loading 0: 44%|████▎ | 127/291 [00:32<00:31, 5.19it/s]
Loading 0: 44%|████▍ | 129/291 [00:33<00:37, 4.29it/s]
Loading 0: 45%|████▍ | 130/291 [00:34<00:49, 3.24it/s]
Loading 0: 45%|████▌ | 131/291 [00:34<01:03, 2.54it/s]
Loading 0: 46%|████▌ | 134/291 [00:34<00:35, 4.37it/s]
Loading 0: 46%|████▋ | 135/291 [00:35<00:33, 4.59it/s]
Loading 0: 47%|████▋ | 136/291 [00:35<00:29, 5.20it/s]
Loading 0: 47%|████▋ | 138/291 [00:35<00:35, 4.33it/s]
Loading 0: 48%|████▊ | 139/291 [00:36<00:46, 3.25it/s]
Loading 0: 48%|████▊ | 140/291 [00:37<00:59, 2.54it/s]
Loading 0: 49%|████▉ | 143/291 [00:37<00:34, 4.35it/s]
Loading 0: 49%|████▉ | 144/291 [00:37<00:32, 4.56it/s]
Loading 0: 50%|████▉ | 145/291 [00:37<00:28, 5.15it/s]
Loading 0: 51%|█████ | 147/291 [00:38<00:33, 4.30it/s]
Loading 0: 51%|█████ | 148/291 [00:38<00:44, 3.24it/s]
Loading 0: 51%|█████ | 149/291 [00:39<00:55, 2.56it/s]
Loading 0: 52%|█████▏ | 152/291 [00:39<00:31, 4.36it/s]
Loading 0: 53%|█████▎ | 153/291 [00:39<00:30, 4.60it/s]
Loading 0: 53%|█████▎ | 154/291 [00:39<00:26, 5.19it/s]
Loading 0: 54%|█████▎ | 156/291 [00:40<00:31, 4.32it/s]
Loading 0: 54%|█████▍ | 157/291 [00:41<00:41, 3.24it/s]
Loading 0: 54%|█████▍ | 158/291 [00:41<00:51, 2.57it/s]
Loading 0: 55%|█████▌ | 161/291 [00:41<00:29, 4.35it/s]
Loading 0: 56%|█████▌ | 162/291 [00:42<00:28, 4.58it/s]
Loading 0: 56%|█████▌ | 163/291 [00:42<00:24, 5.18it/s]
Loading 0: 57%|█████▋ | 165/291 [00:42<00:29, 4.32it/s]
Loading 0: 57%|█████▋ | 166/291 [00:43<00:38, 3.26it/s]
Loading 0: 57%|█████▋ | 167/291 [00:44<00:48, 2.58it/s]
Loading 0: 58%|█████▊ | 170/291 [00:44<00:27, 4.43it/s]
Loading 0: 59%|█████▉ | 171/291 [00:44<00:25, 4.66it/s]
Loading 0: 59%|█████▉ | 172/291 [00:44<00:22, 5.27it/s]
Loading 0: 59%|█████▉ | 173/291 [00:45<00:33, 3.52it/s]
Loading 0: 60%|██████ | 175/291 [00:45<00:24, 4.74it/s]
Loading 0: 60%|██████ | 176/291 [00:45<00:23, 4.98it/s]
Loading 0: 61%|██████ | 177/291 [00:45<00:20, 5.59it/s]
Loading 0: 62%|██████▏ | 179/291 [00:46<00:25, 4.43it/s]
Loading 0: 62%|██████▏ | 180/291 [00:46<00:33, 3.27it/s]
Loading 0: 62%|██████▏ | 181/291 [00:47<00:42, 2.57it/s]
Loading 0: 63%|██████▎ | 184/291 [00:47<00:23, 4.48it/s]
Loading 0: 64%|██████▎ | 185/291 [00:47<00:22, 4.71it/s]
Loading 0: 64%|██████▍ | 186/291 [00:47<00:19, 5.32it/s]
Loading 0: 64%|██████▍ | 187/291 [00:48<00:19, 5.28it/s]
Loading 0: 65%|██████▍ | 188/291 [00:48<00:29, 3.45it/s]
Loading 0: 65%|██████▍ | 189/291 [00:49<00:39, 2.59it/s]
Loading 0: 66%|██████▌ | 192/291 [00:49<00:28, 3.51it/s]
Loading 0: 66%|██████▋ | 193/291 [00:50<00:33, 2.90it/s]
Loading 0: 67%|██████▋ | 194/291 [00:51<00:39, 2.43it/s]
Loading 0: 68%|██████▊ | 197/291 [00:51<00:22, 4.10it/s]
Loading 0: 68%|██████▊ | 198/291 [00:51<00:21, 4.36it/s]
Loading 0: 68%|██████▊ | 199/291 [00:51<00:18, 4.95it/s]
Loading 0: 69%|██████▉ | 201/291 [00:52<00:21, 4.24it/s]
Loading 0: 69%|██████▉ | 202/291 [00:52<00:27, 3.22it/s]
Loading 0: 70%|██████▉ | 203/291 [00:53<00:34, 2.56it/s]
Loading 0: 71%|███████ | 206/291 [00:53<00:19, 4.34it/s]
Loading 0: 71%|███████ | 207/291 [00:53<00:18, 4.58it/s]
Loading 0: 71%|███████▏ | 208/291 [00:53<00:16, 5.14it/s]
Loading 0: 72%|███████▏ | 210/291 [00:54<00:18, 4.33it/s]
Loading 0: 73%|███████▎ | 211/291 [00:55<00:24, 3.26it/s]
Loading 0: 73%|███████▎ | 212/291 [00:55<00:30, 2.59it/s]
Loading 0: 74%|███████▍ | 215/291 [00:55<00:17, 4.45it/s]
Loading 0: 74%|███████▍ | 216/291 [00:56<00:16, 4.68it/s]
Loading 0: 75%|███████▍ | 217/291 [00:56<00:14, 5.27it/s]
Loading 0: 75%|███████▌ | 219/291 [00:56<00:16, 4.38it/s]
Loading 0: 76%|███████▌ | 220/291 [00:57<00:21, 3.30it/s]
Loading 0: 76%|███████▌ | 221/291 [00:57<00:26, 2.60it/s]
Loading 0: 77%|███████▋ | 224/291 [00:58<00:15, 4.41it/s]
Loading 0: 77%|███████▋ | 225/291 [00:58<00:14, 4.63it/s]
Loading 0: 78%|███████▊ | 226/291 [00:58<00:12, 5.20it/s]
Loading 0: 78%|███████▊ | 228/291 [00:59<00:14, 4.25it/s]
Loading 0: 79%|███████▊ | 229/291 [00:59<00:19, 3.22it/s]
Loading 0: 79%|███████▉ | 230/291 [01:00<00:23, 2.57it/s]
Loading 0: 80%|████████ | 233/291 [01:00<00:13, 4.42it/s]
Loading 0: 80%|████████ | 234/291 [01:00<00:12, 4.66it/s]
Loading 0: 81%|████████ | 235/291 [01:00<00:10, 5.22it/s]
Loading 0: 81%|████████▏ | 237/291 [01:01<00:12, 4.32it/s]
Loading 0: 82%|████████▏ | 238/291 [01:01<00:16, 3.26it/s]
Loading 0: 82%|████████▏ | 239/291 [01:02<00:20, 2.59it/s]
Loading 0: 83%|████████▎ | 242/291 [01:02<00:11, 4.44it/s]
Loading 0: 84%|████████▎ | 243/291 [01:02<00:10, 4.68it/s]
Loading 0: 84%|████████▍ | 244/291 [01:03<00:08, 5.25it/s]
Loading 0: 85%|████████▍ | 246/291 [01:03<00:10, 4.36it/s]
Loading 0: 85%|████████▍ | 247/291 [01:04<00:13, 3.27it/s]
Loading 0: 85%|████████▌ | 248/291 [01:04<00:16, 2.60it/s]
Loading 0: 86%|████████▋ | 251/291 [01:05<00:09, 4.41it/s]
Loading 0: 87%|████████▋ | 252/291 [01:05<00:08, 4.65it/s]
Loading 0: 87%|████████▋ | 253/291 [01:05<00:07, 5.25it/s]
Loading 0: 88%|████████▊ | 255/291 [01:05<00:08, 4.36it/s]
Loading 0: 88%|████████▊ | 256/291 [01:06<00:10, 3.30it/s]
Loading 0: 88%|████████▊ | 257/291 [01:07<00:13, 2.60it/s]
Loading 0: 89%|████████▉ | 260/291 [01:07<00:07, 4.41it/s]
Loading 0: 90%|████████▉ | 261/291 [01:07<00:06, 4.65it/s]
Loading 0: 90%|█████████ | 262/291 [01:07<00:05, 5.21it/s]
Loading 0: 91%|█████████ | 264/291 [01:08<00:06, 4.33it/s]
Loading 0: 91%|█████████ | 265/291 [01:08<00:07, 3.26it/s]
Loading 0: 91%|█████████▏| 266/291 [01:09<00:09, 2.57it/s]
Loading 0: 92%|█████████▏| 269/291 [01:09<00:05, 4.35it/s]
Loading 0: 93%|█████████▎| 270/291 [01:09<00:04, 4.58it/s]
Loading 0: 93%|█████████▎| 271/291 [01:09<00:03, 5.14it/s]
Loading 0: 94%|█████████▍| 273/291 [01:10<00:04, 4.32it/s]
Loading 0: 94%|█████████▍| 274/291 [01:11<00:05, 3.27it/s]
Loading 0: 95%|█████████▍| 275/291 [01:11<00:06, 2.58it/s]
Loading 0: 96%|█████████▌| 278/291 [01:11<00:02, 4.38it/s]
Loading 0: 96%|█████████▌| 279/291 [01:12<00:02, 4.62it/s]
Loading 0: 96%|█████████▌| 280/291 [01:12<00:02, 5.18it/s]
Loading 0: 97%|█████████▋| 281/291 [01:12<00:02, 3.50it/s]
Loading 0: 97%|█████████▋| 282/291 [01:13<00:03, 2.65it/s]
Loading 0: 98%|█████████▊| 284/291 [01:13<00:01, 3.79it/s]
Loading 0: 98%|█████████▊| 285/291 [01:13<00:01, 4.03it/s]
Loading 0: 98%|█████████▊| 286/291 [01:14<00:01, 4.69it/s]
Loading 0: 99%|█████████▊| 287/291 [01:14<00:00, 4.81it/s]
Loading 0: 99%|█████████▉| 288/291 [01:14<00:00, 3.23it/s]
Job alexdaoud-trainer-bagir-8476-v1-mkmlizer completed after 157.18s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-8476-v1-mkmlizer
Pipeline stage MKMLizer completed in 157.69s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-8476-v1
Waiting for inference service alexdaoud-trainer-bagir-8476-v1 to be ready
Inference service alexdaoud-trainer-bagir-8476-v1 ready after 191.21333384513855s
Pipeline stage MKMLDeployer completed in 191.75s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.4142396450042725s
Received healthy response to inference request in 3.5224220752716064s
Received healthy response to inference request in 2.274441957473755s
Received healthy response to inference request in 5.499939203262329s
Received healthy response to inference request in 2.5021281242370605s
5 requests
0 failed requests
5th percentile: 2.319979190826416
10th percentile: 2.365516424179077
20th percentile: 2.4565908908843994
30th percentile: 2.7061869144439696
40th percentile: 3.1143044948577883
50th percentile: 3.5224220752716064
60th percentile: 3.8791491031646728
70th percentile: 4.235876131057739
80th percentile: 4.631379556655884
90th percentile: 5.065659379959106
95th percentile: 5.282799291610718
99th percentile: 5.456511220932007
mean time: 3.6426342010498045
%s, retrying in %s seconds...
Received healthy response to inference request in 3.517759084701538s
Received healthy response to inference request in 4.490128755569458s
Received healthy response to inference request in 4.006576776504517s
Received healthy response to inference request in 5.142945289611816s
Received healthy response to inference request in 2.6166481971740723s
5 requests
0 failed requests
5th percentile: 2.7968703746795653
10th percentile: 2.9770925521850584
20th percentile: 3.337536907196045
30th percentile: 3.615522623062134
40th percentile: 3.811049699783325
50th percentile: 4.006576776504517
60th percentile: 4.1999975681304935
70th percentile: 4.3934183597564695
80th percentile: 4.6206920623779295
90th percentile: 4.881818675994873
95th percentile: 5.012381982803345
99th percentile: 5.116832628250122
mean time: 3.95481162071228
%s, retrying in %s seconds...
Received healthy response to inference request in 2.4099438190460205s
Received healthy response to inference request in 3.688307762145996s
Received healthy response to inference request in 6.25715708732605s
Received healthy response to inference request in 2.410527467727661s
Received healthy response to inference request in 2.201697826385498s
5 requests
0 failed requests
5th percentile: 2.2433470249176026
10th percentile: 2.284996223449707
20th percentile: 2.368294620513916
30th percentile: 2.4100605487823485
40th percentile: 2.410294008255005
50th percentile: 2.410527467727661
60th percentile: 2.921639585494995
70th percentile: 3.432751703262329
80th percentile: 4.202077627182007
90th percentile: 5.229617357254028
95th percentile: 5.743387222290039
99th percentile: 6.154403114318848
mean time: 3.393526792526245
Pipeline stage StressChecker completed in 59.77s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.25s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.04s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_8476_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8476-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8476-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8476-v1-profiler ready after 190.40573263168335s
Pipeline stage MKMLProfilerDeployer completed in 190.80s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplokq5zx:/code/chaiverse_profiler_1734000248 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplokq5zx --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734000248 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734000248/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplokq5zx:/code/chaiverse_profiler_1734003017 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplokq5zx:/code/chaiverse_profiler_1734003018 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplokq5zx --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734003018 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734003018/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8476-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8476-v1-profiler
Service alexdaoud-trainer-bagir-8476-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.90s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8476-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.90s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8476-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8476-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8476-v1-profiler ready after 100.24697923660278s
Pipeline stage MKMLProfilerDeployer completed in 100.66s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplohhcck:/code/chaiverse_profiler_1734003798 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplohhcck --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734003798 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734003798/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplohhcck:/code/chaiverse_profiler_1734006596 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplohhcck:/code/chaiverse_profiler_1734006596 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplohhcck --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734006596 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734006596/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8476-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8476-v1-profiler
Service alexdaoud-trainer-bagir-8476-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.04s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8476-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.97s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8476-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8476-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8476-v1-profiler ready after 180.4099588394165s
Pipeline stage MKMLProfilerDeployer completed in 180.78s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplo8lmpg:/code/chaiverse_profiler_1734007509 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplo8lmpg --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734007509 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734007509/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplo8lmpg:/code/chaiverse_profiler_1734010264 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplo8lmpg:/code/chaiverse_profiler_1734010264 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1a2a958efd9c0ad4b5dc0d50219e8a2b-deplo8lmpg --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734010264 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734010264/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8476-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8476-v1-profiler
Service alexdaoud-trainer-bagir-8476-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.09s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_8476_v1 status is now inactive due to auto deactivation removed underperforming models
alexdaoud-trainer-bagir-_8476_v1 status is now torndown due to DeploymentManager action
alexdaoud-trainer-bagir-_8476_v1 status is now torndown due to DeploymentManager action
alexdaoud-trainer-bagir-_8476_v1 status is now torndown due to DeploymentManager action