Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-8434-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-8434-v1-mkmlizer to finish
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-8434-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-8434-v1-mkmlizer: Downloaded to shared memory in 33.554s
alexdaoud-trainer-bagir-8434-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp_54rhhiz, device:0
alexdaoud-trainer-bagir-8434-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-8434-v1-mkmlizer: quantized model in 87.357s
alexdaoud-trainer-bagir-8434-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-16 in 120.912s
alexdaoud-trainer-bagir-8434-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-8434-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-8434-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1
alexdaoud-trainer-bagir-8434-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1/config.json
alexdaoud-trainer-bagir-8434-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1/special_tokens_map.json
alexdaoud-trainer-bagir-8434-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1/tokenizer_config.json
alexdaoud-trainer-bagir-8434-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1/tokenizer.json
alexdaoud-trainer-bagir-8434-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-8434-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-8434-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:57, 4.99it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:33, 3.06it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:06, 2.27it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:05, 4.35it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:03, 4.42it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:55, 5.04it/s]
Loading 0: 4%|▍ | 12/291 [00:03<01:06, 4.20it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:28, 3.14it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:53, 2.45it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:04, 4.22it/s]
Loading 0: 6%|▌ | 18/291 [00:04<01:01, 4.46it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:54, 5.02it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:03, 4.23it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:24, 3.20it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:46, 2.52it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:01, 4.28it/s]
Loading 0: 9%|▉ | 27/291 [00:07<00:58, 4.51it/s]
Loading 0: 10%|▉ | 28/291 [00:07<00:51, 5.07it/s]
Loading 0: 10%|█ | 30/291 [00:07<01:01, 4.24it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:21, 3.18it/s]
Loading 0: 11%|█ | 32/291 [00:09<01:44, 2.49it/s]
Loading 0: 12%|█▏ | 35/291 [00:09<01:00, 4.22it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:57, 4.44it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:50, 5.01it/s]
Loading 0: 13%|█▎ | 39/291 [00:10<00:59, 4.21it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:19, 3.16it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:39, 2.51it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:57, 4.28it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:54, 4.51it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:49, 4.99it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:57, 4.23it/s]
Loading 0: 17%|█▋ | 49/291 [00:13<01:15, 3.21it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:35, 2.51it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:55, 4.32it/s]
Loading 0: 19%|█▊ | 54/291 [00:14<00:52, 4.54it/s]
Loading 0: 19%|█▉ | 55/291 [00:14<00:46, 5.10it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:54, 4.26it/s]
Loading 0: 20%|█▉ | 58/291 [00:15<01:12, 3.22it/s]
Loading 0: 20%|██ | 59/291 [00:16<01:31, 2.52it/s]
Loading 0: 21%|██▏ | 62/291 [00:16<00:53, 4.28it/s]
Loading 0: 22%|██▏ | 63/291 [00:16<00:50, 4.53it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:44, 5.11it/s]
Loading 0: 23%|██▎ | 66/291 [00:17<00:52, 4.27it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:09, 3.22it/s]
Loading 0: 23%|██▎ | 68/291 [00:18<01:28, 2.53it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:51, 4.30it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:48, 4.53it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:43, 5.06it/s]
Loading 0: 26%|██▌ | 75/291 [00:19<00:50, 4.25it/s]
Loading 0: 26%|██▌ | 76/291 [00:20<01:06, 3.22it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:24, 2.53it/s]
Loading 0: 27%|██▋ | 80/291 [00:21<00:49, 4.29it/s]
Loading 0: 28%|██▊ | 81/291 [00:21<00:46, 4.53it/s]
Loading 0: 28%|██▊ | 82/291 [00:21<00:41, 5.09it/s]
Loading 0: 29%|██▊ | 83/291 [00:21<00:41, 5.05it/s]
Loading 0: 29%|██▉ | 84/291 [00:22<01:01, 3.37it/s]
Loading 0: 29%|██▉ | 85/291 [00:22<01:17, 2.66it/s]
Loading 0: 30%|██▉ | 86/291 [00:23<01:32, 2.21it/s]
Loading 0: 31%|███ | 89/291 [00:23<00:49, 4.11it/s]
Loading 0: 31%|███ | 90/291 [00:23<00:45, 4.38it/s]
Loading 0: 31%|███▏ | 91/291 [00:23<00:40, 4.99it/s]
Loading 0: 32%|███▏ | 93/291 [00:24<00:47, 4.18it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:02, 3.17it/s]
Loading 0: 33%|███▎ | 95/291 [00:25<01:18, 2.50it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:44, 4.32it/s]
Loading 0: 34%|███▍ | 99/291 [00:26<00:42, 4.56it/s]
Loading 0: 34%|███▍ | 100/291 [00:26<00:37, 5.08it/s]
Loading 0: 35%|███▌ | 102/291 [00:26<00:44, 4.27it/s]
Loading 0: 35%|███▌ | 103/291 [00:27<00:58, 3.23it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:13, 2.56it/s]
Loading 0: 37%|███▋ | 107/291 [00:28<00:42, 4.36it/s]
Loading 0: 37%|███▋ | 108/291 [00:28<00:39, 4.59it/s]
Loading 0: 37%|███▋ | 109/291 [00:28<00:35, 5.20it/s]
Loading 0: 38%|███▊ | 111/291 [00:29<00:41, 4.32it/s]
Loading 0: 38%|███▊ | 112/291 [00:29<00:54, 3.26it/s]
Loading 0: 39%|███▉ | 113/291 [00:30<01:09, 2.57it/s]
Loading 0: 40%|███▉ | 116/291 [00:30<00:40, 4.35it/s]
Loading 0: 40%|████ | 117/291 [00:30<00:37, 4.58it/s]
Loading 0: 41%|████ | 118/291 [00:30<00:33, 5.16it/s]
Loading 0: 41%|████ | 120/291 [00:31<00:39, 4.33it/s]
Loading 0: 42%|████▏ | 121/291 [00:31<00:52, 3.27it/s]
Loading 0: 42%|████▏ | 122/291 [00:32<01:06, 2.55it/s]
Loading 0: 43%|████▎ | 125/291 [00:32<00:37, 4.39it/s]
Loading 0: 43%|████▎ | 126/291 [00:32<00:35, 4.62it/s]
Loading 0: 44%|████▎ | 127/291 [00:33<00:31, 5.19it/s]
Loading 0: 44%|████▍ | 129/291 [00:33<00:37, 4.33it/s]
Loading 0: 45%|████▍ | 130/291 [00:34<00:49, 3.27it/s]
Loading 0: 45%|████▌ | 131/291 [00:34<01:01, 2.59it/s]
Loading 0: 46%|████▌ | 134/291 [00:35<00:35, 4.45it/s]
Loading 0: 46%|████▋ | 135/291 [00:35<00:33, 4.67it/s]
Loading 0: 47%|████▋ | 136/291 [00:35<00:29, 5.24it/s]
Loading 0: 47%|████▋ | 138/291 [00:35<00:35, 4.37it/s]
Loading 0: 48%|████▊ | 139/291 [00:36<00:46, 3.29it/s]
Loading 0: 48%|████▊ | 140/291 [00:37<00:58, 2.58it/s]
Loading 0: 49%|████▉ | 143/291 [00:37<00:33, 4.45it/s]
Loading 0: 49%|████▉ | 144/291 [00:37<00:31, 4.68it/s]
Loading 0: 50%|████▉ | 145/291 [00:37<00:27, 5.27it/s]
Loading 0: 51%|█████ | 147/291 [00:38<00:32, 4.37it/s]
Loading 0: 51%|█████ | 148/291 [00:38<00:43, 3.29it/s]
Loading 0: 51%|█████ | 149/291 [00:39<00:54, 2.59it/s]
Loading 0: 52%|█████▏ | 152/291 [00:39<00:31, 4.45it/s]
Loading 0: 53%|█████▎ | 153/291 [00:39<00:29, 4.61it/s]
Loading 0: 53%|█████▎ | 154/291 [00:39<00:26, 5.21it/s]
Loading 0: 54%|█████▎ | 156/291 [00:40<00:31, 4.33it/s]
Loading 0: 54%|█████▍ | 157/291 [00:41<00:41, 3.27it/s]
Loading 0: 54%|█████▍ | 158/291 [00:41<00:51, 2.58it/s]
Loading 0: 55%|█████▌ | 161/291 [00:41<00:29, 4.44it/s]
Loading 0: 56%|█████▌ | 162/291 [00:42<00:27, 4.67it/s]
Loading 0: 56%|█████▌ | 163/291 [00:42<00:24, 5.26it/s]
Loading 0: 57%|█████▋ | 165/291 [00:42<00:28, 4.36it/s]
Loading 0: 57%|█████▋ | 166/291 [00:43<00:38, 3.29it/s]
Loading 0: 57%|█████▋ | 167/291 [00:44<00:47, 2.60it/s]
Loading 0: 58%|█████▊ | 170/291 [00:44<00:27, 4.42it/s]
Loading 0: 59%|█████▉ | 171/291 [00:44<00:25, 4.65it/s]
Loading 0: 59%|█████▉ | 172/291 [00:44<00:23, 5.15it/s]
Loading 0: 59%|█████▉ | 173/291 [00:45<00:33, 3.50it/s]
Loading 0: 60%|██████ | 175/291 [00:45<00:24, 4.81it/s]
Loading 0: 60%|██████ | 176/291 [00:45<00:22, 5.03it/s]
Loading 0: 61%|██████ | 177/291 [00:45<00:20, 5.67it/s]
Loading 0: 62%|██████▏ | 179/291 [00:46<00:25, 4.47it/s]
Loading 0: 62%|██████▏ | 180/291 [00:46<00:33, 3.29it/s]
Loading 0: 62%|██████▏ | 181/291 [00:47<00:42, 2.59it/s]
Loading 0: 63%|██████▎ | 184/291 [00:47<00:23, 4.54it/s]
Loading 0: 64%|██████▎ | 185/291 [00:47<00:22, 4.76it/s]
Loading 0: 64%|██████▍ | 186/291 [00:47<00:19, 5.35it/s]
Loading 0: 64%|██████▍ | 187/291 [00:48<00:19, 5.32it/s]
Loading 0: 65%|██████▍ | 188/291 [00:48<00:29, 3.47it/s]
Loading 0: 65%|██████▍ | 189/291 [00:49<00:39, 2.60it/s]
Loading 0: 66%|██████▌ | 192/291 [00:49<00:27, 3.54it/s]
Loading 0: 66%|██████▋ | 193/291 [00:50<00:33, 2.91it/s]
Loading 0: 67%|██████▋ | 194/291 [00:51<00:40, 2.41it/s]
Loading 0: 68%|██████▊ | 197/291 [00:51<00:23, 4.08it/s]
Loading 0: 68%|██████▊ | 198/291 [00:51<00:21, 4.34it/s]
Loading 0: 68%|██████▊ | 199/291 [00:51<00:18, 4.91it/s]
Loading 0: 69%|██████▉ | 201/291 [00:52<00:21, 4.22it/s]
Loading 0: 69%|██████▉ | 202/291 [00:52<00:27, 3.22it/s]
Loading 0: 70%|██████▉ | 203/291 [00:53<00:34, 2.55it/s]
Loading 0: 71%|███████ | 206/291 [00:53<00:19, 4.34it/s]
Loading 0: 71%|███████ | 207/291 [00:53<00:18, 4.58it/s]
Loading 0: 71%|███████▏ | 208/291 [00:53<00:16, 5.10it/s]
Loading 0: 72%|███████▏ | 210/291 [00:54<00:18, 4.30it/s]
Loading 0: 73%|███████▎ | 211/291 [00:55<00:24, 3.25it/s]
Loading 0: 73%|███████▎ | 212/291 [00:55<00:30, 2.57it/s]
Loading 0: 74%|███████▍ | 215/291 [00:55<00:17, 4.36it/s]
Loading 0: 74%|███████▍ | 216/291 [00:56<00:16, 4.59it/s]
Loading 0: 75%|███████▍ | 217/291 [00:56<00:14, 5.13it/s]
Loading 0: 75%|███████▌ | 219/291 [00:56<00:16, 4.32it/s]
Loading 0: 76%|███████▌ | 220/291 [00:57<00:21, 3.27it/s]
Loading 0: 76%|███████▌ | 221/291 [00:57<00:27, 2.59it/s]
Loading 0: 77%|███████▋ | 224/291 [00:58<00:15, 4.43it/s]
Loading 0: 77%|███████▋ | 225/291 [00:58<00:14, 4.66it/s]
Loading 0: 78%|███████▊ | 226/291 [00:58<00:12, 5.22it/s]
Loading 0: 78%|███████▊ | 228/291 [00:59<00:14, 4.35it/s]
Loading 0: 79%|███████▊ | 229/291 [00:59<00:18, 3.27it/s]
Loading 0: 79%|███████▉ | 230/291 [01:00<00:23, 2.59it/s]
Loading 0: 80%|████████ | 233/291 [01:00<00:13, 4.44it/s]
Loading 0: 80%|████████ | 234/291 [01:00<00:12, 4.67it/s]
Loading 0: 81%|████████ | 235/291 [01:00<00:10, 5.24it/s]
Loading 0: 81%|████████▏ | 237/291 [01:01<00:12, 4.36it/s]
Loading 0: 82%|████████▏ | 238/291 [01:01<00:16, 3.29it/s]
Loading 0: 82%|████████▏ | 239/291 [01:02<00:20, 2.59it/s]
Loading 0: 83%|████████▎ | 242/291 [01:02<00:11, 4.45it/s]
Loading 0: 84%|████████▎ | 243/291 [01:02<00:10, 4.69it/s]
Loading 0: 84%|████████▍ | 244/291 [01:03<00:08, 5.28it/s]
Loading 0: 85%|████████▍ | 246/291 [01:03<00:10, 4.37it/s]
Loading 0: 85%|████████▍ | 247/291 [01:04<00:13, 3.27it/s]
Loading 0: 85%|████████▌ | 248/291 [01:04<00:16, 2.59it/s]
Loading 0: 86%|████████▋ | 251/291 [01:05<00:08, 4.45it/s]
Loading 0: 87%|████████▋ | 252/291 [01:05<00:08, 4.69it/s]
Loading 0: 87%|████████▋ | 253/291 [01:05<00:07, 5.26it/s]
Loading 0: 88%|████████▊ | 255/291 [01:05<00:08, 4.35it/s]
Loading 0: 88%|████████▊ | 256/291 [01:06<00:10, 3.29it/s]
Loading 0: 88%|████████▊ | 257/291 [01:07<00:13, 2.60it/s]
Loading 0: 89%|████████▉ | 260/291 [01:07<00:07, 4.41it/s]
Loading 0: 90%|████████▉ | 261/291 [01:07<00:06, 4.65it/s]
Loading 0: 90%|█████████ | 262/291 [01:07<00:05, 5.23it/s]
Loading 0: 91%|█████████ | 264/291 [01:08<00:06, 4.35it/s]
Loading 0: 91%|█████████ | 265/291 [01:08<00:07, 3.27it/s]
Loading 0: 91%|█████████▏| 266/291 [01:09<00:09, 2.55it/s]
Loading 0: 92%|█████████▏| 269/291 [01:09<00:05, 4.40it/s]
Loading 0: 93%|█████████▎| 270/291 [01:09<00:04, 4.64it/s]
Loading 0: 93%|█████████▎| 271/291 [01:09<00:03, 5.21it/s]
Loading 0: 94%|█████████▍| 273/291 [01:10<00:04, 4.36it/s]
Loading 0: 94%|█████████▍| 274/291 [01:11<00:05, 3.30it/s]
Loading 0: 95%|█████████▍| 275/291 [01:11<00:06, 2.62it/s]
Loading 0: 96%|█████████▌| 278/291 [01:11<00:02, 4.50it/s]
Loading 0: 96%|█████████▌| 279/291 [01:12<00:02, 4.73it/s]
Loading 0: 96%|█████████▌| 280/291 [01:12<00:02, 5.31it/s]
Loading 0: 97%|█████████▋| 281/291 [01:12<00:02, 3.54it/s]
Loading 0: 97%|█████████▋| 282/291 [01:13<00:03, 2.67it/s]
Loading 0: 98%|█████████▊| 284/291 [01:13<00:01, 3.82it/s]
Loading 0: 98%|█████████▊| 285/291 [01:13<00:01, 4.17it/s]
Loading 0: 98%|█████████▊| 286/291 [01:13<00:01, 4.80it/s]
Loading 0: 99%|█████████▊| 287/291 [01:14<00:00, 4.90it/s]
Loading 0: 99%|█████████▉| 288/291 [01:14<00:00, 3.24it/s]
Job alexdaoud-trainer-bagir-8434-v1-mkmlizer completed after 145.54s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-8434-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.10s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-8434-v1
Waiting for inference service alexdaoud-trainer-bagir-8434-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service alexdaoud-trainer-bagir-8434-v1 ready after 190.66939687728882s
Pipeline stage MKMLDeployer completed in 191.26s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.1720290184020996s
Received healthy response to inference request in 4.29815149307251s
Received healthy response to inference request in 2.5899412631988525s
Received healthy response to inference request in 4.519265413284302s
Received healthy response to inference request in 4.06724739074707s
5 requests
0 failed requests
5th percentile: 2.706358814239502
10th percentile: 2.822776365280151
20th percentile: 3.0556114673614503
30th percentile: 3.3510726928710937
40th percentile: 3.709160041809082
50th percentile: 4.06724739074707
60th percentile: 4.159609031677246
70th percentile: 4.251970672607422
80th percentile: 4.342374277114868
90th percentile: 4.430819845199585
95th percentile: 4.475042629241943
99th percentile: 4.51042085647583
mean time: 3.729326915740967
%s, retrying in %s seconds...
Received healthy response to inference request in 2.751596689224243s
Received healthy response to inference request in 3.35988450050354s
Received healthy response to inference request in 3.2621266841888428s
Received healthy response to inference request in 3.6548044681549072s
Received healthy response to inference request in 4.145944833755493s
5 requests
0 failed requests
5th percentile: 2.853702688217163
10th percentile: 2.955808687210083
20th percentile: 3.160020685195923
30th percentile: 3.281678247451782
40th percentile: 3.320781373977661
50th percentile: 3.35988450050354
60th percentile: 3.477852487564087
70th percentile: 3.5958204746246336
80th percentile: 3.7530325412750245
90th percentile: 3.9494886875152586
95th percentile: 4.047716760635376
99th percentile: 4.12629921913147
mean time: 3.4348714351654053
Pipeline stage StressChecker completed in 39.10s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.26s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.11s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_8434_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8434-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8434-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8434-v1-profiler ready after 190.4460642337799s
Pipeline stage MKMLProfilerDeployer completed in 190.82s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplolhx2q:/code/chaiverse_profiler_1734020887 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplolhx2q --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734020887 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734020887/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplolhx2q:/code/chaiverse_profiler_1734023663 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplolhx2q:/code/chaiverse_profiler_1734023664 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplolhx2q --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734023664 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734023664/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8434-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8434-v1-profiler
Service alexdaoud-trainer-bagir-8434-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.11s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8434-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.05s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8434-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8434-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8434-v1-profiler ready after 200.44724988937378s
Pipeline stage MKMLProfilerDeployer completed in 200.79s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplovm8zc:/code/chaiverse_profiler_1734024541 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplovm8zc --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734024541 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734024541/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplovm8zc:/code/chaiverse_profiler_1734027313 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplovm8zc:/code/chaiverse_profiler_1734027314 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplovm8zc --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734027314 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734027314/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8434-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8434-v1-profiler
Service alexdaoud-trainer-bagir-8434-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.12s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8434-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.33s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-8434-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-8434-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-8434-v1-profiler ready after 140.35678339004517s
Pipeline stage MKMLProfilerDeployer completed in 140.70s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplo2xfbs:/code/chaiverse_profiler_1734028113 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplo2xfbs --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734028113 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734028113/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplo2xfbs:/code/chaiverse_profiler_1734030906 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplo2xfbs:/code/chaiverse_profiler_1734030907 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba1e97f6106a34bcf887d42b1912f4461e-deplo2xfbs --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734030907 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734030907/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-8434-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-8434-v1-profiler
Service alexdaoud-trainer-bagir-8434-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.11s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_8434_v1 status is now inactive due to auto deactivation removed underperforming models