Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-54351-v2-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-54351-v2-mkmlizer to finish
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ║ ║
alexdaoud-trainer-bagir-54351-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-54351-v2-mkmlizer: Downloaded to shared memory in 32.378s
alexdaoud-trainer-bagir-54351-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmptqipa6pv, device:0
alexdaoud-trainer-bagir-54351-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
alexdaoud-trainer-bagir-54351-v2-mkmlizer: quantized model in 85.644s
alexdaoud-trainer-bagir-54351-v2-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-76 in 118.023s
alexdaoud-trainer-bagir-54351-v2-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-54351-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-54351-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2
alexdaoud-trainer-bagir-54351-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2/config.json
alexdaoud-trainer-bagir-54351-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2/special_tokens_map.json
alexdaoud-trainer-bagir-54351-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2/tokenizer_config.json
alexdaoud-trainer-bagir-54351-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2/tokenizer.json
alexdaoud-trainer-bagir-54351-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-54351-v2/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-54351-v2-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:57, 5.02it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:33, 3.08it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:04, 2.29it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:04, 4.39it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:03, 4.43it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:55, 5.04it/s]
Loading 0: 4%|▍ | 12/291 [00:03<01:06, 4.20it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:29, 3.12it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:51, 2.49it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:03, 4.30it/s]
Loading 0: 6%|▌ | 18/291 [00:04<01:00, 4.54it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:53, 5.13it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:02, 4.30it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:22, 3.24it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:45, 2.55it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:01, 4.34it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:57, 4.58it/s]
Loading 0: 10%|▉ | 28/291 [00:07<00:50, 5.18it/s]
Loading 0: 10%|█ | 30/291 [00:07<01:00, 4.33it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:19, 3.27it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:40, 2.58it/s]
Loading 0: 12%|█▏ | 35/291 [00:09<00:58, 4.39it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:55, 4.62it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:48, 5.21it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:58, 4.33it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:16, 3.27it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:37, 2.56it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:56, 4.34it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:53, 4.58it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:47, 5.11it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:56, 4.30it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:14, 3.26it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:34, 2.55it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:54, 4.33it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:51, 4.57it/s]
Loading 0: 19%|█▉ | 55/291 [00:14<00:45, 5.16it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:54, 4.32it/s]
Loading 0: 20%|█▉ | 58/291 [00:15<01:11, 3.28it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:29, 2.58it/s]
Loading 0: 21%|██▏ | 62/291 [00:16<00:51, 4.43it/s]
Loading 0: 22%|██▏ | 63/291 [00:16<00:48, 4.67it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:43, 5.26it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:51, 4.38it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:08, 3.29it/s]
Loading 0: 23%|██▎ | 68/291 [00:18<01:26, 2.57it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:50, 4.35it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:47, 4.59it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:42, 5.18it/s]
Loading 0: 26%|██▌ | 75/291 [00:19<00:49, 4.33it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:05, 3.27it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:22, 2.59it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:48, 4.38it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:45, 4.62it/s]
Loading 0: 28%|██▊ | 82/291 [00:20<00:40, 5.19it/s]
Loading 0: 29%|██▊ | 83/291 [00:21<00:39, 5.21it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<01:00, 3.45it/s]
Loading 0: 29%|██▉ | 85/291 [00:22<01:15, 2.73it/s]
Loading 0: 30%|██▉ | 86/291 [00:22<01:30, 2.26it/s]
Loading 0: 31%|███ | 89/291 [00:23<00:48, 4.15it/s]
Loading 0: 31%|███ | 90/291 [00:23<00:45, 4.42it/s]
Loading 0: 31%|███▏ | 91/291 [00:23<00:40, 4.99it/s]
Loading 0: 32%|███▏ | 93/291 [00:23<00:46, 4.24it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:01, 3.21it/s]
Loading 0: 33%|███▎ | 95/291 [00:25<01:16, 2.56it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:44, 4.37it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:41, 4.61it/s]
Loading 0: 34%|███▍ | 100/291 [00:25<00:36, 5.17it/s]
Loading 0: 35%|███▌ | 102/291 [00:26<00:43, 4.33it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:57, 3.27it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:12, 2.58it/s]
Loading 0: 37%|███▋ | 107/291 [00:27<00:41, 4.44it/s]
Loading 0: 37%|███▋ | 108/291 [00:27<00:39, 4.68it/s]
Loading 0: 37%|███▋ | 109/291 [00:27<00:34, 5.28it/s]
Loading 0: 38%|███▊ | 111/291 [00:28<00:41, 4.37it/s]
Loading 0: 38%|███▊ | 112/291 [00:29<00:54, 3.28it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:08, 2.58it/s]
Loading 0: 40%|███▉ | 116/291 [00:29<00:40, 4.37it/s]
Loading 0: 40%|████ | 117/291 [00:30<00:37, 4.62it/s]
Loading 0: 41%|████ | 118/291 [00:30<00:33, 5.21it/s]
Loading 0: 41%|████ | 120/291 [00:30<00:39, 4.34it/s]
Loading 0: 42%|████▏ | 121/291 [00:31<00:51, 3.28it/s]
Loading 0: 42%|████▏ | 122/291 [00:32<01:05, 2.58it/s]
Loading 0: 43%|████▎ | 125/291 [00:32<00:37, 4.42it/s]
Loading 0: 43%|████▎ | 126/291 [00:32<00:35, 4.66it/s]
Loading 0: 44%|████▎ | 127/291 [00:32<00:31, 5.25it/s]
Loading 0: 44%|████▍ | 129/291 [00:33<00:37, 4.37it/s]
Loading 0: 45%|████▍ | 130/291 [00:33<00:48, 3.30it/s]
Loading 0: 45%|████▌ | 131/291 [00:34<01:01, 2.61it/s]
Loading 0: 46%|████▌ | 134/291 [00:34<00:35, 4.43it/s]
Loading 0: 46%|████▋ | 135/291 [00:34<00:33, 4.67it/s]
Loading 0: 47%|████▋ | 136/291 [00:34<00:29, 5.26it/s]
Loading 0: 47%|████▋ | 138/291 [00:35<00:35, 4.37it/s]
Loading 0: 48%|████▊ | 139/291 [00:35<00:46, 3.30it/s]
Loading 0: 48%|████▊ | 140/291 [00:36<00:57, 2.61it/s]
Loading 0: 49%|████▉ | 143/291 [00:36<00:33, 4.43it/s]
Loading 0: 49%|████▉ | 144/291 [00:37<00:31, 4.68it/s]
Loading 0: 50%|████▉ | 145/291 [00:37<00:27, 5.27it/s]
Loading 0: 51%|█████ | 147/291 [00:37<00:32, 4.40it/s]
Loading 0: 51%|█████ | 148/291 [00:38<00:43, 3.32it/s]
Loading 0: 51%|█████ | 149/291 [00:38<00:54, 2.61it/s]
Loading 0: 52%|█████▏ | 152/291 [00:39<00:30, 4.48it/s]
Loading 0: 53%|█████▎ | 153/291 [00:39<00:29, 4.72it/s]
Loading 0: 53%|█████▎ | 154/291 [00:39<00:25, 5.31it/s]
Loading 0: 54%|█████▎ | 156/291 [00:39<00:30, 4.41it/s]
Loading 0: 54%|█████▍ | 157/291 [00:40<00:40, 3.33it/s]
Loading 0: 54%|█████▍ | 158/291 [00:41<00:50, 2.62it/s]
Loading 0: 55%|█████▌ | 161/291 [00:41<00:28, 4.51it/s]
Loading 0: 56%|█████▌ | 162/291 [00:41<00:27, 4.74it/s]
Loading 0: 56%|█████▌ | 163/291 [00:41<00:23, 5.34it/s]
Loading 0: 57%|█████▋ | 165/291 [00:42<00:28, 4.44it/s]
Loading 0: 57%|█████▋ | 166/291 [00:42<00:37, 3.34it/s]
Loading 0: 57%|█████▋ | 167/291 [00:43<00:47, 2.63it/s]
Loading 0: 58%|█████▊ | 170/291 [00:43<00:27, 4.47it/s]
Loading 0: 59%|█████▉ | 171/291 [00:43<00:25, 4.71it/s]
Loading 0: 59%|█████▉ | 172/291 [00:43<00:22, 5.29it/s]
Loading 0: 59%|█████▉ | 173/291 [00:44<00:33, 3.56it/s]
Loading 0: 60%|██████ | 175/291 [00:44<00:23, 4.89it/s]
Loading 0: 60%|██████ | 176/291 [00:44<00:22, 5.10it/s]
Loading 0: 61%|██████ | 177/291 [00:44<00:19, 5.75it/s]
Loading 0: 62%|██████▏ | 179/291 [00:45<00:24, 4.53it/s]
Loading 0: 62%|██████▏ | 180/291 [00:46<00:33, 3.33it/s]
Loading 0: 62%|██████▏ | 181/291 [00:46<00:42, 2.60it/s]
Loading 0: 63%|██████▎ | 184/291 [00:46<00:23, 4.55it/s]
Loading 0: 64%|██████▎ | 185/291 [00:47<00:22, 4.79it/s]
Loading 0: 64%|██████▍ | 186/291 [00:47<00:19, 5.39it/s]
Loading 0: 64%|██████▍ | 187/291 [00:47<00:18, 5.57it/s]
Loading 0: 65%|██████▍ | 188/291 [00:47<00:28, 3.57it/s]
Loading 0: 65%|██████▍ | 189/291 [00:48<00:38, 2.66it/s]
Loading 0: 66%|██████▌ | 192/291 [00:49<00:27, 3.61it/s]
Loading 0: 66%|██████▋ | 193/291 [00:49<00:33, 2.96it/s]
Loading 0: 67%|██████▋ | 194/291 [00:50<00:39, 2.45it/s]
Loading 0: 68%|██████▊ | 197/291 [00:50<00:22, 4.15it/s]
Loading 0: 68%|██████▊ | 198/291 [00:50<00:21, 4.42it/s]
Loading 0: 68%|██████▊ | 199/291 [00:50<00:18, 4.99it/s]
Loading 0: 69%|██████▉ | 201/291 [00:51<00:20, 4.29it/s]
Loading 0: 69%|██████▉ | 202/291 [00:51<00:27, 3.26it/s]
Loading 0: 70%|██████▉ | 203/291 [00:52<00:33, 2.60it/s]
Loading 0: 71%|███████ | 206/291 [00:52<00:19, 4.41it/s]
Loading 0: 71%|███████ | 207/291 [00:52<00:18, 4.66it/s]
Loading 0: 71%|███████▏ | 208/291 [00:53<00:15, 5.24it/s]
Loading 0: 72%|███████▏ | 210/291 [00:53<00:18, 4.41it/s]
Loading 0: 73%|███████▎ | 211/291 [00:54<00:24, 3.32it/s]
Loading 0: 73%|███████▎ | 212/291 [00:54<00:30, 2.63it/s]
Loading 0: 74%|███████▍ | 215/291 [00:55<00:17, 4.46it/s]
Loading 0: 74%|███████▍ | 216/291 [00:55<00:15, 4.70it/s]
Loading 0: 75%|███████▍ | 217/291 [00:55<00:14, 5.22it/s]
Loading 0: 75%|███████▌ | 219/291 [00:55<00:16, 4.38it/s]
Loading 0: 76%|███████▌ | 220/291 [00:56<00:21, 3.33it/s]
Loading 0: 76%|███████▌ | 221/291 [00:57<00:26, 2.65it/s]
Loading 0: 77%|███████▋ | 224/291 [00:57<00:14, 4.53it/s]
Loading 0: 77%|███████▋ | 225/291 [00:57<00:13, 4.77it/s]
Loading 0: 78%|███████▊ | 226/291 [00:57<00:12, 5.31it/s]
Loading 0: 78%|███████▊ | 228/291 [00:58<00:14, 4.40it/s]
Loading 0: 79%|███████▊ | 229/291 [00:58<00:18, 3.31it/s]
Loading 0: 79%|███████▉ | 230/291 [00:59<00:23, 2.62it/s]
Loading 0: 80%|████████ | 233/291 [00:59<00:13, 4.44it/s]
Loading 0: 80%|████████ | 234/291 [00:59<00:12, 4.68it/s]
Loading 0: 81%|████████ | 235/291 [00:59<00:10, 5.21it/s]
Loading 0: 81%|████████▏ | 237/291 [01:00<00:12, 4.36it/s]
Loading 0: 82%|████████▏ | 238/291 [01:01<00:16, 3.30it/s]
Loading 0: 82%|████████▏ | 239/291 [01:01<00:19, 2.61it/s]
Loading 0: 83%|████████▎ | 242/291 [01:01<00:10, 4.49it/s]
Loading 0: 84%|████████▎ | 243/291 [01:02<00:10, 4.72it/s]
Loading 0: 84%|████████▍ | 244/291 [01:02<00:08, 5.26it/s]
Loading 0: 85%|████████▍ | 246/291 [01:02<00:10, 4.39it/s]
Loading 0: 85%|████████▍ | 247/291 [01:03<00:13, 3.30it/s]
Loading 0: 85%|████████▌ | 248/291 [01:03<00:16, 2.63it/s]
Loading 0: 86%|████████▋ | 251/291 [01:04<00:08, 4.45it/s]
Loading 0: 87%|████████▋ | 252/291 [01:04<00:08, 4.69it/s]
Loading 0: 87%|████████▋ | 253/291 [01:04<00:07, 5.28it/s]
Loading 0: 88%|████████▊ | 255/291 [01:04<00:08, 4.40it/s]
Loading 0: 88%|████████▊ | 256/291 [01:05<00:10, 3.32it/s]
Loading 0: 88%|████████▊ | 257/291 [01:06<00:12, 2.64it/s]
Loading 0: 89%|████████▉ | 260/291 [01:06<00:06, 4.52it/s]
Loading 0: 90%|████████▉ | 261/291 [01:06<00:06, 4.76it/s]
Loading 0: 90%|█████████ | 262/291 [01:06<00:05, 5.34it/s]
Loading 0: 91%|█████████ | 264/291 [01:07<00:06, 4.44it/s]
Loading 0: 91%|█████████ | 265/291 [01:07<00:07, 3.35it/s]
Loading 0: 91%|█████████▏| 266/291 [01:08<00:09, 2.61it/s]
Loading 0: 92%|█████████▏| 269/291 [01:08<00:04, 4.44it/s]
Loading 0: 93%|█████████▎| 270/291 [01:08<00:04, 4.69it/s]
Loading 0: 93%|█████████▎| 271/291 [01:08<00:03, 5.20it/s]
Loading 0: 94%|█████████▍| 273/291 [01:09<00:04, 4.38it/s]
Loading 0: 94%|█████████▍| 274/291 [01:10<00:05, 3.32it/s]
Loading 0: 95%|█████████▍| 275/291 [01:10<00:06, 2.64it/s]
Loading 0: 96%|█████████▌| 278/291 [01:10<00:02, 4.53it/s]
Loading 0: 96%|█████████▌| 279/291 [01:11<00:02, 4.76it/s]
Loading 0: 96%|█████████▌| 280/291 [01:11<00:02, 5.34it/s]
Loading 0: 97%|█████████▋| 281/291 [01:11<00:02, 3.59it/s]
Loading 0: 97%|█████████▋| 282/291 [01:12<00:03, 2.70it/s]
Loading 0: 98%|█████████▊| 284/291 [01:12<00:01, 3.87it/s]
Loading 0: 98%|█████████▊| 285/291 [01:12<00:01, 4.23it/s]
Loading 0: 98%|█████████▊| 286/291 [01:12<00:01, 4.91it/s]
Loading 0: 99%|█████████▊| 287/291 [01:13<00:00, 5.20it/s]
Loading 0: 99%|█████████▉| 288/291 [01:13<00:00, 3.39it/s]
Job alexdaoud-trainer-bagir-54351-v2-mkmlizer completed after 145.37s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-54351-v2-mkmlizer
Pipeline stage MKMLizer completed in 145.84s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-54351-v2
Waiting for inference service alexdaoud-trainer-bagir-54351-v2 to be ready
Inference service alexdaoud-trainer-bagir-54351-v2 ready after 120.4925332069397s
Pipeline stage MKMLDeployer completed in 121.10s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 5.329441785812378s
Received healthy response to inference request in 4.426349401473999s
Received healthy response to inference request in 2.92429518699646s
Received healthy response to inference request in 3.257283926010132s
Received healthy response to inference request in 3.0741541385650635s
5 requests
0 failed requests
5th percentile: 2.954266977310181
10th percentile: 2.9842387676239013
20th percentile: 3.0441823482513426
30th percentile: 3.1107800960540772
40th percentile: 3.1840320110321043
50th percentile: 3.257283926010132
60th percentile: 3.724910116195679
70th percentile: 4.192536306381226
80th percentile: 4.606967878341675
90th percentile: 4.968204832077026
95th percentile: 5.148823308944702
99th percentile: 5.293318090438842
mean time: 3.8023048877716064
%s, retrying in %s seconds...
Received healthy response to inference request in 2.8005268573760986s
Received healthy response to inference request in 2.639204263687134s
Received healthy response to inference request in 3.5984914302825928s
Received healthy response to inference request in 4.577802658081055s
Received healthy response to inference request in 3.1225223541259766s
5 requests
0 failed requests
5th percentile: 2.6714687824249266
10th percentile: 2.70373330116272
20th percentile: 2.768262338638306
30th percentile: 2.8649259567260743
40th percentile: 2.993724155426025
50th percentile: 3.1225223541259766
60th percentile: 3.312909984588623
70th percentile: 3.5032976150512694
80th percentile: 3.7943536758422853
90th percentile: 4.18607816696167
95th percentile: 4.381940412521362
99th percentile: 4.538630208969116
mean time: 3.3477095127105714
Pipeline stage StressChecker completed in 38.34s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.71s
Shutdown handler de-registered
alexdaoud-trainer-bagir_54351_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3555.40s
Shutdown handler de-registered
alexdaoud-trainer-bagir_54351_v2 status is now inactive due to auto deactivation removed underperforming models
alexdaoud-trainer-bagir_54351_v2 status is now torndown due to DeploymentManager action