Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-10344-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-10344-v1-mkmlizer to finish
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-10344-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-10344-v1-mkmlizer: quantized model in 83.786s
alexdaoud-trainer-bagir-10344-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-113 in 115.867s
alexdaoud-trainer-bagir-10344-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-10344-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-10344-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-10344-v1
alexdaoud-trainer-bagir-10344-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-10344-v1/config.json
alexdaoud-trainer-bagir-10344-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-10344-v1/special_tokens_map.json
alexdaoud-trainer-bagir-10344-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-10344-v1/tokenizer_config.json
alexdaoud-trainer-bagir-10344-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-10344-v1/tokenizer.json
alexdaoud-trainer-bagir-10344-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:58, 4.89it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:33, 3.08it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:01, 2.35it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:02, 4.49it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:01, 4.57it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:53, 5.23it/s]
Loading 0: 4%|▍ | 12/291 [00:02<01:04, 4.32it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:25, 3.23it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:48, 2.55it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:02, 4.37it/s]
Loading 0: 6%|▌ | 18/291 [00:04<00:59, 4.61it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:52, 5.22it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:02, 4.33it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:22, 3.27it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:43, 2.59it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:00, 4.40it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:56, 4.64it/s]
Loading 0: 10%|▉ | 28/291 [00:06<00:50, 5.20it/s]
Loading 0: 10%|█ | 30/291 [00:07<00:59, 4.36it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:18, 3.29it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:39, 2.61it/s]
Loading 0: 12%|█▏ | 35/291 [00:08<00:57, 4.44it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:54, 4.69it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:48, 5.28it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:57, 4.40it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:15, 3.33it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:36, 2.60it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:56, 4.40it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:52, 4.64it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:53, 4.52it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:08, 3.53it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:26, 2.80it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:53, 4.47it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:50, 4.69it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:51, 4.59it/s]
Loading 0: 20%|█▉ | 58/291 [00:14<01:05, 3.53it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:21, 2.84it/s]
Loading 0: 21%|██▏ | 62/291 [00:15<00:51, 4.48it/s]
Loading 0: 22%|██▏ | 63/291 [00:15<00:49, 4.63it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:49, 4.55it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:02, 3.58it/s]
Loading 0: 23%|██▎ | 68/291 [00:17<01:17, 2.87it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:48, 4.53it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:45, 4.81it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:40, 5.34it/s]
Loading 0: 26%|██▌ | 75/291 [00:18<00:48, 4.47it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:03, 3.38it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:20, 2.67it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:46, 4.55it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:43, 4.78it/s]
Loading 0: 29%|██▊ | 83/291 [00:20<00:39, 5.22it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<00:56, 3.70it/s]
Loading 0: 29%|██▉ | 85/291 [00:21<01:10, 2.94it/s]
Loading 0: 30%|██▉ | 86/291 [00:22<01:24, 2.43it/s]
Loading 0: 31%|███ | 89/291 [00:22<00:47, 4.28it/s]
Loading 0: 31%|███ | 90/291 [00:22<00:44, 4.54it/s]
Loading 0: 32%|███▏ | 93/291 [00:23<00:43, 4.51it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<00:55, 3.54it/s]
Loading 0: 33%|███▎ | 95/291 [00:24<01:09, 2.84it/s]
Loading 0: 34%|███▎ | 98/291 [00:24<00:42, 4.57it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:40, 4.75it/s]
Loading 0: 35%|███▌ | 102/291 [00:25<00:40, 4.65it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:52, 3.60it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:06, 2.81it/s]
Loading 0: 37%|███▋ | 107/291 [00:27<00:40, 4.51it/s]
Loading 0: 37%|███▋ | 108/291 [00:27<00:38, 4.73it/s]
Loading 0: 38%|███▊ | 110/291 [00:27<00:28, 6.38it/s]
Loading 0: 38%|███▊ | 112/291 [00:28<00:51, 3.45it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:02, 2.83it/s]
Loading 0: 40%|███▉ | 116/291 [00:29<00:39, 4.42it/s]
Loading 0: 40%|████ | 117/291 [00:29<00:37, 4.65it/s]
Loading 0: 41%|████ | 118/291 [00:29<00:33, 5.20it/s]
Loading 0: 41%|████ | 120/291 [00:30<00:38, 4.44it/s]
Loading 0: 42%|████▏ | 121/291 [00:30<00:50, 3.38it/s]
Loading 0: 42%|████▏ | 122/291 [00:31<01:03, 2.66it/s]
Loading 0: 43%|████▎ | 125/291 [00:31<00:37, 4.46it/s]
Loading 0: 43%|████▎ | 126/291 [00:31<00:35, 4.70it/s]
Loading 0: 44%|████▍ | 129/291 [00:32<00:35, 4.61it/s]
Loading 0: 45%|████▍ | 130/291 [00:33<00:44, 3.60it/s]
Loading 0: 45%|████▌ | 131/291 [00:33<00:56, 2.85it/s]
Loading 0: 46%|████▌ | 134/291 [00:33<00:34, 4.55it/s]
Loading 0: 46%|████▋ | 135/291 [00:34<00:32, 4.78it/s]
Loading 0: 47%|████▋ | 138/291 [00:34<00:32, 4.65it/s]
Loading 0: 48%|████▊ | 139/291 [00:35<00:41, 3.64it/s]
Loading 0: 48%|████▊ | 140/291 [00:35<00:52, 2.89it/s]
Loading 0: 49%|████▉ | 143/291 [00:36<00:32, 4.61it/s]
Loading 0: 49%|████▉ | 144/291 [00:36<00:30, 4.82it/s]
Loading 0: 51%|█████ | 147/291 [00:37<00:30, 4.69it/s]
Loading 0: 51%|█████ | 148/291 [00:37<00:39, 3.64it/s]
Loading 0: 51%|█████ | 149/291 [00:38<00:49, 2.87it/s]
Loading 0: 52%|█████▏ | 152/291 [00:38<00:30, 4.54it/s]
Loading 0: 53%|█████▎ | 153/291 [00:38<00:29, 4.76it/s]
Loading 0: 53%|█████▎ | 154/291 [00:38<00:25, 5.31it/s]
Loading 0: 54%|█████▎ | 156/291 [00:39<00:30, 4.47it/s]
Loading 0: 54%|█████▍ | 157/291 [00:39<00:39, 3.40it/s]
Loading 0: 54%|█████▍ | 158/291 [00:40<00:49, 2.68it/s]
Loading 0: 55%|█████▌ | 161/291 [00:40<00:28, 4.54it/s]
Loading 0: 56%|█████▌ | 162/291 [00:40<00:26, 4.85it/s]
Loading 0: 57%|█████▋ | 165/291 [00:41<00:26, 4.69it/s]
Loading 0: 57%|█████▋ | 166/291 [00:42<00:34, 3.63it/s]
Loading 0: 57%|█████▋ | 167/291 [00:42<00:43, 2.87it/s]
Loading 0: 58%|█████▊ | 170/291 [00:42<00:26, 4.64it/s]
Loading 0: 59%|█████▉ | 171/291 [00:43<00:24, 4.85it/s]
Loading 0: 59%|█████▉ | 173/291 [00:43<00:29, 4.03it/s]
Loading 0: 60%|██████ | 175/291 [00:43<00:22, 5.09it/s]
Loading 0: 60%|██████ | 176/291 [00:44<00:21, 5.26it/s]
Loading 0: 62%|██████▏ | 179/291 [00:44<00:22, 4.90it/s]
Loading 0: 62%|██████▏ | 180/291 [00:45<00:29, 3.75it/s]
Loading 0: 62%|██████▏ | 181/291 [00:45<00:37, 2.96it/s]
Loading 0: 63%|██████▎ | 184/291 [00:46<00:22, 4.77it/s]
Loading 0: 64%|██████▎ | 185/291 [00:46<00:21, 4.97it/s]
Loading 0: 64%|██████▍ | 187/291 [00:46<00:18, 5.47it/s]
Loading 0: 65%|██████▍ | 188/291 [00:47<00:26, 3.89it/s]
Loading 0: 65%|██████▍ | 189/291 [00:47<00:34, 2.96it/s]
Loading 0: 66%|██████▌ | 192/291 [00:48<00:26, 3.78it/s]
Loading 0: 66%|██████▋ | 193/291 [00:48<00:31, 3.13it/s]
Loading 0: 67%|██████▋ | 194/291 [00:49<00:37, 2.59it/s]
Loading 0: 68%|██████▊ | 197/291 [00:49<00:21, 4.28it/s]
Loading 0: 68%|██████▊ | 198/291 [00:49<00:20, 4.54it/s]
Loading 0: 68%|██████▊ | 199/291 [00:49<00:18, 5.11it/s]
Loading 0: 69%|██████▉ | 201/291 [00:50<00:20, 4.39it/s]
Loading 0: 69%|██████▉ | 202/291 [00:51<00:26, 3.36it/s]
Loading 0: 70%|██████▉ | 203/291 [00:51<00:32, 2.68it/s]
Loading 0: 71%|███████ | 206/291 [00:51<00:18, 4.54it/s]
Loading 0: 71%|███████ | 207/291 [00:52<00:17, 4.78it/s]
Loading 0: 71%|███████▏ | 208/291 [00:52<00:15, 5.37it/s]
Loading 0: 72%|███████▏ | 210/291 [00:52<00:18, 4.49it/s]
Loading 0: 73%|███████▎ | 211/291 [00:53<00:23, 3.39it/s]
Loading 0: 73%|███████▎ | 212/291 [00:53<00:29, 2.68it/s]
Loading 0: 74%|███████▍ | 215/291 [00:54<00:16, 4.55it/s]
Loading 0: 74%|███████▍ | 216/291 [00:54<00:15, 4.80it/s]
Loading 0: 75%|███████▌ | 219/291 [00:54<00:15, 4.67it/s]
Loading 0: 76%|███████▌ | 220/291 [00:55<00:19, 3.63it/s]
Loading 0: 76%|███████▌ | 221/291 [00:56<00:24, 2.90it/s]
Loading 0: 77%|███████▋ | 224/291 [00:56<00:14, 4.62it/s]
Loading 0: 77%|███████▋ | 225/291 [00:56<00:13, 4.85it/s]
Loading 0: 78%|███████▊ | 228/291 [00:57<00:13, 4.69it/s]
Loading 0: 79%|███████▊ | 229/291 [00:57<00:16, 3.66it/s]
Loading 0: 79%|███████▉ | 230/291 [00:58<00:20, 2.91it/s]
Loading 0: 80%|████████ | 233/291 [00:58<00:12, 4.61it/s]
Loading 0: 80%|████████ | 234/291 [00:58<00:11, 4.83it/s]
Loading 0: 81%|████████ | 235/291 [00:58<00:10, 5.36it/s]
Loading 0: 81%|████████▏ | 237/291 [00:59<00:11, 4.52it/s]
Loading 0: 82%|████████▏ | 238/291 [00:59<00:15, 3.42it/s]
Loading 0: 82%|████████▏ | 239/291 [01:00<00:19, 2.73it/s]
Loading 0: 83%|████████▎ | 242/291 [01:00<00:10, 4.64it/s]
Loading 0: 84%|████████▎ | 243/291 [01:00<00:09, 4.87it/s]
Loading 0: 84%|████████▍ | 244/291 [01:00<00:08, 5.46it/s]
Loading 0: 85%|████████▍ | 246/291 [01:01<00:09, 4.53it/s]
Loading 0: 85%|████████▍ | 247/291 [01:02<00:12, 3.39it/s]
Loading 0: 85%|████████▌ | 248/291 [01:02<00:15, 2.69it/s]
Loading 0: 86%|████████▋ | 251/291 [01:02<00:08, 4.56it/s]
Loading 0: 87%|████████▋ | 252/291 [01:03<00:08, 4.81it/s]
Loading 0: 87%|████████▋ | 253/291 [01:03<00:07, 5.41it/s]
Loading 0: 88%|████████▊ | 255/291 [01:03<00:07, 4.52it/s]
Loading 0: 88%|████████▊ | 256/291 [01:04<00:10, 3.41it/s]
Loading 0: 88%|████████▊ | 257/291 [01:04<00:12, 2.70it/s]
Loading 0: 89%|████████▉ | 260/291 [01:05<00:06, 4.58it/s]
Loading 0: 90%|████████▉ | 261/291 [01:05<00:06, 4.78it/s]
Loading 0: 90%|█████████ | 262/291 [01:05<00:05, 5.34it/s]
Loading 0: 91%|█████████ | 264/291 [01:05<00:06, 4.49it/s]
Loading 0: 91%|█████████ | 265/291 [01:06<00:07, 3.39it/s]
Loading 0: 91%|█████████▏| 266/291 [01:07<00:09, 2.70it/s]
Loading 0: 92%|█████████▏| 269/291 [01:07<00:04, 4.62it/s]
Loading 0: 93%|█████████▎| 270/291 [01:07<00:04, 4.84it/s]
Loading 0: 93%|█████████▎| 272/291 [01:07<00:02, 6.60it/s]
Loading 0: 94%|█████████▍| 274/291 [01:08<00:04, 3.47it/s]
Loading 0: 95%|█████████▍| 275/291 [01:09<00:05, 2.83it/s]
Loading 0: 96%|█████████▌| 278/291 [01:09<00:02, 4.53it/s]
Loading 0: 96%|█████████▌| 279/291 [01:09<00:02, 4.76it/s]
Loading 0: 97%|█████████▋| 281/291 [01:10<00:02, 3.99it/s]
Loading 0: 97%|█████████▋| 282/291 [01:10<00:02, 3.11it/s]
Loading 0: 98%|█████████▊| 284/291 [01:11<00:01, 4.17it/s]
Loading 0: 98%|█████████▊| 285/291 [01:11<00:01, 4.47it/s]
Loading 0: 99%|█████████▊| 287/291 [01:11<00:00, 4.97it/s]
Loading 0: 99%|█████████▉| 288/291 [01:12<00:00, 3.63it/s]
Job alexdaoud-trainer-bagir-10344-v1-mkmlizer completed after 146.27s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-10344-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.82s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-10344-v1
Waiting for inference service alexdaoud-trainer-bagir-10344-v1 to be ready
Inference service alexdaoud-trainer-bagir-10344-v1 ready after 180.98958802223206s
Pipeline stage MKMLDeployer completed in 181.47s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.4170243740081787s
Received healthy response to inference request in 3.8611035346984863s
Received healthy response to inference request in 2.891700506210327s
Received healthy response to inference request in 6.075774192810059s
Received healthy response to inference request in 3.954000473022461s
5 requests
0 failed requests
5th percentile: 2.9967652797698974
10th percentile: 3.1018300533294676
20th percentile: 3.3119596004486085
30th percentile: 3.50584020614624
40th percentile: 3.6834718704223635
50th percentile: 3.8611035346984863
60th percentile: 3.898262310028076
70th percentile: 3.935421085357666
80th percentile: 4.378355216979981
90th percentile: 5.22706470489502
95th percentile: 5.651419448852539
99th percentile: 5.990903244018555
mean time: 4.039920616149902
%s, retrying in %s seconds...
Received healthy response to inference request in 2.9122231006622314s
Received healthy response to inference request in 3.522273540496826s
Received healthy response to inference request in 5.4783384799957275s
Received healthy response to inference request in 3.2627792358398438s
Received healthy response to inference request in 3.9581081867218018s
5 requests
0 failed requests
5th percentile: 2.982334327697754
10th percentile: 3.0524455547332763
20th percentile: 3.192668008804321
30th percentile: 3.31467809677124
40th percentile: 3.4184758186340334
50th percentile: 3.522273540496826
60th percentile: 3.6966073989868162
70th percentile: 3.8709412574768067
80th percentile: 4.262154245376587
90th percentile: 4.870246362686157
95th percentile: 5.174292421340942
99th percentile: 5.41752926826477
mean time: 3.8267445087432863
%s, retrying in %s seconds...
Received healthy response to inference request in 2.7971808910369873s
Received healthy response to inference request in 3.394509792327881s
Received healthy response to inference request in 3.863422393798828s
Received healthy response to inference request in 2.3346598148345947s
Received healthy response to inference request in 2.788529872894287s
5 requests
0 failed requests
5th percentile: 2.4254338264465334
10th percentile: 2.5162078380584716
20th percentile: 2.6977558612823485
30th percentile: 2.790260076522827
40th percentile: 2.7937204837799072
50th percentile: 2.7971808910369873
60th percentile: 3.0361124515533446
70th percentile: 3.275044012069702
80th percentile: 3.4882923126220704
90th percentile: 3.675857353210449
95th percentile: 3.7696398735046386
99th percentile: 3.84466588973999
mean time: 3.0356605529785154
Pipeline stage StressChecker completed in 58.51s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.77s
Shutdown handler de-registered
alexdaoud-trainer-bagir_10344_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 4437.16s
Shutdown handler de-registered
alexdaoud-trainer-bagir_10344_v1 status is now inactive due to auto deactivation removed underperforming models
alexdaoud-trainer-bagir_10344_v1 status is now torndown due to DeploymentManager action