Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name alexdaoud-trainer-bagir-6040-v1-mkmlizer
Waiting for job on alexdaoud-trainer-bagir-6040-v1-mkmlizer to finish
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ _____ __ __ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ /___/ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ Version: 0.11.12 ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ https://mk1.ai ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ The license key for the current software has been verified as ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ belonging to: ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ Chai Research Corp. ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ║ ║
alexdaoud-trainer-bagir-6040-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
alexdaoud-trainer-bagir-6040-v1-mkmlizer: quantized model in 83.656s
alexdaoud-trainer-bagir-6040-v1-mkmlizer: Processed model alexdaoud/trainer_bagir_2024-12-11-checkpoint-87 in 118.478s
alexdaoud-trainer-bagir-6040-v1-mkmlizer: creating bucket guanaco-mkml-models
alexdaoud-trainer-bagir-6040-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
alexdaoud-trainer-bagir-6040-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1
alexdaoud-trainer-bagir-6040-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1/config.json
alexdaoud-trainer-bagir-6040-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1/special_tokens_map.json
alexdaoud-trainer-bagir-6040-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1/tokenizer_config.json
alexdaoud-trainer-bagir-6040-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1/tokenizer.json
alexdaoud-trainer-bagir-6040-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/alexdaoud-trainer-bagir-6040-v1/flywheel_model.0.safetensors
alexdaoud-trainer-bagir-6040-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:55, 5.19it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:30, 3.17it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:01, 2.35it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:02, 4.52it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:01, 4.59it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:53, 5.22it/s]
Loading 0: 4%|▍ | 12/291 [00:02<01:04, 4.36it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:24, 3.27it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:47, 2.58it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:01, 4.43it/s]
Loading 0: 6%|▌ | 18/291 [00:04<00:58, 4.66it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:52, 5.22it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:01, 4.39it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:21, 3.30it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:43, 2.59it/s]
Loading 0: 9%|▉ | 26/291 [00:06<00:59, 4.42it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:56, 4.67it/s]
Loading 0: 10%|█ | 30/291 [00:07<00:56, 4.60it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:12, 3.57it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:31, 2.83it/s]
Loading 0: 12%|█▏ | 35/291 [00:08<00:56, 4.54it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:53, 4.76it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:54, 4.63it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:09, 3.62it/s]
Loading 0: 14%|█▍ | 41/291 [00:10<01:27, 2.87it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:53, 4.60it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:51, 4.79it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:46, 5.27it/s]
Loading 0: 16%|█▋ | 48/291 [00:11<00:54, 4.44it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:11, 3.37it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:31, 2.64it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:53, 4.45it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:50, 4.70it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:50, 4.60it/s]
Loading 0: 20%|█▉ | 58/291 [00:14<01:04, 3.59it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:21, 2.86it/s]
Loading 0: 21%|██▏ | 62/291 [00:15<00:50, 4.55it/s]
Loading 0: 22%|██▏ | 63/291 [00:15<00:47, 4.77it/s]
Loading 0: 22%|██▏ | 64/291 [00:15<00:42, 5.30it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:50, 4.46it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:06, 3.39it/s]
Loading 0: 23%|██▎ | 68/291 [00:17<01:23, 2.69it/s]
Loading 0: 24%|██▍ | 71/291 [00:17<00:48, 4.51it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:46, 4.74it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:41, 5.25it/s]
Loading 0: 26%|██▌ | 75/291 [00:18<00:48, 4.41it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:04, 3.34it/s]
Loading 0: 26%|██▋ | 77/291 [00:19<01:20, 2.65it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:46, 4.54it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:43, 4.78it/s]
Loading 0: 29%|██▊ | 83/291 [00:20<00:38, 5.37it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<00:54, 3.79it/s]
Loading 0: 29%|██▉ | 85/291 [00:21<01:08, 3.01it/s]
Loading 0: 30%|██▉ | 86/291 [00:22<01:22, 2.48it/s]
Loading 0: 31%|███ | 89/291 [00:22<00:46, 4.31it/s]
Loading 0: 31%|███ | 90/291 [00:22<00:43, 4.57it/s]
Loading 0: 32%|███▏ | 93/291 [00:23<00:43, 4.54it/s]
Loading 0: 32%|███▏ | 94/291 [00:23<00:55, 3.54it/s]
Loading 0: 33%|███▎ | 95/291 [00:24<01:09, 2.82it/s]
Loading 0: 34%|███▎ | 98/291 [00:24<00:42, 4.58it/s]
Loading 0: 34%|███▍ | 99/291 [00:24<00:40, 4.80it/s]
Loading 0: 35%|███▌ | 102/291 [00:25<00:40, 4.66it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:51, 3.63it/s]
Loading 0: 36%|███▌ | 104/291 [00:26<01:05, 2.86it/s]
Loading 0: 37%|███▋ | 107/291 [00:26<00:40, 4.54it/s]
Loading 0: 37%|███▋ | 108/291 [00:27<00:38, 4.77it/s]
Loading 0: 38%|███▊ | 111/291 [00:27<00:38, 4.67it/s]
Loading 0: 38%|███▊ | 112/291 [00:28<00:49, 3.64it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:01, 2.90it/s]
Loading 0: 40%|███▉ | 116/291 [00:29<00:38, 4.58it/s]
Loading 0: 40%|████ | 117/291 [00:29<00:36, 4.81it/s]
Loading 0: 41%|████ | 118/291 [00:29<00:32, 5.37it/s]
Loading 0: 41%|████ | 120/291 [00:30<00:38, 4.48it/s]
Loading 0: 42%|████▏ | 121/291 [00:30<00:50, 3.38it/s]
Loading 0: 42%|████▏ | 122/291 [00:31<01:03, 2.68it/s]
Loading 0: 43%|████▎ | 125/291 [00:31<00:36, 4.55it/s]
Loading 0: 43%|████▎ | 126/291 [00:31<00:34, 4.78it/s]
Loading 0: 44%|████▍ | 129/291 [00:32<00:34, 4.67it/s]
Loading 0: 45%|████▍ | 130/291 [00:32<00:44, 3.62it/s]
Loading 0: 45%|████▌ | 131/291 [00:33<00:55, 2.87it/s]
Loading 0: 46%|████▌ | 134/291 [00:33<00:34, 4.59it/s]
Loading 0: 46%|████▋ | 135/291 [00:33<00:32, 4.81it/s]
Loading 0: 47%|████▋ | 136/291 [00:33<00:29, 5.31it/s]
Loading 0: 47%|████▋ | 138/291 [00:34<00:34, 4.46it/s]
Loading 0: 48%|████▊ | 139/291 [00:35<00:44, 3.38it/s]
Loading 0: 48%|████▊ | 140/291 [00:35<00:56, 2.69it/s]
Loading 0: 49%|████▉ | 143/291 [00:35<00:32, 4.58it/s]
Loading 0: 49%|████▉ | 144/291 [00:36<00:30, 4.82it/s]
Loading 0: 51%|█████ | 147/291 [00:36<00:30, 4.69it/s]
Loading 0: 51%|█████ | 148/291 [00:37<00:39, 3.64it/s]
Loading 0: 51%|█████ | 149/291 [00:37<00:49, 2.89it/s]
Loading 0: 52%|█████▏ | 152/291 [00:38<00:30, 4.62it/s]
Loading 0: 53%|█████▎ | 153/291 [00:38<00:28, 4.83it/s]
Loading 0: 54%|█████▎ | 156/291 [00:38<00:28, 4.69it/s]
Loading 0: 54%|█████▍ | 157/291 [00:39<00:36, 3.66it/s]
Loading 0: 54%|█████▍ | 158/291 [00:40<00:45, 2.91it/s]
Loading 0: 55%|█████▌ | 161/291 [00:40<00:27, 4.65it/s]
Loading 0: 56%|█████▌ | 162/291 [00:40<00:26, 4.87it/s]
Loading 0: 57%|█████▋ | 165/291 [00:41<00:26, 4.71it/s]
Loading 0: 57%|█████▋ | 166/291 [00:41<00:34, 3.66it/s]
Loading 0: 57%|█████▋ | 167/291 [00:42<00:42, 2.90it/s]
Loading 0: 58%|█████▊ | 170/291 [00:42<00:26, 4.64it/s]
Loading 0: 59%|█████▉ | 171/291 [00:42<00:24, 4.85it/s]
Loading 0: 59%|█████▉ | 172/291 [00:42<00:22, 5.40it/s]
Loading 0: 59%|█████▉ | 173/291 [00:43<00:31, 3.70it/s]
Loading 0: 60%|██████ | 175/291 [00:43<00:23, 5.00it/s]
Loading 0: 60%|██████ | 176/291 [00:43<00:22, 5.22it/s]
Loading 0: 61%|██████ | 177/291 [00:43<00:19, 5.89it/s]
Loading 0: 62%|██████▏ | 179/291 [00:44<00:24, 4.63it/s]
Loading 0: 62%|██████▏ | 180/291 [00:44<00:32, 3.40it/s]
Loading 0: 62%|██████▏ | 181/291 [00:45<00:41, 2.67it/s]
Loading 0: 63%|██████▎ | 184/291 [00:45<00:22, 4.68it/s]
Loading 0: 64%|██████▎ | 185/291 [00:45<00:21, 4.90it/s]
Loading 0: 64%|██████▍ | 186/291 [00:46<00:19, 5.49it/s]
Loading 0: 64%|██████▍ | 187/291 [00:46<00:18, 5.57it/s]
Loading 0: 65%|██████▍ | 188/291 [00:46<00:28, 3.63it/s]
Loading 0: 65%|██████▍ | 189/291 [00:47<00:37, 2.73it/s]
Loading 0: 66%|██████▌ | 192/291 [00:47<00:26, 3.69it/s]
Loading 0: 66%|██████▋ | 193/291 [00:48<00:32, 3.03it/s]
Loading 0: 67%|██████▋ | 194/291 [00:49<00:38, 2.52it/s]
Loading 0: 68%|██████▊ | 197/291 [00:49<00:22, 4.26it/s]
Loading 0: 68%|██████▊ | 198/291 [00:49<00:20, 4.53it/s]
Loading 0: 69%|██████▉ | 201/291 [00:50<00:19, 4.53it/s]
Loading 0: 69%|██████▉ | 202/291 [00:50<00:25, 3.55it/s]
Loading 0: 70%|██████▉ | 203/291 [00:51<00:31, 2.83it/s]
Loading 0: 71%|███████ | 206/291 [00:51<00:18, 4.52it/s]
Loading 0: 71%|███████ | 207/291 [00:51<00:17, 4.75it/s]
Loading 0: 72%|███████▏ | 210/291 [00:52<00:17, 4.65it/s]
Loading 0: 73%|███████▎ | 211/291 [00:52<00:22, 3.62it/s]
Loading 0: 73%|███████▎ | 212/291 [00:53<00:27, 2.90it/s]
Loading 0: 74%|███████▍ | 215/291 [00:53<00:16, 4.58it/s]
Loading 0: 74%|███████▍ | 216/291 [00:53<00:15, 4.80it/s]
Loading 0: 75%|███████▍ | 217/291 [00:54<00:13, 5.34it/s]
Loading 0: 75%|███████▌ | 219/291 [00:54<00:15, 4.50it/s]
Loading 0: 76%|███████▌ | 220/291 [00:55<00:20, 3.41it/s]
Loading 0: 76%|███████▌ | 221/291 [00:55<00:25, 2.72it/s]
Loading 0: 77%|███████▋ | 224/291 [00:55<00:14, 4.57it/s]
Loading 0: 77%|███████▋ | 225/291 [00:56<00:13, 4.81it/s]
Loading 0: 78%|███████▊ | 228/291 [00:56<00:13, 4.70it/s]
Loading 0: 79%|███████▊ | 229/291 [00:57<00:17, 3.65it/s]
Loading 0: 79%|███████▉ | 230/291 [00:57<00:21, 2.88it/s]
Loading 0: 80%|████████ | 233/291 [00:58<00:12, 4.61it/s]
Loading 0: 80%|████████ | 234/291 [00:58<00:11, 4.83it/s]
Loading 0: 81%|████████ | 235/291 [00:58<00:10, 5.37it/s]
Loading 0: 81%|████████▏ | 237/291 [00:59<00:11, 4.52it/s]
Loading 0: 82%|████████▏ | 238/291 [00:59<00:15, 3.44it/s]
Loading 0: 82%|████████▏ | 239/291 [01:00<00:19, 2.72it/s]
Loading 0: 83%|████████▎ | 242/291 [01:00<00:10, 4.64it/s]
Loading 0: 84%|████████▎ | 243/291 [01:00<00:09, 4.87it/s]
Loading 0: 84%|████████▍ | 244/291 [01:00<00:08, 5.38it/s]
Loading 0: 85%|████████▍ | 246/291 [01:01<00:10, 4.49it/s]
Loading 0: 85%|████████▍ | 247/291 [01:01<00:12, 3.39it/s]
Loading 0: 85%|████████▌ | 248/291 [01:02<00:16, 2.69it/s]
Loading 0: 86%|████████▋ | 251/291 [01:02<00:08, 4.61it/s]
Loading 0: 87%|████████▋ | 252/291 [01:02<00:08, 4.85it/s]
Loading 0: 88%|████████▊ | 255/291 [01:03<00:07, 4.68it/s]
Loading 0: 88%|████████▊ | 256/291 [01:03<00:09, 3.64it/s]
Loading 0: 88%|████████▊ | 257/291 [01:04<00:11, 2.87it/s]
Loading 0: 89%|████████▉ | 260/291 [01:04<00:06, 4.59it/s]
Loading 0: 90%|████████▉ | 261/291 [01:04<00:06, 4.82it/s]
Loading 0: 90%|█████████ | 262/291 [01:05<00:05, 5.35it/s]
Loading 0: 91%|█████████ | 264/291 [01:05<00:05, 4.50it/s]
Loading 0: 91%|█████████ | 265/291 [01:06<00:07, 3.43it/s]
Loading 0: 91%|█████████▏| 266/291 [01:06<00:09, 2.72it/s]
Loading 0: 92%|█████████▏| 269/291 [01:07<00:04, 4.64it/s]
Loading 0: 93%|█████████▎| 270/291 [01:07<00:04, 4.87it/s]
Loading 0: 93%|█████████▎| 271/291 [01:07<00:03, 5.46it/s]
Loading 0: 94%|█████████▍| 273/291 [01:07<00:03, 4.55it/s]
Loading 0: 94%|█████████▍| 274/291 [01:08<00:04, 3.42it/s]
Loading 0: 95%|█████████▍| 275/291 [01:09<00:05, 2.70it/s]
Loading 0: 96%|█████████▌| 278/291 [01:09<00:02, 4.59it/s]
Loading 0: 96%|█████████▌| 279/291 [01:09<00:02, 4.83it/s]
Loading 0: 96%|█████████▌| 280/291 [01:09<00:02, 5.42it/s]
Loading 0: 97%|█████████▋| 281/291 [01:10<00:02, 3.65it/s]
Loading 0: 97%|█████████▋| 282/291 [01:10<00:03, 2.73it/s]
Loading 0: 98%|█████████▊| 284/291 [01:10<00:01, 3.92it/s]
Loading 0: 98%|█████████▊| 285/291 [01:11<00:01, 4.28it/s]
Loading 0: 98%|█████████▊| 286/291 [01:11<00:01, 4.86it/s]
Loading 0: 99%|█████████▊| 287/291 [01:11<00:00, 5.11it/s]
Loading 0: 99%|█████████▉| 288/291 [01:11<00:00, 3.38it/s]
Job alexdaoud-trainer-bagir-6040-v1-mkmlizer completed after 146.11s with status: succeeded
Stopping job with name alexdaoud-trainer-bagir-6040-v1-mkmlizer
Pipeline stage MKMLizer completed in 146.61s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service alexdaoud-trainer-bagir-6040-v1
Waiting for inference service alexdaoud-trainer-bagir-6040-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service alexdaoud-trainer-bagir-6040-v1 ready after 221.45291137695312s
Pipeline stage MKMLDeployer completed in 222.05s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.50866436958313s
Received healthy response to inference request in 4.761958599090576s
Received healthy response to inference request in 3.5913963317871094s
Received healthy response to inference request in 4.014760255813599s
Received healthy response to inference request in 3.8497893810272217s
5 requests
0 failed requests
5th percentile: 3.5252107620239257
10th percentile: 3.5417571544647215
20th percentile: 3.5748499393463136
30th percentile: 3.643074941635132
40th percentile: 3.746432161331177
50th percentile: 3.8497893810272217
60th percentile: 3.9157777309417723
70th percentile: 3.9817660808563233
80th percentile: 4.164199924468995
90th percentile: 4.463079261779785
95th percentile: 4.61251893043518
99th percentile: 4.732070665359497
mean time: 3.9453137874603272
%s, retrying in %s seconds...
Received healthy response to inference request in 3.2659404277801514s
Received healthy response to inference request in 3.7703781127929688s
Received healthy response to inference request in 3.314518690109253s
Received healthy response to inference request in 5.231776237487793s
Received healthy response to inference request in 4.735229969024658s
5 requests
0 failed requests
5th percentile: 3.2756560802459718
10th percentile: 3.285371732711792
20th percentile: 3.3048030376434325
30th percentile: 3.405690574645996
40th percentile: 3.5880343437194826
50th percentile: 3.7703781127929688
60th percentile: 4.156318855285645
70th percentile: 4.54225959777832
80th percentile: 4.834539222717285
90th percentile: 5.033157730102539
95th percentile: 5.132466983795166
99th percentile: 5.211914386749267
mean time: 4.0635686874389645
%s, retrying in %s seconds...
Received healthy response to inference request in 3.448490858078003s
Received healthy response to inference request in 3.2833313941955566s
Received healthy response to inference request in 3.3376986980438232s
Received healthy response to inference request in 2.093634843826294s
Received healthy response to inference request in 5.312940835952759s
5 requests
0 failed requests
5th percentile: 2.3315741539001467
10th percentile: 2.569513463973999
20th percentile: 3.045392084121704
30th percentile: 3.29420485496521
40th percentile: 3.315951776504517
50th percentile: 3.3376986980438232
60th percentile: 3.382015562057495
70th percentile: 3.426332426071167
80th percentile: 3.8213808536529545
90th percentile: 4.5671608448028564
95th percentile: 4.940050840377808
99th percentile: 5.238362836837768
mean time: 3.495219326019287
Pipeline stage StressChecker completed in 62.02s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.50s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.10s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_6040_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6040-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6040-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6040-v1-profiler ready after 230.51018905639648s
Pipeline stage MKMLProfilerDeployer completed in 230.91s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplogwnlv:/code/chaiverse_profiler_1734445975 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplogwnlv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734445975 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734445975/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplogwnlv:/code/chaiverse_profiler_1734448743 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplogwnlv:/code/chaiverse_profiler_1734448744 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplogwnlv --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734448744 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734448744/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6040-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6040-v1-profiler
Service alexdaoud-trainer-bagir-6040-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.35s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6040-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.68s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6040-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6040-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6040-v1-profiler ready after 130.31182217597961s
Pipeline stage MKMLProfilerDeployer completed in 130.64s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplotkwb7:/code/chaiverse_profiler_1734449503 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplotkwb7 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734449503 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734449503/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplotkwb7:/code/chaiverse_profiler_1734452280 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplotkwb7 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734452280 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734452280/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6040-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6040-v1-profiler
Service alexdaoud-trainer-bagir-6040-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.45s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6040-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.65s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service alexdaoud-trainer-bagir-6040-v1-profiler
Waiting for inference service alexdaoud-trainer-bagir-6040-v1-profiler to be ready
Inference service alexdaoud-trainer-bagir-6040-v1-profiler ready after 50.173731327056885s
Pipeline stage MKMLProfilerDeployer completed in 50.66s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplojmrf8:/code/chaiverse_profiler_1734453057 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplojmrf8 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734453057 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734453057/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplojmrf8:/code/chaiverse_profiler_1734455842 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplojmrf8:/code/chaiverse_profiler_1734455843 --namespace tenant-chaiml-guanaco
kubectl exec -it alexdaoud-trainer-ba96d6d0db1a3cb325e0592b201aa4ff30-deplojmrf8 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1734455843 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 256 --output_tokens 1 --summary /code/chaiverse_profiler_1734455843/summary.json'
Received signal 2, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service alexdaoud-trainer-bagir-6040-v1-profiler is running
Tearing down inference service alexdaoud-trainer-bagir-6040-v1-profiler
Service alexdaoud-trainer-bagir-6040-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.66s
Shutdown handler de-registered
alexdaoud-trainer-bagir-_6040_v1 status is now inactive due to auto deactivation removed underperforming models