Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-20250411-reward-33516-v3-mkmlizer
Waiting for job on rirv938-20250411-reward-33516-v3-mkmlizer to finish
rirv938-20250411-reward-33516-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-20250411-reward-33516-v3-mkmlizer: ║ _____ __ __ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ /___/ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ Version: 0.12.8 ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ https://mk1.ai ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ belonging to: ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ Chai Research Corp. ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
rirv938-20250411-reward-33516-v3-mkmlizer: ║ ║
rirv938-20250411-reward-33516-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-20250411-reward-33516-v3-mkmlizer: Downloaded to shared memory in 24.283s
rirv938-20250411-reward-33516-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpmf425tpr, device:0
rirv938-20250411-reward-33516-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-20250411-reward-33516-v3-mkmlizer: quantized model in 85.739s
rirv938-20250411-reward-33516-v3-mkmlizer: Processed model rirv938/20250411_reward_safety_1_2_removed in 110.022s
rirv938-20250411-reward-33516-v3-mkmlizer: creating bucket guanaco-mkml-models
rirv938-20250411-reward-33516-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-20250411-reward-33516-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v3
rirv938-20250411-reward-33516-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v3/config.json
rirv938-20250411-reward-33516-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v3/special_tokens_map.json
rirv938-20250411-reward-33516-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v3/tokenizer_config.json
rirv938-20250411-reward-33516-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-20250411-reward-33516-v3/tokenizer.json
rirv938-20250411-reward-33516-v3-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:57, 5.03it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:32, 3.09it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:04, 2.29it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:04, 4.38it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:03, 4.45it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:55, 5.07it/s]
Loading 0: 4%|▍ | 12/291 [00:02<01:06, 4.22it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:27, 3.17it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:50, 2.50it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:03, 4.31it/s]
Loading 0: 6%|▌ | 18/291 [00:04<01:00, 4.55it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:53, 5.11it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:03, 4.27it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:23, 3.22it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:46, 2.52it/s]
Loading 0: 9%|▉ | 26/291 [00:06<01:01, 4.30it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:58, 4.54it/s]
Loading 0: 10%|▉ | 28/291 [00:07<00:51, 5.12it/s]
Loading 0: 10%|█ | 30/291 [00:07<01:00, 4.29it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:20, 3.24it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:40, 2.57it/s]
Loading 0: 12%|█▏ | 35/291 [00:09<00:57, 4.42it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:55, 4.63it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:49, 5.15it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:58, 4.30it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:17, 3.24it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:38, 2.54it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:57, 4.33it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:53, 4.57it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:47, 5.14it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:56, 4.30it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:14, 3.26it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:34, 2.55it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:55, 4.31it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:52, 4.55it/s]
Loading 0: 19%|█▉ | 55/291 [00:14<00:46, 5.09it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:54, 4.29it/s]
Loading 0: 20%|█▉ | 58/291 [00:15<01:11, 3.24it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:30, 2.55it/s]
Loading 0: 21%|██▏ | 62/291 [00:16<00:52, 4.33it/s]
Loading 0: 22%|██▏ | 63/291 [00:16<00:49, 4.57it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:44, 5.13it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:52, 4.31it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:08, 3.25it/s]
Loading 0: 23%|██▎ | 68/291 [00:18<01:27, 2.56it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:50, 4.34it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:47, 4.58it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:42, 5.17it/s]
Loading 0: 26%|██▌ | 75/291 [00:19<00:49, 4.32it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:06, 3.23it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:24, 2.54it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:48, 4.36it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:45, 4.60it/s]
Loading 0: 28%|██▊ | 82/291 [00:21<00:40, 5.16it/s]
Loading 0: 29%|██▊ | 83/291 [00:21<00:40, 5.08it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<01:00, 3.40it/s]
Loading 0: 29%|██▉ | 85/291 [00:22<01:16, 2.71it/s]
Loading 0: 30%|██▉ | 86/291 [00:23<01:31, 2.24it/s]
Loading 0: 31%|███ | 89/291 [00:23<00:49, 4.11it/s]
Loading 0: 31%|███ | 90/291 [00:23<00:45, 4.39it/s]
Loading 0: 31%|███▏ | 91/291 [00:23<00:39, 5.01it/s]
Loading 0: 32%|███▏ | 93/291 [00:24<00:46, 4.23it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:01, 3.20it/s]
Loading 0: 33%|███▎ | 95/291 [00:25<01:17, 2.52it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:44, 4.37it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:41, 4.61it/s]
Loading 0: 34%|███▍ | 100/291 [00:25<00:36, 5.19it/s]
Loading 0: 35%|███▌ | 102/291 [00:26<00:43, 4.34it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:57, 3.26it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:12, 2.57it/s]
Loading 0: 37%|███▋ | 107/291 [00:27<00:41, 4.42it/s]
Loading 0: 37%|███▋ | 108/291 [00:28<00:39, 4.65it/s]
Loading 0: 37%|███▋ | 109/291 [00:28<00:34, 5.24it/s]
Loading 0: 38%|███▊ | 111/291 [00:28<00:41, 4.37it/s]
Loading 0: 38%|███▊ | 112/291 [00:29<00:54, 3.29it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:09, 2.58it/s]
Loading 0: 40%|███▉ | 116/291 [00:30<00:39, 4.39it/s]
Loading 0: 40%|████ | 117/291 [00:30<00:37, 4.63it/s]
Loading 0: 41%|████ | 118/291 [00:30<00:33, 5.20it/s]
Loading 0: 41%|████ | 120/291 [00:30<00:39, 4.36it/s]
Loading 0: 42%|████▏ | 121/291 [00:31<00:51, 3.29it/s]
Loading 0: 42%|████▏ | 122/291 [00:32<01:05, 2.60it/s]
Loading 0: 43%|████▎ | 125/291 [00:32<00:37, 4.39it/s]
Loading 0: 43%|████▎ | 126/291 [00:32<00:35, 4.63it/s]
Loading 0: 44%|████▎ | 127/291 [00:32<00:31, 5.16it/s]
Loading 0: 44%|████▍ | 129/291 [00:33<00:37, 4.34it/s]
Loading 0: 45%|████▍ | 130/291 [00:33<00:49, 3.28it/s]
Loading 0: 45%|████▌ | 131/291 [00:34<01:01, 2.59it/s]
Loading 0: 46%|████▌ | 134/291 [00:34<00:35, 4.44it/s]
Loading 0: 46%|████▋ | 135/291 [00:34<00:33, 4.66it/s]
Loading 0: 47%|████▋ | 136/291 [00:34<00:29, 5.24it/s]
Loading 0: 47%|████▋ | 138/291 [00:35<00:35, 4.35it/s]
Loading 0: 48%|████▊ | 139/291 [00:36<00:46, 3.28it/s]
Loading 0: 48%|████▊ | 140/291 [00:36<00:58, 2.59it/s]
Loading 0: 49%|████▉ | 143/291 [00:36<00:33, 4.46it/s]
Loading 0: 49%|████▉ | 144/291 [00:37<00:31, 4.69it/s]
Loading 0: 50%|████▉ | 145/291 [00:37<00:27, 5.27it/s]
Loading 0: 51%|█████ | 147/291 [00:37<00:32, 4.40it/s]
Loading 0: 51%|█████ | 148/291 [00:38<00:43, 3.32it/s]
Loading 0: 51%|█████ | 149/291 [00:39<00:54, 2.60it/s]
Loading 0: 52%|█████▏ | 152/291 [00:39<00:31, 4.48it/s]
Loading 0: 53%|█████▎ | 153/291 [00:39<00:29, 4.70it/s]
Loading 0: 53%|█████▎ | 154/291 [00:39<00:25, 5.28it/s]
Loading 0: 54%|█████▎ | 156/291 [00:40<00:30, 4.41it/s]
Loading 0: 54%|█████▍ | 157/291 [00:40<00:40, 3.33it/s]
Loading 0: 54%|█████▍ | 158/291 [00:41<00:50, 2.63it/s]
Loading 0: 55%|█████▌ | 161/291 [00:41<00:29, 4.45it/s]
Loading 0: 56%|█████▌ | 162/291 [00:41<00:27, 4.69it/s]
Loading 0: 56%|█████▌ | 163/291 [00:41<00:24, 5.23it/s]
Loading 0: 57%|█████▋ | 165/291 [00:42<00:28, 4.39it/s]
Loading 0: 57%|█████▋ | 166/291 [00:42<00:37, 3.32it/s]
Loading 0: 57%|█████▋ | 167/291 [00:43<00:47, 2.62it/s]
Loading 0: 58%|█████▊ | 170/291 [00:43<00:27, 4.44it/s]
Loading 0: 59%|█████▉ | 171/291 [00:43<00:25, 4.68it/s]
Loading 0: 59%|█████▉ | 172/291 [00:44<00:22, 5.26it/s]
Loading 0: 59%|█████▉ | 173/291 [00:44<00:33, 3.57it/s]
Loading 0: 60%|██████ | 175/291 [00:44<00:23, 4.89it/s]
Loading 0: 60%|██████ | 176/291 [00:44<00:22, 5.11it/s]
Loading 0: 61%|██████ | 177/291 [00:45<00:19, 5.72it/s]
Loading 0: 62%|██████▏ | 179/291 [00:45<00:24, 4.53it/s]
Loading 0: 62%|██████▏ | 180/291 [00:46<00:33, 3.33it/s]
Loading 0: 62%|██████▏ | 181/291 [00:46<00:42, 2.61it/s]
Loading 0: 63%|██████▎ | 184/291 [00:47<00:23, 4.49it/s]
Loading 0: 64%|██████▎ | 185/291 [00:47<00:22, 4.73it/s]
Loading 0: 64%|██████▍ | 186/291 [00:47<00:19, 5.31it/s]
Loading 0: 64%|██████▍ | 187/291 [00:47<00:19, 5.31it/s]
Loading 0: 65%|██████▍ | 188/291 [00:48<00:29, 3.50it/s]
Loading 0: 65%|██████▍ | 189/291 [00:48<00:38, 2.62it/s]
Loading 0: 66%|██████▌ | 192/291 [00:49<00:27, 3.58it/s]
Loading 0: 66%|██████▋ | 193/291 [00:49<00:33, 2.97it/s]
Loading 0: 67%|██████▋ | 194/291 [00:50<00:39, 2.48it/s]
Loading 0: 68%|██████▊ | 197/291 [00:50<00:22, 4.16it/s]
Loading 0: 68%|██████▊ | 198/291 [00:50<00:21, 4.43it/s]
Loading 0: 68%|██████▊ | 199/291 [00:51<00:18, 4.99it/s]
Loading 0: 69%|██████▉ | 201/291 [00:51<00:20, 4.30it/s]
Loading 0: 69%|██████▉ | 202/291 [00:52<00:27, 3.29it/s]
Loading 0: 70%|██████▉ | 203/291 [00:52<00:33, 2.62it/s]
Loading 0: 71%|███████ | 206/291 [00:53<00:19, 4.44it/s]
Loading 0: 71%|███████ | 207/291 [00:53<00:17, 4.67it/s]
Loading 0: 71%|███████▏ | 208/291 [00:53<00:15, 5.24it/s]
Loading 0: 72%|███████▏ | 210/291 [00:53<00:18, 4.40it/s]
Loading 0: 73%|███████▎ | 211/291 [00:54<00:24, 3.32it/s]
Loading 0: 73%|███████▎ | 212/291 [00:55<00:29, 2.64it/s]
Loading 0: 74%|███████▍ | 215/291 [00:55<00:16, 4.53it/s]
Loading 0: 74%|███████▍ | 216/291 [00:55<00:15, 4.83it/s]
Loading 0: 75%|███████▍ | 217/291 [00:55<00:13, 5.43it/s]
Loading 0: 75%|███████▌ | 219/291 [00:56<00:16, 4.48it/s]
Loading 0: 76%|███████▌ | 220/291 [00:56<00:21, 3.34it/s]
Loading 0: 76%|███████▌ | 221/291 [00:57<00:26, 2.59it/s]
Loading 0: 77%|███████▋ | 224/291 [00:57<00:15, 4.47it/s]
Loading 0: 77%|███████▋ | 225/291 [00:57<00:14, 4.70it/s]
Loading 0: 78%|███████▊ | 226/291 [00:57<00:12, 5.28it/s]
Loading 0: 78%|███████▊ | 228/291 [00:58<00:14, 4.41it/s]
Loading 0: 79%|███████▊ | 229/291 [00:58<00:18, 3.34it/s]
Loading 0: 79%|███████▉ | 230/291 [00:59<00:23, 2.65it/s]
Loading 0: 80%|████████ | 233/291 [00:59<00:12, 4.55it/s]
Loading 0: 80%|████████ | 234/291 [00:59<00:11, 4.78it/s]
Loading 0: 81%|████████ | 235/291 [01:00<00:10, 5.34it/s]
Loading 0: 81%|████████▏ | 237/291 [01:00<00:12, 4.45it/s]
Loading 0: 82%|████████▏ | 238/291 [01:01<00:15, 3.35it/s]
Loading 0: 82%|████████▏ | 239/291 [01:01<00:19, 2.64it/s]
Loading 0: 83%|████████▎ | 242/291 [01:01<00:10, 4.48it/s]
Loading 0: 84%|████████▎ | 243/291 [01:02<00:10, 4.72it/s]
Loading 0: 84%|████████▍ | 244/291 [01:02<00:08, 5.29it/s]
Loading 0: 85%|████████▍ | 246/291 [01:02<00:10, 4.43it/s]
Loading 0: 85%|████████▍ | 247/291 [01:03<00:13, 3.35it/s]
Loading 0: 85%|████████▌ | 248/291 [01:04<00:16, 2.65it/s]
Loading 0: 86%|████████▋ | 251/291 [01:04<00:08, 4.50it/s]
Loading 0: 87%|████████▋ | 252/291 [01:04<00:08, 4.74it/s]
Loading 0: 87%|████████▋ | 253/291 [01:04<00:07, 5.28it/s]
Loading 0: 88%|████████▊ | 255/291 [01:05<00:08, 4.42it/s]
Loading 0: 88%|████████▊ | 256/291 [01:05<00:10, 3.35it/s]
Loading 0: 88%|████████▊ | 257/291 [01:06<00:12, 2.65it/s]
Loading 0: 89%|████████▉ | 260/291 [01:06<00:06, 4.55it/s]
Loading 0: 90%|████████▉ | 261/291 [01:06<00:06, 4.78it/s]
Loading 0: 90%|█████████ | 262/291 [01:06<00:05, 5.37it/s]
Loading 0: 91%|█████████ | 264/291 [01:07<00:06, 4.46it/s]
Loading 0: 91%|█████████ | 265/291 [01:07<00:07, 3.36it/s]
Loading 0: 91%|█████████▏| 266/291 [01:08<00:09, 2.65it/s]
Loading 0: 92%|█████████▏| 269/291 [01:08<00:04, 4.55it/s]
Loading 0: 93%|█████████▎| 270/291 [01:08<00:04, 4.78it/s]
Loading 0: 93%|█████████▎| 271/291 [01:08<00:03, 5.37it/s]
Loading 0: 94%|█████████▍| 273/291 [01:09<00:04, 4.46it/s]
Loading 0: 94%|█████████▍| 274/291 [01:10<00:05, 3.36it/s]
Loading 0: 95%|█████████▍| 275/291 [01:10<00:06, 2.66it/s]
Loading 0: 96%|█████████▌| 278/291 [01:10<00:02, 4.57it/s]
Loading 0: 96%|█████████▌| 279/291 [01:11<00:02, 4.80it/s]
Loading 0: 96%|█████████▌| 280/291 [01:11<00:02, 5.40it/s]
Loading 0: 97%|█████████▋| 281/291 [01:11<00:02, 3.63it/s]
Loading 0: 97%|█████████▋| 282/291 [01:12<00:03, 2.73it/s]
Loading 0: 98%|█████████▊| 284/291 [01:12<00:01, 3.91it/s]
Loading 0: 98%|█████████▊| 285/291 [01:12<00:01, 4.26it/s]
Loading 0: 98%|█████████▊| 286/291 [01:12<00:01, 4.85it/s]
Loading 0: 99%|█████████▊| 287/291 [01:13<00:00, 4.99it/s]
Loading 0: 99%|█████████▉| 288/291 [01:13<00:00, 3.33it/s]
Job rirv938-20250411-reward-33516-v3-mkmlizer completed after 134.97s with status: succeeded
Stopping job with name rirv938-20250411-reward-33516-v3-mkmlizer
Pipeline stage MKMLizer completed in 135.43s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-20250411-reward-33516-v3
Waiting for inference service rirv938-20250411-reward-33516-v3 to be ready
Inference service rirv938-20250411-reward-33516-v3 ready after 140.50802326202393s
Pipeline stage MKMLDeployer completed in 141.00s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.9055092334747314s
Received healthy response to inference request in 3.9379734992980957s
Received healthy response to inference request in 3.893483877182007s
Received healthy response to inference request in 4.228971481323242s
Received healthy response to inference request in 3.554919481277466s
5 requests
0 failed requests
5th percentile: 3.622632360458374
10th percentile: 3.6903452396392824
20th percentile: 3.8257709980010985
30th percentile: 3.8958889484405517
40th percentile: 3.900699090957642
50th percentile: 3.9055092334747314
60th percentile: 3.918494939804077
70th percentile: 3.931480646133423
80th percentile: 3.996173095703125
90th percentile: 4.112572288513183
95th percentile: 4.170771884918213
99th percentile: 4.2173315620422365
mean time: 3.904171514511108
%s, retrying in %s seconds...
Received healthy response to inference request in 2.7098026275634766s
Received healthy response to inference request in 2.9890518188476562s
Received healthy response to inference request in 5.706108808517456s
Received healthy response to inference request in 2.623260974884033s
Received healthy response to inference request in 2.4270927906036377s
5 requests
0 failed requests
5th percentile: 2.466326427459717
10th percentile: 2.505560064315796
20th percentile: 2.584027338027954
30th percentile: 2.640569305419922
40th percentile: 2.675185966491699
50th percentile: 2.7098026275634766
60th percentile: 2.8215023040771485
70th percentile: 2.93320198059082
80th percentile: 3.5324632167816166
90th percentile: 4.619286012649536
95th percentile: 5.162697410583496
99th percentile: 5.597426528930664
mean time: 3.291063404083252
Pipeline stage StressChecker completed in 38.47s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.71s
Shutdown handler de-registered
rirv938-20250411-reward_33516_v3 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.13s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v3-profiler
Waiting for inference service rirv938-20250411-reward-33516-v3-profiler to be ready
Inference service rirv938-20250411-reward-33516-v3-profiler ready after 140.93961906433105s
Pipeline stage MKMLProfilerDeployer completed in 141.47s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplof549p:/code/chaiverse_profiler_1746667187 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplof549p --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1746667187 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1746667187/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v3-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v3-profiler
Service rirv938-20250411-reward-33516-v3-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.86s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v3-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.81s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v3-profiler
Waiting for inference service rirv938-20250411-reward-33516-v3-profiler to be ready
Inference service rirv938-20250411-reward-33516-v3-profiler ready after 80.49171113967896s
Pipeline stage MKMLProfilerDeployer completed in 80.95s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplodxbrk:/code/chaiverse_profiler_1746670758 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplodxbrk --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1746670758 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1746670758/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v3-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v3-profiler
Service rirv938-20250411-reward-33516-v3-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.96s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v3-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.08s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250411-reward-33516-v3-profiler
Waiting for inference service rirv938-20250411-reward-33516-v3-profiler to be ready
Inference service rirv938-20250411-reward-33516-v3-profiler ready after 130.76901865005493s
Pipeline stage MKMLProfilerDeployer completed in 131.20s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplow25n5:/code/chaiverse_profiler_1746674435 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250411-rew9d6a600fe46607e3c24ccb074479bb2d-deplow25n5 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1746674435 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1746674435/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250411-reward-33516-v3-profiler is running
Tearing down inference service rirv938-20250411-reward-33516-v3-profiler
Service rirv938-20250411-reward-33516-v3-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.07s
Shutdown handler de-registered
rirv938-20250411-reward_33516_v3 status is now inactive due to auto deactivation removed underperforming models
rirv938-20250411-reward_33516_v3 status is now torndown due to DeploymentManager action