Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-20250331-reward-29517-v1-mkmlizer
Waiting for job on rirv938-20250331-reward-29517-v1-mkmlizer to finish
rirv938-20250331-reward-29517-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-20250331-reward-29517-v1-mkmlizer: ║ _____ __ __ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ /___/ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ Version: 0.12.8 ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ https://mk1.ai ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ belonging to: ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ Chai Research Corp. ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
rirv938-20250331-reward-29517-v1-mkmlizer: ║ ║
rirv938-20250331-reward-29517-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rirv938-20250331-reward-29517-v1-mkmlizer: Downloaded to shared memory in 29.420s
rirv938-20250331-reward-29517-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpv2vaqrch, device:0
rirv938-20250331-reward-29517-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-20250331-reward-29517-v1-mkmlizer: quantized model in 84.922s
rirv938-20250331-reward-29517-v1-mkmlizer: Processed model rirv938/20250331_reward_ablation_random_512_old in 114.342s
rirv938-20250331-reward-29517-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-20250331-reward-29517-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-20250331-reward-29517-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-20250331-reward-29517-v1
rirv938-20250331-reward-29517-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-20250331-reward-29517-v1/flywheel_model.0.safetensors
rirv938-20250331-reward-29517-v1-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 3/291 [00:00<00:55, 5.19it/s]
Loading 0: 1%|▏ | 4/291 [00:01<01:30, 3.17it/s]
Loading 0: 2%|▏ | 5/291 [00:01<02:01, 2.35it/s]
Loading 0: 3%|▎ | 8/291 [00:02<01:02, 4.51it/s]
Loading 0: 3%|▎ | 9/291 [00:02<01:01, 4.55it/s]
Loading 0: 3%|▎ | 10/291 [00:02<00:55, 5.06it/s]
Loading 0: 4%|▍ | 12/291 [00:02<01:05, 4.28it/s]
Loading 0: 4%|▍ | 13/291 [00:03<01:25, 3.24it/s]
Loading 0: 5%|▍ | 14/291 [00:04<01:47, 2.57it/s]
Loading 0: 6%|▌ | 17/291 [00:04<01:02, 4.40it/s]
Loading 0: 6%|▌ | 18/291 [00:04<00:58, 4.64it/s]
Loading 0: 7%|▋ | 19/291 [00:04<00:53, 5.12it/s]
Loading 0: 7%|▋ | 21/291 [00:05<01:02, 4.34it/s]
Loading 0: 8%|▊ | 22/291 [00:05<01:21, 3.29it/s]
Loading 0: 8%|▊ | 23/291 [00:06<01:42, 2.62it/s]
Loading 0: 9%|▉ | 26/291 [00:06<00:59, 4.45it/s]
Loading 0: 9%|▉ | 27/291 [00:06<00:56, 4.68it/s]
Loading 0: 10%|▉ | 28/291 [00:06<00:50, 5.17it/s]
Loading 0: 10%|█ | 30/291 [00:07<00:59, 4.37it/s]
Loading 0: 11%|█ | 31/291 [00:08<01:18, 3.31it/s]
Loading 0: 11%|█ | 32/291 [00:08<01:39, 2.61it/s]
Loading 0: 12%|█▏ | 35/291 [00:08<00:57, 4.44it/s]
Loading 0: 12%|█▏ | 36/291 [00:09<00:54, 4.67it/s]
Loading 0: 13%|█▎ | 37/291 [00:09<00:49, 5.14it/s]
Loading 0: 13%|█▎ | 39/291 [00:09<00:58, 4.34it/s]
Loading 0: 14%|█▎ | 40/291 [00:10<01:16, 3.29it/s]
Loading 0: 14%|█▍ | 41/291 [00:11<01:36, 2.59it/s]
Loading 0: 15%|█▌ | 44/291 [00:11<00:55, 4.46it/s]
Loading 0: 15%|█▌ | 45/291 [00:11<00:52, 4.69it/s]
Loading 0: 16%|█▌ | 46/291 [00:11<00:46, 5.27it/s]
Loading 0: 16%|█▋ | 48/291 [00:12<00:55, 4.41it/s]
Loading 0: 17%|█▋ | 49/291 [00:12<01:13, 3.30it/s]
Loading 0: 17%|█▋ | 50/291 [00:13<01:33, 2.57it/s]
Loading 0: 18%|█▊ | 53/291 [00:13<00:53, 4.42it/s]
Loading 0: 19%|█▊ | 54/291 [00:13<00:50, 4.66it/s]
Loading 0: 19%|█▉ | 55/291 [00:13<00:45, 5.17it/s]
Loading 0: 20%|█▉ | 57/291 [00:14<00:53, 4.36it/s]
Loading 0: 20%|█▉ | 58/291 [00:14<01:10, 3.31it/s]
Loading 0: 20%|██ | 59/291 [00:15<01:30, 2.58it/s]
Loading 0: 21%|██▏ | 62/291 [00:15<00:52, 4.38it/s]
Loading 0: 22%|██▏ | 63/291 [00:15<00:49, 4.60it/s]
Loading 0: 22%|██▏ | 64/291 [00:16<00:43, 5.17it/s]
Loading 0: 23%|██▎ | 66/291 [00:16<00:51, 4.37it/s]
Loading 0: 23%|██▎ | 67/291 [00:17<01:07, 3.32it/s]
Loading 0: 23%|██▎ | 68/291 [00:17<01:24, 2.63it/s]
Loading 0: 24%|██▍ | 71/291 [00:18<00:48, 4.50it/s]
Loading 0: 25%|██▍ | 72/291 [00:18<00:46, 4.73it/s]
Loading 0: 25%|██▌ | 73/291 [00:18<00:41, 5.23it/s]
Loading 0: 26%|██▌ | 75/291 [00:18<00:49, 4.40it/s]
Loading 0: 26%|██▌ | 76/291 [00:19<01:04, 3.32it/s]
Loading 0: 26%|██▋ | 77/291 [00:20<01:21, 2.63it/s]
Loading 0: 27%|██▋ | 80/291 [00:20<00:47, 4.46it/s]
Loading 0: 28%|██▊ | 81/291 [00:20<00:44, 4.68it/s]
Loading 0: 28%|██▊ | 82/291 [00:20<00:40, 5.20it/s]
Loading 0: 29%|██▊ | 83/291 [00:20<00:38, 5.36it/s]
Loading 0: 29%|██▉ | 84/291 [00:21<00:58, 3.52it/s]
Loading 0: 29%|██▉ | 85/291 [00:21<01:13, 2.79it/s]
Loading 0: 30%|██▉ | 86/291 [00:22<01:29, 2.30it/s]
Loading 0: 31%|███ | 89/291 [00:22<00:47, 4.27it/s]
Loading 0: 31%|███ | 90/291 [00:22<00:44, 4.54it/s]
Loading 0: 31%|███▏ | 91/291 [00:23<00:39, 5.11it/s]
Loading 0: 32%|███▏ | 93/291 [00:23<00:45, 4.32it/s]
Loading 0: 32%|███▏ | 94/291 [00:24<01:00, 3.27it/s]
Loading 0: 33%|███▎ | 95/291 [00:24<01:15, 2.59it/s]
Loading 0: 34%|███▎ | 98/291 [00:25<00:43, 4.44it/s]
Loading 0: 34%|███▍ | 99/291 [00:25<00:41, 4.68it/s]
Loading 0: 34%|███▍ | 100/291 [00:25<00:36, 5.26it/s]
Loading 0: 35%|███▌ | 102/291 [00:25<00:42, 4.40it/s]
Loading 0: 35%|███▌ | 103/291 [00:26<00:56, 3.32it/s]
Loading 0: 36%|███▌ | 104/291 [00:27<01:11, 2.61it/s]
Loading 0: 37%|███▋ | 107/291 [00:27<00:41, 4.42it/s]
Loading 0: 37%|███▋ | 108/291 [00:27<00:39, 4.67it/s]
Loading 0: 37%|███▋ | 109/291 [00:27<00:34, 5.24it/s]
Loading 0: 38%|███▊ | 111/291 [00:28<00:40, 4.41it/s]
Loading 0: 38%|███▊ | 112/291 [00:28<00:53, 3.35it/s]
Loading 0: 39%|███▉ | 113/291 [00:29<01:07, 2.63it/s]
Loading 0: 40%|███▉ | 116/291 [00:29<00:39, 4.46it/s]
Loading 0: 40%|████ | 117/291 [00:29<00:36, 4.70it/s]
Loading 0: 41%|████ | 118/291 [00:29<00:32, 5.26it/s]
Loading 0: 41%|████ | 120/291 [00:30<00:38, 4.44it/s]
Loading 0: 42%|████▏ | 121/291 [00:30<00:50, 3.36it/s]
Loading 0: 42%|████▏ | 122/291 [00:31<01:04, 2.63it/s]
Loading 0: 43%|████▎ | 125/291 [00:31<00:37, 4.45it/s]
Loading 0: 43%|████▎ | 126/291 [00:31<00:35, 4.69it/s]
Loading 0: 44%|████▎ | 127/291 [00:32<00:31, 5.24it/s]
Loading 0: 44%|████▍ | 129/291 [00:32<00:36, 4.40it/s]
Loading 0: 45%|████▍ | 130/291 [00:33<00:48, 3.33it/s]
Loading 0: 45%|████▌ | 131/291 [00:33<01:01, 2.60it/s]
Loading 0: 46%|████▌ | 134/291 [00:34<00:35, 4.41it/s]
Loading 0: 46%|████▋ | 135/291 [00:34<00:33, 4.65it/s]
Loading 0: 47%|████▋ | 136/291 [00:34<00:29, 5.17it/s]
Loading 0: 47%|████▋ | 138/291 [00:34<00:34, 4.38it/s]
Loading 0: 48%|████▊ | 139/291 [00:35<00:45, 3.31it/s]
Loading 0: 48%|████▊ | 140/291 [00:36<00:58, 2.60it/s]
Loading 0: 49%|████▉ | 143/291 [00:36<00:33, 4.41it/s]
Loading 0: 49%|████▉ | 144/291 [00:36<00:31, 4.65it/s]
Loading 0: 50%|████▉ | 145/291 [00:36<00:28, 5.20it/s]
Loading 0: 51%|█████ | 147/291 [00:37<00:32, 4.40it/s]
Loading 0: 51%|█████ | 148/291 [00:37<00:42, 3.34it/s]
Loading 0: 51%|█████ | 149/291 [00:38<00:53, 2.64it/s]
Loading 0: 52%|█████▏ | 152/291 [00:38<00:30, 4.48it/s]
Loading 0: 53%|█████▎ | 153/291 [00:38<00:29, 4.73it/s]
Loading 0: 53%|█████▎ | 154/291 [00:38<00:25, 5.29it/s]
Loading 0: 54%|█████▎ | 156/291 [00:39<00:30, 4.47it/s]
Loading 0: 54%|█████▍ | 157/291 [00:40<00:39, 3.37it/s]
Loading 0: 54%|█████▍ | 158/291 [00:40<00:50, 2.66it/s]
Loading 0: 55%|█████▌ | 161/291 [00:40<00:28, 4.56it/s]
Loading 0: 56%|█████▌ | 162/291 [00:41<00:26, 4.80it/s]
Loading 0: 56%|█████▌ | 163/291 [00:41<00:23, 5.38it/s]
Loading 0: 57%|█████▋ | 165/291 [00:41<00:27, 4.50it/s]
Loading 0: 57%|█████▋ | 166/291 [00:42<00:36, 3.40it/s]
Loading 0: 57%|█████▋ | 167/291 [00:42<00:46, 2.68it/s]
Loading 0: 58%|█████▊ | 170/291 [00:43<00:26, 4.60it/s]
Loading 0: 59%|█████▉ | 171/291 [00:43<00:24, 4.84it/s]
Loading 0: 59%|█████▉ | 172/291 [00:43<00:21, 5.42it/s]
Loading 0: 59%|█████▉ | 173/291 [00:43<00:32, 3.64it/s]
Loading 0: 60%|██████ | 175/291 [00:44<00:23, 4.95it/s]
Loading 0: 60%|██████ | 176/291 [00:44<00:22, 5.17it/s]
Loading 0: 61%|██████ | 177/291 [00:44<00:19, 5.74it/s]
Loading 0: 62%|██████▏ | 179/291 [00:44<00:24, 4.58it/s]
Loading 0: 62%|██████▏ | 180/291 [00:45<00:32, 3.38it/s]
Loading 0: 62%|██████▏ | 181/291 [00:46<00:41, 2.65it/s]
Loading 0: 63%|██████▎ | 184/291 [00:46<00:23, 4.57it/s]
Loading 0: 64%|██████▎ | 185/291 [00:46<00:22, 4.81it/s]
Loading 0: 64%|██████▍ | 186/291 [00:46<00:19, 5.36it/s]
Loading 0: 64%|██████▍ | 187/291 [00:46<00:18, 5.56it/s]
Loading 0: 65%|██████▍ | 188/291 [00:47<00:28, 3.63it/s]
Loading 0: 65%|██████▍ | 189/291 [00:47<00:37, 2.69it/s]
Loading 0: 66%|██████▌ | 192/291 [00:48<00:27, 3.65it/s]
Loading 0: 66%|██████▋ | 193/291 [00:49<00:32, 3.00it/s]
Loading 0: 67%|██████▋ | 194/291 [00:49<00:38, 2.49it/s]
Loading 0: 68%|██████▊ | 197/291 [00:49<00:22, 4.20it/s]
Loading 0: 68%|██████▊ | 198/291 [00:50<00:20, 4.47it/s]
Loading 0: 68%|██████▊ | 199/291 [00:50<00:18, 5.02it/s]
Loading 0: 69%|██████▉ | 201/291 [00:50<00:20, 4.33it/s]
Loading 0: 69%|██████▉ | 202/291 [00:51<00:26, 3.31it/s]
Loading 0: 70%|██████▉ | 203/291 [00:51<00:33, 2.63it/s]
Loading 0: 71%|███████ | 206/291 [00:52<00:19, 4.44it/s]
Loading 0: 71%|███████ | 207/291 [00:52<00:17, 4.68it/s]
Loading 0: 71%|███████▏ | 208/291 [00:52<00:16, 5.17it/s]
Loading 0: 72%|███████▏ | 210/291 [00:53<00:18, 4.40it/s]
Loading 0: 73%|███████▎ | 211/291 [00:53<00:23, 3.33it/s]
Loading 0: 73%|███████▎ | 212/291 [00:54<00:29, 2.64it/s]
Loading 0: 74%|███████▍ | 215/291 [00:54<00:16, 4.48it/s]
Loading 0: 74%|███████▍ | 216/291 [00:54<00:15, 4.73it/s]
Loading 0: 75%|███████▍ | 217/291 [00:54<00:14, 5.27it/s]
Loading 0: 75%|███████▌ | 219/291 [00:55<00:16, 4.42it/s]
Loading 0: 76%|███████▌ | 220/291 [00:55<00:21, 3.35it/s]
Loading 0: 76%|███████▌ | 221/291 [00:56<00:26, 2.64it/s]
Loading 0: 77%|███████▋ | 224/291 [00:56<00:14, 4.50it/s]
Loading 0: 77%|███████▋ | 225/291 [00:56<00:13, 4.75it/s]
Loading 0: 78%|███████▊ | 226/291 [00:56<00:12, 5.26it/s]
Loading 0: 78%|███████▊ | 228/291 [00:57<00:14, 4.44it/s]
Loading 0: 79%|███████▊ | 229/291 [00:58<00:18, 3.36it/s]
Loading 0: 79%|███████▉ | 230/291 [00:58<00:22, 2.66it/s]
Loading 0: 80%|████████ | 233/291 [00:58<00:12, 4.50it/s]
Loading 0: 80%|████████ | 234/291 [00:59<00:12, 4.75it/s]
Loading 0: 81%|████████ | 235/291 [00:59<00:10, 5.27it/s]
Loading 0: 81%|████████▏ | 237/291 [00:59<00:12, 4.45it/s]
Loading 0: 82%|████████▏ | 238/291 [01:00<00:15, 3.36it/s]
Loading 0: 82%|████████▏ | 239/291 [01:00<00:19, 2.65it/s]
Loading 0: 83%|████████▎ | 242/291 [01:01<00:10, 4.50it/s]
Loading 0: 84%|████████▎ | 243/291 [01:01<00:10, 4.74it/s]
Loading 0: 84%|████████▍ | 244/291 [01:01<00:08, 5.27it/s]
Loading 0: 85%|████████▍ | 246/291 [01:02<00:10, 4.41it/s]
Loading 0: 85%|████████▍ | 247/291 [01:02<00:13, 3.35it/s]
Loading 0: 85%|████████▌ | 248/291 [01:03<00:16, 2.65it/s]
Loading 0: 86%|████████▋ | 251/291 [01:03<00:08, 4.55it/s]
Loading 0: 87%|████████▋ | 252/291 [01:03<00:08, 4.79it/s]
Loading 0: 87%|████████▋ | 253/291 [01:03<00:07, 5.31it/s]
Loading 0: 88%|████████▊ | 255/291 [01:04<00:08, 4.44it/s]
Loading 0: 88%|████████▊ | 256/291 [01:04<00:10, 3.36it/s]
Loading 0: 88%|████████▊ | 257/291 [01:05<00:12, 2.65it/s]
Loading 0: 89%|████████▉ | 260/291 [01:05<00:06, 4.52it/s]
Loading 0: 90%|████████▉ | 261/291 [01:05<00:06, 4.76it/s]
Loading 0: 90%|█████████ | 262/291 [01:05<00:05, 5.29it/s]
Loading 0: 91%|█████████ | 264/291 [01:06<00:06, 4.44it/s]
Loading 0: 91%|█████████ | 265/291 [01:07<00:07, 3.37it/s]
Loading 0: 91%|█████████▏| 266/291 [01:07<00:09, 2.65it/s]
Loading 0: 92%|█████████▏| 269/291 [01:07<00:04, 4.56it/s]
Loading 0: 93%|█████████▎| 270/291 [01:08<00:04, 4.81it/s]
Loading 0: 93%|█████████▎| 271/291 [01:08<00:03, 5.37it/s]
Loading 0: 94%|█████████▍| 273/291 [01:08<00:03, 4.52it/s]
Loading 0: 94%|█████████▍| 274/291 [01:09<00:04, 3.41it/s]
Loading 0: 95%|█████████▍| 275/291 [01:09<00:05, 2.69it/s]
Loading 0: 96%|█████████▌| 278/291 [01:10<00:02, 4.58it/s]
Loading 0: 96%|█████████▌| 279/291 [01:10<00:02, 4.83it/s]
Loading 0: 96%|█████████▌| 280/291 [01:10<00:02, 5.38it/s]
Loading 0: 97%|█████████▋| 281/291 [01:10<00:02, 3.65it/s]
Loading 0: 97%|█████████▋| 282/291 [01:11<00:03, 2.77it/s]
Loading 0: 98%|█████████▊| 284/291 [01:11<00:01, 3.97it/s]
Loading 0: 98%|█████████▊| 285/291 [01:11<00:01, 4.33it/s]
Loading 0: 98%|█████████▊| 286/291 [01:11<00:01, 4.99it/s]
Loading 0: 99%|█████████▊| 287/291 [01:12<00:00, 5.26it/s]
Loading 0: 99%|█████████▉| 288/291 [01:12<00:00, 3.45it/s]
Job rirv938-20250331-reward-29517-v1-mkmlizer completed after 137.21s with status: succeeded
Stopping job with name rirv938-20250331-reward-29517-v1-mkmlizer
Pipeline stage MKMLizer completed in 137.77s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-20250331-reward-29517-v1
Waiting for inference service rirv938-20250331-reward-29517-v1 to be ready
Inference service rirv938-20250331-reward-29517-v1 ready after 120.47721028327942s
Pipeline stage MKMLDeployer completed in 121.23s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.454972743988037s
Received healthy response to inference request in 2.554433584213257s
Received healthy response to inference request in 3.917017698287964s
Received healthy response to inference request in 2.0533199310302734s
Received healthy response to inference request in 4.462721347808838s
5 requests
0 failed requests
5th percentile: 2.15354266166687
10th percentile: 2.2537653923034666
20th percentile: 2.4542108535766602
30th percentile: 2.8269504070281983
40th percentile: 3.3719840526580813
50th percentile: 3.917017698287964
60th percentile: 4.132199716567993
70th percentile: 4.347381734848023
80th percentile: 4.456522464752197
90th percentile: 4.459621906280518
95th percentile: 4.461171627044678
99th percentile: 4.462411403656006
mean time: 3.488493061065674
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5925865173339844s
Received healthy response to inference request in 3.7024030685424805s
Received healthy response to inference request in 4.838760137557983s
Received healthy response to inference request in 4.195252180099487s
Received healthy response to inference request in 2.4024460315704346s
5 requests
0 failed requests
5th percentile: 2.4404741287231446
10th percentile: 2.4785022258758547
20th percentile: 2.5545584201812743
30th percentile: 2.8145498275756835
40th percentile: 3.258476448059082
50th percentile: 3.7024030685424805
60th percentile: 3.899542713165283
70th percentile: 4.096682357788086
80th percentile: 4.323953771591187
90th percentile: 4.581356954574585
95th percentile: 4.710058546066284
99th percentile: 4.813019819259644
mean time: 3.546289587020874
%s, retrying in %s seconds...
Received healthy response to inference request in 2.803217887878418s
Received healthy response to inference request in 3.6756105422973633s
Received healthy response to inference request in 2.475379705429077s
Received healthy response to inference request in 5.58915114402771s
Received healthy response to inference request in 2.7069969177246094s
5 requests
0 failed requests
5th percentile: 2.5217031478881835
10th percentile: 2.56802659034729
20th percentile: 2.660673475265503
30th percentile: 2.726241111755371
40th percentile: 2.7647294998168945
50th percentile: 2.803217887878418
60th percentile: 3.152174949645996
70th percentile: 3.501132011413574
80th percentile: 4.058318662643433
90th percentile: 4.823734903335572
95th percentile: 5.20644302368164
99th percentile: 5.512609519958496
mean time: 3.4500712394714355
Pipeline stage StressChecker completed in 56.69s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.70s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.71s
Shutdown handler de-registered
rirv938-20250331-reward_29517_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250331-reward-29517-v1-profiler
Waiting for inference service rirv938-20250331-reward-29517-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3589.66s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250331-reward-29517-v1-profiler
Waiting for inference service rirv938-20250331-reward-29517-v1-profiler to be ready
Inference service rirv938-20250331-reward-29517-v1-profiler ready after 30.151768922805786s
Pipeline stage MKMLProfilerDeployer completed in 30.61s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250331-rewbcc05ae1876df23fcf5390eea08ab1f6-deplo6h825:/code/chaiverse_profiler_1743462318 --namespace tenant-chaiml-guanaco
kubectl exec -it rirv938-20250331-rewbcc05ae1876df23fcf5390eea08ab1f6-deplo6h825 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1743462318 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 1 --summary /code/chaiverse_profiler_1743462318/summary.json'
Received signal 15, running shutdown handler
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250331-reward-29517-v1-profiler is running
Tearing down inference service rirv938-20250331-reward-29517-v1-profiler
Service rirv938-20250331-reward-29517-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.53s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250331-reward-29517-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 1.48s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service rirv938-20250331-reward-29517-v1-profiler
Waiting for inference service rirv938-20250331-reward-29517-v1-profiler to be ready
Inference service rirv938-20250331-reward-29517-v1-profiler ready after 30.146501541137695s
Pipeline stage MKMLProfilerDeployer completed in 30.50s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250331-rewbcc05ae1876df23fcf5390eea08ab1f6-deplo6h825:/code/chaiverse_profiler_1743465935 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250331-rewbcc05ae1876df23fcf5390eea08ab1f6-deplo6h825:/code/chaiverse_profiler_1743465937 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/rirv938-20250331-rewbcc05ae1876df23fcf5390eea08ab1f6-deplo6h825:/code/chaiverse_profiler_1743465938 --namespace tenant-chaiml-guanaco
clean up pipeline due to error=ISVCScriptError('Command failed with error: Defaulted container "kserve-container" out of: kserve-container, queue-proxy\nerror: unable to upgrade connection: container not found ("kserve-container")\n, output: ')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service rirv938-20250331-reward-29517-v1-profiler is running
Tearing down inference service rirv938-20250331-reward-29517-v1-profiler
Service rirv938-20250331-reward-29517-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.38s
Shutdown handler de-registered
rirv938-20250331-reward_29517_v1 status is now inactive due to auto deactivation removed underperforming models