Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiharshitsheoran-mn12-30209-v1-mkmlizer
Waiting for job on chaiharshitsheoran-mn12-30209-v1-mkmlizer to finish
HTTP Request: %s %s "%s %d %s"
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Downloaded to shared memory in 27.693s
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Checking if ChaiHarshitSheoran/mn12b_syn1_tune2 already exists in ChaiML
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Creating repo ChaiML/mn12b_syn1_tune2 and uploading /tmp/tmp7g8gc1aw to it
chaiharshitsheoran-mn12-30209-v1-mkmlizer:
0%| | 0/6 [00:00<?, ?it/s]
17%|█▋ | 1/6 [00:06<00:33, 6.61s/it]
33%|███▎ | 2/6 [00:11<00:23, 5.79s/it]
50%|█████ | 3/6 [00:17<00:17, 5.68s/it]
67%|██████▋ | 4/6 [00:25<00:13, 6.57s/it]
83%|████████▎ | 5/6 [00:33<00:07, 7.14s/it]
100%|██████████| 6/6 [00:33<00:00, 4.87s/it]
100%|██████████| 6/6 [00:33<00:00, 5.65s/it]
chaiharshitsheoran-mn12-30209-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmp7g8gc1aw, device:0
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-nis-sft-24b-osmo_64694_v1: HTTPConnectionPool(host='chaiml-nis-sft-24b-osmo-64694-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiharshitsheoran-mn12-30209-v1-mkmlizer: quantized model in 146.003s
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Processed model ChaiHarshitSheoran/mn12b_syn1_tune2 in 226.999s
chaiharshitsheoran-mn12-30209-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiharshitsheoran-mn12-30209-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiharshitsheoran-mn12-30209-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia
chaiharshitsheoran-mn12-30209-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia/config.json
chaiharshitsheoran-mn12-30209-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia/special_tokens_map.json
chaiharshitsheoran-mn12-30209-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia/tokenizer_config.json
chaiharshitsheoran-mn12-30209-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia/tokenizer.json
chaiharshitsheoran-mn12-30209-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiharshitsheoran-mn12-30209-v1/nvidia/flywheel_model.0.safetensors
chaiharshitsheoran-mn12-30209-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 3/363 [00:00<01:45, 3.42it/s]
Loading 0: 1%| | 4/363 [00:01<02:52, 2.08it/s]
Loading 0: 1%|▏ | 5/363 [00:02<03:47, 1.57it/s]
Loading 0: 2%|▏ | 8/363 [00:03<01:58, 3.00it/s]
Loading 0: 2%|▏ | 9/363 [00:03<01:54, 3.10it/s]
Loading 0: 3%|▎ | 10/363 [00:03<01:38, 3.60it/s]
Loading 0: 3%|▎ | 12/363 [00:04<01:59, 2.94it/s]
Loading 0: 4%|▎ | 13/363 [00:05<02:39, 2.19it/s]
Loading 0: 4%|▍ | 14/363 [00:06<03:20, 1.74it/s]
Loading 0: 5%|▍ | 17/363 [00:06<01:55, 2.99it/s]
Loading 0: 5%|▍ | 18/363 [00:06<01:49, 3.15it/s]
Loading 0: 5%|▌ | 19/363 [00:06<01:34, 3.64it/s]
Loading 0: 6%|▌ | 21/363 [00:07<01:54, 2.99it/s]
Loading 0: 6%|▌ | 22/363 [00:08<02:32, 2.23it/s]
Loading 0: 6%|▋ | 23/363 [00:09<03:10, 1.79it/s]
Loading 0: 7%|▋ | 26/363 [00:09<01:50, 3.04it/s]
Loading 0: 7%|▋ | 27/363 [00:10<01:45, 3.19it/s]
Loading 0: 8%|▊ | 28/363 [00:10<01:31, 3.67it/s]
Loading 0: 8%|▊ | 30/363 [00:10<01:50, 3.00it/s]
Loading 0: 9%|▊ | 31/363 [00:11<02:27, 2.24it/s]
Loading 0: 9%|▉ | 32/363 [00:12<03:04, 1.79it/s]
Loading 0: 10%|▉ | 35/363 [00:13<01:48, 3.04it/s]
Loading 0: 10%|▉ | 36/363 [00:13<01:42, 3.18it/s]
Loading 0: 10%|█ | 37/363 [00:13<01:28, 3.68it/s]
Loading 0: 11%|█ | 39/363 [00:14<01:48, 2.99it/s]
Loading 0: 11%|█ | 40/363 [00:15<02:24, 2.23it/s]
Loading 0: 11%|█▏ | 41/363 [00:16<03:00, 1.79it/s]
Loading 0: 12%|█▏ | 44/363 [00:16<01:45, 3.03it/s]
Loading 0: 12%|█▏ | 45/363 [00:16<01:40, 3.18it/s]
Loading 0: 13%|█▎ | 46/363 [00:16<01:26, 3.65it/s]
Loading 0: 13%|█▎ | 48/363 [00:17<01:45, 2.98it/s]
Loading 0: 13%|█▎ | 49/363 [00:18<02:20, 2.23it/s]
Loading 0: 14%|█▍ | 50/363 [00:19<02:55, 1.79it/s]
Loading 0: 15%|█▍ | 53/363 [00:19<01:42, 3.02it/s]
Loading 0: 15%|█▍ | 54/363 [00:20<01:37, 3.17it/s]
Loading 0: 15%|█▌ | 55/363 [00:20<01:25, 3.62it/s]
Loading 0: 15%|█▌ | 56/363 [00:21<02:07, 2.41it/s]
Loading 0: 16%|█▌ | 58/363 [00:21<01:34, 3.23it/s]
Loading 0: 16%|█▋ | 59/363 [00:21<01:30, 3.37it/s]
Loading 0: 17%|█▋ | 60/363 [00:21<01:16, 3.94it/s]
Loading 0: 17%|█▋ | 61/363 [00:21<01:08, 4.44it/s]
Loading 0: 17%|█▋ | 62/363 [00:22<02:00, 2.50it/s]
Loading 0: 17%|█▋ | 63/363 [00:23<02:38, 1.89it/s]
Loading 0: 18%|█▊ | 64/363 [00:24<03:11, 1.56it/s]
Loading 0: 18%|█▊ | 67/363 [00:24<01:42, 2.89it/s]
Loading 0: 19%|█▊ | 68/363 [00:25<01:36, 3.06it/s]
Loading 0: 19%|█▉ | 69/363 [00:25<01:22, 3.54it/s]
Loading 0: 20%|█▉ | 71/363 [00:26<01:40, 2.91it/s]
Loading 0: 20%|█▉ | 72/363 [00:26<02:13, 2.18it/s]
Loading 0: 20%|██ | 73/363 [00:27<02:46, 1.74it/s]
Loading 0: 21%|██ | 76/363 [00:28<01:36, 2.98it/s]
Loading 0: 21%|██ | 77/363 [00:28<01:31, 3.13it/s]
Loading 0: 21%|██▏ | 78/363 [00:28<01:18, 3.62it/s]
Loading 0: 22%|██▏ | 80/363 [00:29<01:35, 2.96it/s]
Loading 0: 22%|██▏ | 81/363 [00:30<02:07, 2.21it/s]
Loading 0: 23%|██▎ | 82/363 [00:31<02:39, 1.77it/s]
Loading 0: 23%|██▎ | 85/363 [00:31<01:32, 2.99it/s]
Loading 0: 24%|██▎ | 86/363 [00:31<01:28, 3.13it/s]
Loading 0: 24%|██▍ | 87/363 [00:31<01:16, 3.62it/s]
Loading 0: 25%|██▍ | 89/363 [00:32<01:32, 2.97it/s]
Loading 0: 25%|██▍ | 90/363 [00:33<02:03, 2.22it/s]
Loading 0: 25%|██▌ | 91/363 [00:34<02:33, 1.77it/s]
Loading 0: 26%|██▌ | 94/363 [00:34<01:29, 3.00it/s]
Loading 0: 26%|██▌ | 95/363 [00:35<01:24, 3.16it/s]
Loading 0: 26%|██▋ | 96/363 [00:35<01:13, 3.63it/s]
Loading 0: 27%|██▋ | 98/363 [00:36<01:29, 2.97it/s]
Loading 0: 27%|██▋ | 99/363 [00:37<01:59, 2.22it/s]
Loading 0: 28%|██▊ | 100/363 [00:37<02:28, 1.77it/s]
Loading 0: 28%|██▊ | 103/363 [00:38<01:26, 2.99it/s]
Loading 0: 29%|██▊ | 104/363 [00:38<01:22, 3.14it/s]
Loading 0: 29%|██▉ | 105/363 [00:38<01:11, 3.62it/s]
Loading 0: 29%|██▉ | 106/363 [00:39<01:46, 2.40it/s]
Loading 0: 30%|██▉ | 108/363 [00:39<01:19, 3.20it/s]
Loading 0: 30%|███ | 109/363 [00:40<01:15, 3.35it/s]
Loading 0: 30%|███ | 110/363 [00:40<01:04, 3.90it/s]
Loading 0: 31%|███ | 112/363 [00:41<01:22, 3.03it/s]
Loading 0: 31%|███ | 113/363 [00:42<01:57, 2.13it/s]
Loading 0: 32%|███▏ | 116/363 [00:42<01:33, 2.63it/s]
Loading 0: 32%|███▏ | 117/363 [00:43<01:56, 2.12it/s]
Loading 0: 33%|███▎ | 118/363 [00:44<02:19, 1.75it/s]
Loading 0: 33%|███▎ | 121/363 [00:45<01:24, 2.87it/s]
Loading 0: 34%|███▎ | 122/363 [00:45<01:19, 3.03it/s]
Loading 0: 34%|███▍ | 123/363 [00:45<01:08, 3.50it/s]
Loading 0: 34%|███▍ | 125/363 [00:46<01:21, 2.92it/s]
Loading 0: 35%|███▍ | 126/363 [00:47<01:47, 2.21it/s]
Loading 0: 35%|███▍ | 127/363 [00:48<02:12, 1.78it/s]
Loading 0: 36%|███▌ | 130/363 [00:48<01:17, 2.99it/s]
Loading 0: 36%|███▌ | 131/363 [00:48<01:13, 3.15it/s]
Loading 0: 36%|███▋ | 132/363 [00:48<01:03, 3.63it/s]
Loading 0: 37%|███▋ | 134/363 [00:49<01:17, 2.96it/s]
Loading 0: 37%|███▋ | 135/363 [00:50<01:42, 2.22it/s]
Loading 0: 37%|███▋ | 136/363 [00:51<02:08, 1.77it/s]
Loading 0: 38%|███▊ | 139/363 [00:51<01:14, 3.00it/s]
Loading 0: 39%|███▊ | 140/363 [00:51<01:10, 3.15it/s]
Loading 0: 39%|███▉ | 141/363 [00:52<01:00, 3.64it/s]
Loading 0: 39%|███▉ | 142/363 [00:52<00:52, 4.19it/s]
Loading 0: 39%|███▉ | 143/363 [00:53<01:27, 2.51it/s]
Loading 0: 40%|███▉ | 144/363 [00:54<01:57, 1.86it/s]
Loading 0: 40%|████ | 147/363 [00:54<01:28, 2.44it/s]
Loading 0: 41%|████ | 148/363 [00:55<01:47, 1.99it/s]
Loading 0: 41%|████ | 149/363 [00:56<02:08, 1.66it/s]
Loading 0: 42%|████▏ | 152/363 [00:57<01:15, 2.80it/s]
Loading 0: 42%|████▏ | 153/363 [00:57<01:10, 2.97it/s]
Loading 0: 42%|████▏ | 154/363 [00:57<01:00, 3.43it/s]
Loading 0: 43%|████▎ | 156/363 [00:58<01:11, 2.88it/s]
Loading 0: 43%|████▎ | 157/363 [00:59<01:34, 2.18it/s]
Loading 0: 44%|████▎ | 158/363 [01:00<01:56, 1.75it/s]
Loading 0: 44%|████▍ | 161/363 [01:00<01:08, 2.97it/s]
Loading 0: 45%|████▍ | 162/363 [01:00<01:04, 3.12it/s]
Loading 0: 45%|████▍ | 163/363 [01:00<00:55, 3.60it/s]
Loading 0: 45%|████▌ | 165/363 [01:01<01:06, 2.96it/s]
Loading 0: 46%|████▌ | 166/363 [01:02<01:28, 2.21it/s]
Loading 0: 46%|████▌ | 167/363 [01:03<01:50, 1.77it/s]
Loading 0: 47%|████▋ | 170/363 [01:03<01:04, 2.99it/s]
Loading 0: 47%|████▋ | 171/363 [01:04<01:01, 3.14it/s]
Loading 0: 47%|████▋ | 172/363 [01:04<00:52, 3.61it/s]
Loading 0: 48%|████▊ | 174/363 [01:05<01:03, 2.96it/s]
Loading 0: 48%|████▊ | 175/363 [01:05<01:24, 2.22it/s]
Loading 0: 48%|████▊ | 176/363 [01:06<01:45, 1.78it/s]
Loading 0: 49%|████▉ | 179/363 [01:07<01:01, 3.01it/s]
Loading 0: 50%|████▉ | 180/363 [01:07<00:57, 3.16it/s]
Loading 0: 50%|████▉ | 181/363 [01:07<00:49, 3.64it/s]
Loading 0: 50%|█████ | 183/363 [01:08<01:00, 2.98it/s]
Loading 0: 51%|█████ | 184/363 [01:09<01:20, 2.23it/s]
Loading 0: 51%|█████ | 185/363 [01:10<01:40, 1.77it/s]
Loading 0: 52%|█████▏ | 188/363 [01:10<00:58, 3.00it/s]
Loading 0: 52%|█████▏ | 189/363 [01:10<00:55, 3.12it/s]
Loading 0: 52%|█████▏ | 190/363 [01:10<00:47, 3.61it/s]
Loading 0: 53%|█████▎ | 192/363 [01:11<00:57, 2.96it/s]
Loading 0: 53%|█████▎ | 193/363 [01:12<01:16, 2.22it/s]
Loading 0: 53%|█████▎ | 194/363 [01:13<01:35, 1.78it/s]
Loading 0: 54%|█████▍ | 197/363 [01:13<00:55, 3.01it/s]
Loading 0: 55%|█████▍ | 198/363 [01:14<00:52, 3.16it/s]
Loading 0: 55%|█████▍ | 199/363 [01:14<00:45, 3.63it/s]
Loading 0: 55%|█████▌ | 201/363 [01:15<00:54, 2.98it/s]
Loading 0: 56%|█████▌ | 202/363 [01:15<01:12, 2.23it/s]
Loading 0: 56%|█████▌ | 203/363 [01:16<01:29, 1.78it/s]
Loading 0: 57%|█████▋ | 206/363 [01:17<00:52, 3.01it/s]
Loading 0: 57%|█████▋ | 207/363 [01:17<00:49, 3.17it/s]
Loading 0: 57%|█████▋ | 208/363 [01:17<00:43, 3.58it/s]
Loading 0: 58%|█████▊ | 210/363 [01:18<00:51, 2.95it/s]
Loading 0: 58%|█████▊ | 211/363 [01:19<01:08, 2.22it/s]
Loading 0: 58%|█████▊ | 212/363 [01:20<01:24, 1.78it/s]
Loading 0: 59%|█████▉ | 215/363 [01:20<00:49, 3.01it/s]
Loading 0: 60%|█████▉ | 216/363 [01:20<00:46, 3.16it/s]
Loading 0: 60%|█████▉ | 217/363 [01:20<00:40, 3.63it/s]
Loading 0: 60%|██████ | 218/363 [01:21<00:59, 2.42it/s]
Loading 0: 61%|██████ | 220/363 [01:22<00:44, 3.24it/s]
Loading 0: 61%|██████ | 221/363 [01:22<00:41, 3.39it/s]
Loading 0: 61%|██████ | 222/363 [01:22<00:35, 3.95it/s]
Loading 0: 61%|██████▏ | 223/363 [01:22<00:31, 4.50it/s]
Loading 0: 62%|██████▏ | 224/363 [01:23<00:54, 2.54it/s]
Loading 0: 62%|██████▏ | 225/363 [01:24<01:14, 1.85it/s]
Loading 0: 63%|██████▎ | 228/363 [01:25<00:54, 2.48it/s]
Loading 0: 63%|██████▎ | 229/363 [01:26<01:06, 2.01it/s]
Loading 0: 63%|██████▎ | 230/363 [01:27<01:19, 1.68it/s]
Loading 0: 64%|██████▍ | 233/363 [01:27<00:45, 2.84it/s]
Loading 0: 64%|██████▍ | 234/363 [01:27<00:42, 3.01it/s]
Loading 0: 65%|██████▍ | 235/363 [01:27<00:36, 3.48it/s]
Loading 0: 65%|██████▌ | 237/363 [01:28<00:43, 2.93it/s]
Loading 0: 66%|██████▌ | 238/363 [01:29<00:56, 2.21it/s]
Loading 0: 66%|██████▌ | 239/363 [01:30<01:09, 1.77it/s]
Loading 0: 67%|██████▋ | 242/363 [01:30<00:40, 3.00it/s]
Loading 0: 67%|██████▋ | 243/363 [01:30<00:38, 3.16it/s]
Loading 0: 67%|██████▋ | 244/363 [01:31<00:32, 3.64it/s]
Loading 0: 68%|██████▊ | 246/363 [01:31<00:39, 2.99it/s]
Loading 0: 68%|██████▊ | 247/363 [01:32<00:51, 2.24it/s]
Loading 0: 68%|██████▊ | 248/363 [01:33<01:04, 1.79it/s]
Loading 0: 69%|██████▉ | 251/363 [01:33<00:37, 3.02it/s]
Loading 0: 69%|██████▉ | 252/363 [01:34<00:34, 3.17it/s]
Loading 0: 70%|██████▉ | 253/363 [01:34<00:30, 3.66it/s]
Loading 0: 70%|███████ | 255/363 [01:35<00:36, 3.00it/s]
Loading 0: 71%|███████ | 256/363 [01:36<00:47, 2.24it/s]
Loading 0: 71%|███████ | 257/363 [01:37<00:59, 1.79it/s]
Loading 0: 72%|███████▏ | 260/363 [01:37<00:34, 3.02it/s]
Loading 0: 72%|███████▏ | 261/363 [01:37<00:32, 3.17it/s]
Loading 0: 72%|███████▏ | 262/363 [01:37<00:27, 3.63it/s]
Loading 0: 73%|███████▎ | 264/363 [01:38<00:33, 2.99it/s]
Loading 0: 73%|███████▎ | 265/363 [01:39<00:43, 2.24it/s]
Loading 0: 73%|███████▎ | 266/363 [01:40<00:54, 1.80it/s]
Loading 0: 74%|███████▍ | 269/363 [01:40<00:30, 3.04it/s]
Loading 0: 74%|███████▍ | 270/363 [01:40<00:29, 3.19it/s]
Loading 0: 75%|███████▍ | 271/363 [01:41<00:25, 3.67it/s]
Loading 0: 75%|███████▌ | 273/363 [01:41<00:29, 3.00it/s]
Loading 0: 75%|███████▌ | 274/363 [01:42<00:39, 2.24it/s]
Loading 0: 76%|███████▌ | 275/363 [01:43<00:49, 1.79it/s]
Loading 0: 77%|███████▋ | 278/363 [01:43<00:28, 3.03it/s]
Loading 0: 77%|███████▋ | 279/363 [01:44<00:26, 3.18it/s]
Loading 0: 77%|███████▋ | 280/363 [01:44<00:22, 3.64it/s]
Loading 0: 78%|███████▊ | 282/363 [01:45<00:27, 2.99it/s]
Loading 0: 78%|███████▊ | 283/363 [01:46<00:35, 2.24it/s]
Loading 0: 78%|███████▊ | 284/363 [01:47<00:44, 1.78it/s]
Loading 0: 79%|███████▉ | 287/363 [01:47<00:25, 3.01it/s]
Loading 0: 79%|███████▉ | 288/363 [01:47<00:23, 3.17it/s]
Loading 0: 80%|███████▉ | 289/363 [01:47<00:20, 3.65it/s]
Loading 0: 80%|████████ | 291/363 [01:48<00:24, 2.99it/s]
Loading 0: 80%|████████ | 292/363 [01:49<00:31, 2.24it/s]
Loading 0: 81%|████████ | 293/363 [01:50<00:39, 1.79it/s]
Loading 0: 82%|████████▏ | 296/363 [01:50<00:22, 3.03it/s]
Loading 0: 82%|████████▏ | 297/363 [01:50<00:20, 3.18it/s]
Loading 0: 82%|████████▏ | 298/363 [01:51<00:17, 3.66it/s]
Loading 0: 82%|████████▏ | 299/363 [01:51<00:26, 2.43it/s]
Loading 0: 83%|████████▎ | 301/363 [01:52<00:19, 3.25it/s]
Loading 0: 83%|████████▎ | 302/363 [01:52<00:17, 3.40it/s]
Loading 0: 83%|████████▎ | 303/363 [01:52<00:15, 3.96it/s]
Loading 0: 84%|████████▎ | 304/363 [01:52<00:15, 3.81it/s]
Loading 0: 84%|████████▍ | 306/363 [01:53<00:19, 2.99it/s]
Loading 0: 85%|████████▍ | 307/363 [01:54<00:26, 2.13it/s]
Loading 0: 85%|████████▌ | 310/363 [01:55<00:20, 2.64it/s]
Loading 0: 86%|████████▌ | 311/363 [01:56<00:24, 2.13it/s]
Loading 0: 86%|████████▌ | 312/363 [01:57<00:28, 1.76it/s]
Loading 0: 87%|████████▋ | 315/363 [01:57<00:16, 2.90it/s]
Loading 0: 87%|████████▋ | 316/363 [01:57<00:15, 3.06it/s]
Loading 0: 87%|████████▋ | 317/363 [01:57<00:13, 3.52it/s]
Loading 0: 88%|████████▊ | 319/363 [01:58<00:14, 2.94it/s]
Loading 0: 88%|████████▊ | 320/363 [01:59<00:19, 2.22it/s]
Loading 0: 88%|████████▊ | 321/363 [02:00<00:23, 1.77it/s]
Loading 0: 89%|████████▉ | 324/363 [02:00<00:13, 2.99it/s]
Loading 0: 90%|████████▉ | 325/363 [02:01<00:12, 3.15it/s]
Loading 0: 90%|████████▉ | 326/363 [02:01<00:10, 3.64it/s]
Loading 0: 90%|█████████ | 328/363 [02:02<00:11, 2.99it/s]
Loading 0: 91%|█████████ | 329/363 [02:03<00:15, 2.24it/s]
Loading 0: 91%|█████████ | 330/363 [02:03<00:18, 1.78it/s]
Loading 0: 92%|█████████▏| 333/363 [02:04<00:09, 3.01it/s]
Loading 0: 92%|█████████▏| 334/363 [02:04<00:09, 3.17it/s]
Loading 0: 92%|█████████▏| 335/363 [02:04<00:07, 3.66it/s]
Loading 0: 93%|█████████▎| 337/363 [02:05<00:08, 2.99it/s]
Loading 0: 93%|█████████▎| 338/363 [02:06<00:11, 2.24it/s]
Loading 0: 93%|█████████▎| 339/363 [02:07<00:13, 1.79it/s]
Loading 0: 94%|█████████▍| 342/363 [02:07<00:06, 3.03it/s]
Loading 0: 94%|█████████▍| 343/363 [02:07<00:06, 3.18it/s]
Loading 0: 95%|█████████▍| 344/363 [02:07<00:05, 3.66it/s]
Loading 0: 95%|█████████▌| 346/363 [02:08<00:05, 2.98it/s]
Loading 0: 96%|█████████▌| 347/363 [02:09<00:07, 2.23it/s]
Loading 0: 96%|█████████▌| 348/363 [02:10<00:08, 1.79it/s]
Loading 0: 97%|█████████▋| 351/363 [02:10<00:03, 3.02it/s]
Loading 0: 97%|█████████▋| 352/363 [02:11<00:03, 3.17it/s]
Loading 0: 97%|█████████▋| 353/363 [02:11<00:02, 3.65it/s]
Loading 0: 98%|█████████▊| 355/363 [02:12<00:02, 2.96it/s]
Loading 0: 98%|█████████▊| 356/363 [02:13<00:03, 2.22it/s]
Loading 0: 98%|█████████▊| 357/363 [02:13<00:03, 1.77it/s]
Loading 0: 99%|█████████▉| 360/363 [02:14<00:01, 3.00it/s]
Loading 0: 99%|█████████▉| 361/363 [02:14<00:00, 3.15it/s]
Loading 0: 100%|█████████▉| 362/363 [02:14<00:00, 3.64it/s]
Job chaiharshitsheoran-mn12-30209-v1-mkmlizer completed after 307.57s with status: succeeded
Stopping job with name chaiharshitsheoran-mn12-30209-v1-mkmlizer
Pipeline stage MKMLizer completed in 308.01s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiharshitsheoran-mn12-30209-v1
Waiting for inference service chaiharshitsheoran-mn12-30209-v1 to be ready
Inference service chaiharshitsheoran-mn12-30209-v1 ready after 60.07118892669678s
Pipeline stage MKMLDeployer completed in 61.11s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.1105234622955322s
Received healthy response to inference request in 1.3582305908203125s
Received healthy response to inference request in 0.6247367858886719s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 0.993488073348999s
Received healthy response to inference request in 1.178717851638794s
5 requests
0 failed requests
5th percentile: 0.6984870433807373
10th percentile: 0.7722373008728027
20th percentile: 0.9197378158569336
30th percentile: 1.0168951511383058
40th percentile: 1.063709306716919
50th percentile: 1.1105234622955322
60th percentile: 1.137801218032837
70th percentile: 1.1650789737701417
80th percentile: 1.2146203994750977
90th percentile: 1.286425495147705
95th percentile: 1.3223280429840087
99th percentile: 1.3510500812530517
mean time: 1.0531393527984618
Pipeline stage StressChecker completed in 6.72s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.69s
Shutdown handler de-registered
chaiharshitsheoran-mn12_30209_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2166.08s
Shutdown handler de-registered
chaiharshitsheoran-mn12_30209_v1 status is now inactive due to auto deactivation removed underperforming models
chaiharshitsheoran-mn12_30209_v1 status is now torndown due to DeploymentManager action