Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name nischaydnk-exp14-exp9d-94555-v11-mkmlizer
Waiting for job on nischaydnk-exp14-exp9d-94555-v11-mkmlizer to finish
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ https://mk1.ai ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ The license key for the current software has been verified as ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ belonging to: ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Chai Research Corp. ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ║ ║
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: Downloaded to shared memory in 67.526s
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmps9el6zxi, device:0
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: quantized model in 55.135s
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: Processed model NischayDnk/exp14-exp9DPOft-Chaidatav1dpo6k-Mistral24B in 122.662s
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: creating bucket guanaco-mkml-models
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/config.json
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/special_tokens_map.json
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/tokenizer_config.json
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/tokenizer.json
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/flywheel_model.1.safetensors
nischaydnk-exp14-exp9d-94555-v11-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nischaydnk-exp14-exp9d-94555-v11/flywheel_model.0.safetensors
nischaydnk-exp14-exp9d-94555-v11-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 4/363 [00:00<00:09, 37.16it/s]
Loading 0: 2%|▏ | 8/363 [00:00<00:12, 28.66it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:12, 29.21it/s]
Loading 0: 4%|▍ | 15/363 [00:00<00:14, 24.02it/s]
Loading 0: 6%|▌ | 21/363 [00:00<00:11, 31.08it/s]
Loading 0: 7%|▋ | 25/363 [00:00<00:12, 28.09it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:09, 35.43it/s]
Loading 0: 10%|▉ | 35/363 [00:01<00:14, 21.93it/s]
Loading 0: 10%|█ | 38/363 [00:01<00:14, 22.27it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:17, 18.07it/s]
Loading 0: 13%|█▎ | 46/363 [00:01<00:14, 22.54it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:13, 23.13it/s]
Loading 0: 14%|█▍ | 52/363 [00:02<00:14, 21.44it/s]
Loading 0: 15%|█▌ | 55/363 [00:02<00:13, 22.27it/s]
Loading 0: 16%|█▌ | 58/363 [00:02<00:13, 22.97it/s]
Loading 0: 17%|█▋ | 61/363 [00:02<00:13, 21.86it/s]
Loading 0: 18%|█▊ | 64/363 [00:02<00:13, 22.04it/s]
Loading 0: 19%|█▉ | 69/363 [00:02<00:12, 23.48it/s]
Loading 0: 20%|█▉ | 72/363 [00:03<00:17, 16.25it/s]
Loading 0: 20%|██ | 74/363 [00:03<00:18, 15.69it/s]
Loading 0: 21%|██▏ | 78/363 [00:03<00:14, 19.36it/s]
Loading 0: 22%|██▏ | 81/363 [00:03<00:16, 17.47it/s]
Loading 0: 23%|██▎ | 84/363 [00:03<00:15, 18.44it/s]
Loading 0: 24%|██▍ | 87/363 [00:04<00:17, 15.69it/s]
Loading 0: 25%|██▌ | 91/363 [00:04<00:14, 18.76it/s]
Loading 0: 26%|██▌ | 94/363 [00:04<00:14, 18.24it/s]
Loading 0: 27%|██▋ | 97/363 [00:04<00:14, 18.92it/s]
Loading 0: 28%|██▊ | 100/363 [00:04<00:13, 20.12it/s]
Loading 0: 28%|██▊ | 103/363 [00:04<00:12, 20.06it/s]
Loading 0: 29%|██▉ | 106/363 [00:04<00:12, 21.40it/s]
Loading 0: 30%|███ | 109/363 [00:05<00:15, 16.45it/s]
Loading 0: 31%|███ | 112/363 [00:05<00:13, 18.77it/s]
Loading 0: 32%|███▏ | 115/363 [00:05<00:13, 18.57it/s]
Loading 0: 33%|███▎ | 118/363 [00:05<00:12, 19.56it/s]
Loading 0: 33%|███▎ | 121/363 [00:05<00:11, 20.23it/s]
Loading 0: 34%|███▍ | 124/363 [00:05<00:12, 18.83it/s]
Loading 0: 35%|███▍ | 127/363 [00:06<00:11, 20.64it/s]
Loading 0: 36%|███▌ | 130/363 [00:06<00:10, 22.00it/s]
Loading 0: 37%|███▋ | 133/363 [00:06<00:10, 21.74it/s]
Loading 0: 37%|███▋ | 136/363 [00:06<00:09, 23.07it/s]
Loading 0: 38%|███▊ | 139/363 [00:06<00:09, 23.91it/s]
Loading 0: 39%|███▉ | 142/363 [00:06<00:09, 23.44it/s]
Loading 0: 40%|████ | 146/363 [00:06<00:07, 27.29it/s]
Loading 0: 41%|████▏ | 150/363 [00:07<00:10, 20.41it/s]
Loading 0: 42%|████▏ | 153/363 [00:07<00:12, 17.44it/s]
Loading 0: 43%|████▎ | 157/363 [00:07<00:09, 20.91it/s]
Loading 0: 44%|████▍ | 160/363 [00:07<00:09, 21.54it/s]
Loading 0: 45%|████▌ | 165/363 [00:07<00:08, 24.53it/s]
Loading 0: 46%|████▋ | 168/363 [00:07<00:09, 21.05it/s]
Loading 0: 47%|████▋ | 172/363 [00:08<00:07, 23.99it/s]
Loading 0: 48%|████▊ | 175/363 [00:08<00:07, 24.37it/s]
Loading 0: 49%|████▉ | 178/363 [00:08<00:07, 23.28it/s]
Loading 0: 50%|████▉ | 181/363 [00:08<00:07, 24.43it/s]
Loading 0: 51%|█████ | 186/363 [00:08<00:06, 26.09it/s]
Loading 0: 52%|█████▏ | 189/363 [00:09<00:11, 15.16it/s]
Loading 0: 53%|█████▎ | 193/363 [00:09<00:09, 18.24it/s]
Loading 0: 54%|█████▍ | 196/363 [00:09<00:08, 18.57it/s]
Loading 0: 55%|█████▍ | 199/363 [00:09<00:08, 19.97it/s]
Loading 0: 55%|█████▌ | 200/363 [00:20<00:08, 19.97it/s]
Loading 0: 55%|█████▌ | 201/363 [00:25<04:33, 1.69s/it]
Loading 0: 56%|█████▌ | 203/363 [00:25<03:33, 1.33s/it]
Loading 0: 57%|█████▋ | 208/363 [00:25<01:56, 1.33it/s]
Loading 0: 58%|█████▊ | 211/363 [00:26<01:25, 1.77it/s]
Loading 0: 59%|█████▊ | 213/363 [00:26<01:09, 2.17it/s]
Loading 0: 60%|█████▉ | 217/363 [00:26<00:43, 3.34it/s]
Loading 0: 61%|██████ | 220/363 [00:26<00:33, 4.30it/s]
Loading 0: 61%|██████▏ | 223/363 [00:26<00:24, 5.70it/s]
Loading 0: 62%|██████▏ | 226/363 [00:26<00:20, 6.69it/s]
Loading 0: 63%|██████▎ | 230/363 [00:27<00:15, 8.51it/s]
Loading 0: 65%|██████▍ | 235/363 [00:27<00:10, 12.42it/s]
Loading 0: 66%|██████▌ | 238/363 [00:27<00:08, 14.34it/s]
Loading 0: 66%|██████▋ | 241/363 [00:27<00:07, 15.76it/s]
Loading 0: 67%|██████▋ | 244/363 [00:27<00:06, 17.94it/s]
Loading 0: 68%|██████▊ | 247/363 [00:27<00:05, 19.67it/s]
Loading 0: 69%|██████▉ | 250/363 [00:27<00:05, 19.78it/s]
Loading 0: 70%|██████▉ | 253/363 [00:28<00:05, 20.75it/s]
Loading 0: 71%|███████ | 256/363 [00:28<00:04, 21.46it/s]
Loading 0: 71%|███████▏ | 259/363 [00:28<00:05, 20.58it/s]
Loading 0: 72%|███████▏ | 262/363 [00:28<00:04, 21.57it/s]
Loading 0: 73%|███████▎ | 266/363 [00:28<00:03, 24.38it/s]
Loading 0: 74%|███████▍ | 269/363 [00:28<00:05, 16.26it/s]
Loading 0: 75%|███████▍ | 272/363 [00:29<00:05, 17.29it/s]
Loading 0: 76%|███████▌ | 275/363 [00:29<00:05, 15.00it/s]
Loading 0: 77%|███████▋ | 280/363 [00:29<00:04, 19.64it/s]
Loading 0: 78%|███████▊ | 283/363 [00:29<00:03, 20.73it/s]
Loading 0: 79%|███████▉ | 286/363 [00:29<00:03, 20.54it/s]
Loading 0: 80%|███████▉ | 289/363 [00:29<00:03, 21.26it/s]
Loading 0: 80%|████████ | 292/363 [00:30<00:03, 22.24it/s]
Loading 0: 81%|████████▏ | 295/363 [00:30<00:03, 21.13it/s]
Loading 0: 82%|████████▏ | 298/363 [00:30<00:02, 22.23it/s]
Loading 0: 83%|████████▎ | 303/363 [00:30<00:02, 24.50it/s]
Loading 0: 84%|████████▍ | 306/363 [00:30<00:03, 15.18it/s]
Loading 0: 85%|████████▌ | 310/363 [00:31<00:02, 18.72it/s]
Loading 0: 86%|████████▌ | 313/363 [00:31<00:02, 19.15it/s]
Loading 0: 87%|████████▋ | 316/363 [00:31<00:02, 20.92it/s]
Loading 0: 88%|████████▊ | 319/363 [00:31<00:01, 22.34it/s]
Loading 0: 89%|████████▊ | 322/363 [00:31<00:01, 21.74it/s]
Loading 0: 90%|████████▉ | 325/363 [00:31<00:01, 22.33it/s]
Loading 0: 90%|█████████ | 328/363 [00:31<00:01, 19.92it/s]
Loading 0: 91%|█████████ | 331/363 [00:31<00:01, 20.26it/s]
Loading 0: 92%|█████████▏| 334/363 [00:32<00:01, 21.78it/s]
Loading 0: 93%|█████████▎| 337/363 [00:32<00:01, 19.41it/s]
Loading 0: 94%|█████████▎| 340/363 [00:32<00:01, 21.41it/s]
Loading 0: 94%|█████████▍| 343/363 [00:32<00:01, 13.18it/s]
Loading 0: 96%|█████████▌| 348/363 [00:33<00:00, 15.90it/s]
Loading 0: 97%|█████████▋| 353/363 [00:33<00:00, 20.59it/s]
Loading 0: 98%|█████████▊| 357/363 [00:33<00:00, 19.42it/s]
Loading 0: 100%|█████████▉| 362/363 [00:33<00:00, 24.05it/s]
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: quantized model in 57.388s
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: Processed model rirv938/96p_4ff_nischaydnk_exp14_exp9dp_94555_v6_cp1500_merged in 191.659s
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: creating bucket guanaco-mkml-models
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/config.json
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/special_tokens_map.json
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/tokenizer_config.json
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/tokenizer.json
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/flywheel_model.1.safetensors
rirv938-96p-4ff-nischay-86528-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-96p-4ff-nischay-86528-v2/flywheel_model.0.safetensors
rirv938-96p-4ff-nischay-86528-v2-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 3/363 [00:00<00:12, 29.47it/s]
Loading 0: 2%|▏ | 6/363 [00:00<00:26, 13.51it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:17, 19.89it/s]
Loading 0: 4%|▎ | 13/363 [00:00<00:26, 13.22it/s]
Loading 0: 4%|▍ | 15/363 [00:01<00:31, 10.93it/s]
Loading 0: 5%|▌ | 19/363 [00:01<00:22, 15.31it/s]
Loading 0: 6%|▌ | 22/363 [00:01<00:22, 14.99it/s]
Loading 0: 7%|▋ | 24/363 [00:01<00:23, 14.14it/s]
Loading 0: 8%|▊ | 28/363 [00:01<00:18, 17.96it/s]
Loading 0: 9%|▉ | 32/363 [00:01<00:15, 21.55it/s]
Loading 0: 10%|▉ | 35/363 [00:02<00:22, 14.83it/s]
Loading 0: 10%|█ | 37/363 [00:02<00:23, 14.12it/s]
Loading 0: 11%|█ | 39/363 [00:02<00:21, 15.01it/s]
Loading 0: 11%|█▏ | 41/363 [00:02<00:26, 11.96it/s]
Loading 0: 13%|█▎ | 46/363 [00:02<00:17, 17.96it/s]
Loading 0: 14%|█▍ | 50/363 [00:03<00:14, 21.04it/s]
Loading 0: 15%|█▍ | 53/363 [00:03<00:21, 14.38it/s]
Loading 0: 15%|█▌ | 55/363 [00:03<00:22, 13.41it/s]
Loading 0: 16%|█▌ | 57/363 [00:03<00:21, 14.14it/s]
Loading 0: 16%|█▋ | 59/363 [00:04<00:26, 11.52it/s]
Loading 0: 18%|█▊ | 64/363 [00:04<00:17, 17.05it/s]
Loading 0: 19%|█▊ | 68/363 [00:04<00:14, 19.96it/s]
Loading 0: 20%|█▉ | 71/363 [00:04<00:20, 14.08it/s]
Loading 0: 20%|██ | 73/363 [00:04<00:21, 13.38it/s]
Loading 0: 21%|██ | 75/363 [00:04<00:20, 14.31it/s]
Loading 0: 21%|██ | 77/363 [00:05<00:25, 11.31it/s]
Loading 0: 23%|██▎ | 82/363 [00:05<00:16, 16.86it/s]
Loading 0: 24%|██▎ | 86/363 [00:05<00:13, 19.87it/s]
Loading 0: 25%|██▍ | 89/363 [00:05<00:19, 14.02it/s]
Loading 0: 25%|██▌ | 91/363 [00:06<00:20, 13.32it/s]
Loading 0: 26%|██▌ | 95/363 [00:06<00:15, 16.90it/s]
Loading 0: 27%|██▋ | 99/363 [00:06<00:13, 20.14it/s]
Loading 0: 28%|██▊ | 102/363 [00:06<00:13, 18.82it/s]
Loading 0: 29%|██▉ | 105/363 [00:06<00:18, 13.84it/s]
Loading 0: 29%|██▉ | 107/363 [00:07<00:19, 12.94it/s]
Loading 0: 30%|███ | 109/363 [00:07<00:20, 12.66it/s]
Loading 0: 31%|███ | 111/363 [00:07<00:18, 13.71it/s]
Loading 0: 31%|███ | 113/363 [00:07<00:21, 11.51it/s]
Loading 0: 33%|███▎ | 118/363 [00:07<00:13, 17.99it/s]
Loading 0: 34%|███▎ | 122/363 [00:07<00:10, 22.13it/s]
Loading 0: 34%|███▍ | 125/363 [00:08<00:15, 15.72it/s]
Loading 0: 35%|███▌ | 128/363 [00:08<00:14, 16.61it/s]
Loading 0: 36%|███▌ | 131/363 [00:08<00:17, 12.95it/s]
Loading 0: 37%|███▋ | 136/363 [00:08<00:13, 17.31it/s]
Loading 0: 39%|███▊ | 140/363 [00:08<00:10, 20.49it/s]
Loading 0: 39%|███▉ | 143/363 [00:09<00:14, 15.25it/s]
Loading 0: 40%|████ | 146/363 [00:09<00:13, 15.92it/s]
Loading 0: 41%|████ | 148/363 [00:09<00:15, 14.14it/s]
Loading 0: 41%|████▏ | 150/363 [00:09<00:15, 13.48it/s]
Loading 0: 42%|████▏ | 154/363 [00:09<00:11, 17.83it/s]
Loading 0: 44%|████▎ | 158/363 [00:10<00:09, 21.77it/s]
Loading 0: 44%|████▍ | 161/363 [00:10<00:13, 15.47it/s]
Loading 0: 45%|████▌ | 164/363 [00:10<00:12, 16.00it/s]
Loading 0: 46%|████▌ | 166/363 [00:10<00:14, 13.72it/s]
Loading 0: 46%|████▋ | 168/363 [00:10<00:14, 13.14it/s]
Loading 0: 47%|████▋ | 172/363 [00:11<00:10, 17.64it/s]
Loading 0: 48%|████▊ | 176/363 [00:11<00:08, 21.81it/s]
Loading 0: 49%|████▉ | 179/363 [00:11<00:11, 15.48it/s]
Loading 0: 50%|█████ | 182/363 [00:11<00:11, 16.37it/s]
Loading 0: 51%|█████ | 185/363 [00:12<00:13, 12.97it/s]
Loading 0: 52%|█████▏ | 190/363 [00:12<00:09, 18.34it/s]
Loading 0: 53%|█████▎ | 194/363 [00:12<00:07, 21.76it/s]
Loading 0: 54%|█████▍ | 197/363 [00:12<00:10, 16.03it/s]
Loading 0: 55%|█████▌ | 200/363 [00:12<00:09, 16.68it/s]
Loading 0: 56%|█████▌ | 203/363 [00:27<03:45, 1.41s/it]
Loading 0: 57%|█████▋ | 207/363 [00:27<02:25, 1.07it/s]
Loading 0: 58%|█████▊ | 210/363 [00:27<01:46, 1.44it/s]
Loading 0: 59%|█████▊ | 213/363 [00:28<01:18, 1.91it/s]
Loading 0: 60%|█████▉ | 216/363 [00:28<01:00, 2.45it/s]
Loading 0: 60%|██████ | 219/363 [00:28<00:43, 3.33it/s]
Loading 0: 61%|██████ | 222/363 [00:28<00:33, 4.22it/s]
Loading 0: 62%|██████▏ | 226/363 [00:28<00:22, 6.17it/s]
Loading 0: 63%|██████▎ | 230/363 [00:29<00:15, 8.54it/s]
Loading 0: 64%|██████▍ | 233/363 [00:29<00:14, 8.68it/s]
Loading 0: 65%|██████▌ | 236/363 [00:29<00:12, 10.18it/s]
Loading 0: 66%|██████▌ | 239/363 [00:29<00:12, 9.67it/s]
Loading 0: 67%|██████▋ | 244/363 [00:29<00:08, 14.13it/s]
Loading 0: 68%|██████▊ | 248/363 [00:30<00:06, 17.47it/s]
Loading 0: 69%|██████▉ | 251/363 [00:30<00:08, 13.50it/s]
Loading 0: 70%|██████▉ | 254/363 [00:30<00:07, 14.10it/s]
Loading 0: 71%|███████ | 257/363 [00:31<00:09, 11.44it/s]
Loading 0: 72%|███████▏ | 262/363 [00:31<00:06, 16.15it/s]
Loading 0: 73%|███████▎ | 266/363 [00:31<00:05, 19.36it/s]
Loading 0: 74%|███████▍ | 269/363 [00:31<00:06, 14.20it/s]
Loading 0: 75%|███████▍ | 272/363 [00:31<00:06, 14.93it/s]
Loading 0: 76%|███████▌ | 275/363 [00:32<00:07, 11.91it/s]
Loading 0: 77%|███████▋ | 280/363 [00:32<00:04, 16.81it/s]
Loading 0: 78%|███████▊ | 284/363 [00:32<00:03, 20.08it/s]
Loading 0: 79%|███████▉ | 287/363 [00:32<00:05, 14.98it/s]
Loading 0: 80%|███████▉ | 290/363 [00:32<00:04, 15.54it/s]
Loading 0: 81%|████████ | 293/363 [00:33<00:05, 11.94it/s]
Loading 0: 82%|████████▏ | 298/363 [00:33<00:03, 16.76it/s]
Loading 0: 83%|████████▎ | 302/363 [00:33<00:03, 19.90it/s]
Loading 0: 84%|████████▍ | 305/363 [00:33<00:03, 14.99it/s]
Loading 0: 85%|████████▍ | 308/363 [00:34<00:03, 15.86it/s]
Loading 0: 86%|████████▌ | 311/363 [00:34<00:04, 12.91it/s]
Loading 0: 87%|████████▋ | 316/363 [00:34<00:02, 18.05it/s]
Loading 0: 88%|████████▊ | 320/363 [00:34<00:02, 21.44it/s]
Loading 0: 89%|████████▉ | 323/363 [00:35<00:02, 15.56it/s]
Loading 0: 90%|████████▉ | 326/363 [00:35<00:02, 15.91it/s]
Loading 0: 91%|█████████ | 329/363 [00:35<00:02, 12.28it/s]
Loading 0: 92%|█████████▏| 334/363 [00:35<00:01, 17.19it/s]
Loading 0: 93%|█████████▎| 338/363 [00:35<00:01, 20.50it/s]
Loading 0: 94%|█████████▍| 341/363 [00:36<00:01, 14.84it/s]
Loading 0: 95%|█████████▍| 344/363 [00:36<00:01, 15.42it/s]
Loading 0: 96%|█████████▌| 347/363 [00:36<00:01, 12.00it/s]
Loading 0: 97%|█████████▋| 352/363 [00:36<00:00, 16.68it/s]
Loading 0: 98%|█████████▊| 356/363 [00:36<00:00, 19.93it/s]
Loading 0: 99%|█████████▉| 359/363 [00:37<00:00, 10.49it/s]
Loading 0: 100%|█████████▉| 362/363 [00:37<00:00, 10.59it/s]
Job nischaydnk-exp14-exp9d-94555-v11-mkmlizer completed after 251.09s with status: succeeded
Stopping job with name nischaydnk-exp14-exp9d-94555-v11-mkmlizer
Pipeline stage MKMLizer completed in 251.85s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.22s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service nischaydnk-exp14-exp9d-94555-v11
Waiting for inference service nischaydnk-exp14-exp9d-94555-v11 to be ready
Job rirv938-96p-4ff-nischay-86528-v2-mkmlizer completed after 260.36s with status: succeeded
Stopping job with name rirv938-96p-4ff-nischay-86528-v2-mkmlizer
Pipeline stage MKMLizer completed in 261.22s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.46s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-96p-4ff-nischay-86528-v2
Waiting for inference service rirv938-96p-4ff-nischay-86528-v2 to be ready
Inference service nischaydnk-exp14-exp9d-94555-v11 ready after 100.4291524887085s
Pipeline stage MKMLDeployer completed in 101.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6612558364868164s
Inference service rirv938-96p-4ff-nischay-86528-v2 ready after 100.41326022148132s
Pipeline stage MKMLDeployer completed in 101.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.096081018447876s
Received healthy response to inference request in 2.7414169311523438s
Received healthy response to inference request in 2.217937707901001s
Received healthy response to inference request in 2.0307891368865967s
Received healthy response to inference request in 2.1260604858398438s
Received healthy response to inference request in 2.0948166847229004s
Received healthy response to inference request in 2.290090560913086s
5 requests
0 failed requests
5th percentile: 2.0950695514678954
10th percentile: 2.0953224182128904
20th percentile: 2.095828151702881
30th percentile: 2.1020769119262694
40th percentile: 2.114068698883057
50th percentile: 2.1260604858398438
60th percentile: 2.1628113746643067
70th percentile: 2.1995622634887697
80th percentile: 2.3066013336181643
90th percentile: 2.48392858505249
95th percentile: 2.572592210769653
99th percentile: 2.643523111343384
mean time: 2.2392303466796877
Pipeline stage StressChecker completed in 14.60s
Received healthy response to inference request in 2.2879419326782227s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.10s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Received healthy response to inference request in 2.091495990753174s
5 requests
triggered trigger_guanaco_pipeline args=%s
0 failed requests
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.03s
5th percentile: 2.042930507659912
Shutdown handler de-registered
10th percentile: 2.0550718784332274
nischaydnk-exp14-exp9d_94555_v11 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3124.28s
Shutdown handler de-registered
nischaydnk-exp14-exp9d_94555_v11 status is now inactive due to auto deactivation removed underperforming models
nischaydnk-exp14-exp9d_94555_v11 status is now torndown due to DeploymentManager action