Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-nitral-dpos1-bet-5231-v12-mkmlizer
Waiting for job on chaiml-nitral-dpos1-bet-5231-v12-mkmlizer to finish
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Version: 0.30.2 ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ https://mk1.ai ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ belonging to: ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Chai Research Corp. ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: quantized model in 163.396s
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: Processed model ChaiML/rirv938-0sw-96p-4ff-cha_93234_v2_w0 in 195.499s
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: creating bucket guanaco-mkml-models
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia/config.json
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia/special_tokens_map.json
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia/tokenizer_config.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Downloaded to shared memory in 35.202s
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia/tokenizer.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Checking if ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 already exists in ChaiML
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmplfqbkypk, device:0
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-rirv938-0sw-96p-18713-v10/nvidia/flywheel_model.0.safetensors
chaiml-rirv938-0sw-96p-18713-v10-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:00<00:31, 11.32it/s]
Loading 0: 1%| | 4/363 [00:01<01:57, 3.05it/s]
Loading 0: 1%|▏ | 5/363 [00:02<03:07, 1.91it/s]
Loading 0: 2%|▏ | 6/363 [00:03<04:05, 1.45it/s]
Loading 0: 2%|▏ | 9/363 [00:03<02:11, 2.70it/s]
Loading 0: 3%|▎ | 10/363 [00:03<02:05, 2.80it/s]
Loading 0: 3%|▎ | 11/363 [00:04<01:48, 3.26it/s]
Loading 0: 4%|▎ | 13/363 [00:05<02:13, 2.63it/s]
Loading 0: 4%|▍ | 14/363 [00:06<02:58, 1.95it/s]
Loading 0: 4%|▍ | 15/363 [00:07<03:42, 1.56it/s]
Loading 0: 5%|▍ | 18/363 [00:07<02:08, 2.68it/s]
Loading 0: 5%|▌ | 19/363 [00:07<02:02, 2.80it/s]
Loading 0: 6%|▌ | 20/363 [00:07<01:46, 3.23it/s]
Loading 0: 6%|▌ | 22/363 [00:08<02:10, 2.62it/s]
Loading 0: 6%|▋ | 23/363 [00:09<02:53, 1.96it/s]
Loading 0: 7%|▋ | 24/363 [00:10<03:36, 1.57it/s]
Loading 0: 7%|▋ | 27/363 [00:11<02:06, 2.66it/s]
Loading 0: 8%|▊ | 28/363 [00:11<01:59, 2.80it/s]
Loading 0: 8%|▊ | 29/363 [00:11<01:42, 3.25it/s]
Loading 0: 9%|▊ | 31/363 [00:12<02:05, 2.64it/s]
Loading 0: 9%|▉ | 32/363 [00:13<02:48, 1.97it/s]
Loading 0: 9%|▉ | 33/363 [00:14<03:30, 1.57it/s]
Loading 0: 10%|▉ | 36/363 [00:14<02:02, 2.66it/s]
Loading 0: 10%|█ | 37/363 [00:15<01:56, 2.80it/s]
Loading 0: 10%|█ | 38/363 [00:15<01:40, 3.24it/s]
Loading 0: 11%|█ | 40/363 [00:16<02:02, 2.64it/s]
Loading 0: 11%|█▏ | 41/363 [00:17<02:43, 1.97it/s]
Loading 0: 12%|█▏ | 42/363 [00:18<03:24, 1.57it/s]
Loading 0: 12%|█▏ | 45/363 [00:18<01:59, 2.66it/s]
Loading 0: 13%|█▎ | 46/363 [00:19<01:53, 2.80it/s]
Loading 0: 13%|█▎ | 47/363 [00:19<01:37, 3.23it/s]
Loading 0: 13%|█▎ | 49/363 [00:20<01:59, 2.64it/s]
Loading 0: 14%|█▍ | 50/363 [00:21<02:38, 1.97it/s]
Loading 0: 14%|█▍ | 51/363 [00:22<03:16, 1.58it/s]
Loading 0: 15%|█▍ | 54/363 [00:22<01:55, 2.68it/s]
Loading 0: 15%|█▌ | 55/363 [00:22<01:49, 2.81it/s]
Loading 0: 15%|█▌ | 56/363 [00:22<01:34, 3.26it/s]
Loading 0: 16%|█▌ | 58/363 [00:23<01:55, 2.65it/s]
Loading 0: 16%|█▋ | 59/363 [00:24<02:34, 1.97it/s]
Loading 0: 17%|█▋ | 60/363 [00:25<03:12, 1.57it/s]
Loading 0: 17%|█▋ | 63/363 [00:26<01:52, 2.66it/s]
Loading 0: 18%|█▊ | 64/363 [00:26<01:47, 2.79it/s]
Loading 0: 18%|█▊ | 65/363 [00:26<01:32, 3.23it/s]
Loading 0: 18%|█▊ | 67/363 [00:27<01:52, 2.64it/s]
Loading 0: 19%|█▊ | 68/363 [00:28<02:29, 1.97it/s]
Loading 0: 19%|█▉ | 69/363 [00:29<03:06, 1.58it/s]
Loading 0: 20%|█▉ | 72/363 [00:30<01:49, 2.67it/s]
Loading 0: 20%|██ | 73/363 [00:30<01:43, 2.81it/s]
Loading 0: 20%|██ | 74/363 [00:30<01:28, 3.25it/s]
Loading 0: 21%|██ | 76/363 [00:31<01:48, 2.65it/s]
Loading 0: 21%|██ | 77/363 [00:32<02:24, 1.97it/s]
Loading 0: 21%|██▏ | 78/363 [00:33<03:00, 1.58it/s]
Loading 0: 22%|██▏ | 81/363 [00:33<01:45, 2.66it/s]
Loading 0: 23%|██▎ | 82/363 [00:34<01:40, 2.80it/s]
Loading 0: 23%|██▎ | 83/363 [00:34<01:27, 3.22it/s]
Loading 0: 23%|██▎ | 85/363 [00:35<01:45, 2.63it/s]
Loading 0: 24%|██▎ | 86/363 [00:36<02:21, 1.96it/s]
Loading 0: 24%|██▍ | 87/363 [00:37<02:55, 1.57it/s]
Loading 0: 25%|██▍ | 90/363 [00:37<01:42, 2.66it/s]
Loading 0: 25%|██▌ | 91/363 [00:37<01:37, 2.80it/s]
Loading 0: 25%|██▌ | 92/363 [00:38<01:23, 3.23it/s]
Loading 0: 26%|██▌ | 94/363 [00:39<01:42, 2.64it/s]
Loading 0: 26%|██▌ | 95/363 [00:40<02:15, 1.97it/s]
Loading 0: 26%|██▋ | 96/363 [00:41<02:49, 1.57it/s]
Loading 0: 27%|██▋ | 99/363 [00:41<01:39, 2.66it/s]
Loading 0: 28%|██▊ | 100/363 [00:41<01:33, 2.80it/s]
Loading 0: 28%|██▊ | 101/363 [00:41<01:20, 3.25it/s]
Loading 0: 28%|██▊ | 103/363 [00:42<01:38, 2.65it/s]
Loading 0: 29%|██▊ | 104/363 [00:43<02:11, 1.98it/s]
Loading 0: 29%|██▉ | 105/363 [00:44<02:43, 1.58it/s]
Loading 0: 30%|██▉ | 108/363 [00:45<01:35, 2.67it/s]
Loading 0: 30%|███ | 109/363 [00:45<01:30, 2.80it/s]
Loading 0: 30%|███ | 110/363 [00:45<01:18, 3.21it/s]
Loading 0: 31%|███ | 112/363 [00:46<01:35, 2.62it/s]
Loading 0: 31%|███ | 113/363 [00:47<02:07, 1.97it/s]
Loading 0: 31%|███▏ | 114/363 [00:48<02:39, 1.56it/s]
Loading 0: 32%|███▏ | 117/363 [00:49<01:33, 2.64it/s]
Loading 0: 33%|███▎ | 118/363 [00:49<01:28, 2.78it/s]
Loading 0: 33%|███▎ | 119/363 [00:49<01:15, 3.23it/s]
Loading 0: 33%|███▎ | 121/363 [00:50<01:31, 2.64it/s]
Loading 0: 34%|███▎ | 122/363 [00:51<02:02, 1.97it/s]
Loading 0: 34%|███▍ | 123/363 [00:52<02:31, 1.58it/s]
Loading 0: 35%|███▍ | 126/363 [00:52<01:28, 2.67it/s]
Loading 0: 35%|███▍ | 127/363 [00:53<01:23, 2.81it/s]
Loading 0: 35%|███▌ | 128/363 [00:53<01:12, 3.25it/s]
Loading 0: 36%|███▌ | 130/363 [00:54<01:28, 2.65it/s]
Loading 0: 36%|███▌ | 131/363 [00:55<01:57, 1.98it/s]
Loading 0: 36%|███▋ | 132/363 [00:56<02:41, 1.43it/s]
Loading 0: 37%|███▋ | 135/363 [00:56<01:32, 2.46it/s]
Loading 0: 37%|███▋ | 136/363 [00:57<01:26, 2.62it/s]
Loading 0: 38%|███▊ | 137/363 [00:57<01:14, 3.05it/s]
Loading 0: 38%|███▊ | 139/363 [00:58<01:27, 2.57it/s]
Loading 0: 39%|███▊ | 140/363 [00:59<01:54, 1.94it/s]
Loading 0: 39%|███▉ | 141/363 [01:00<02:22, 1.55it/s]
Loading 0: 40%|███▉ | 144/363 [01:00<01:23, 2.63it/s]
Loading 0: 40%|███▉ | 145/363 [01:00<01:18, 2.77it/s]
Loading 0: 40%|████ | 146/363 [01:01<01:07, 3.21it/s]
Loading 0: 41%|████ | 148/363 [01:02<01:21, 2.63it/s]
Loading 0: 41%|████ | 149/363 [01:02<01:48, 1.97it/s]
Loading 0: 41%|████▏ | 150/363 [01:04<02:15, 1.57it/s]
Loading 0: 42%|████▏ | 153/363 [01:04<01:18, 2.66it/s]
Loading 0: 42%|████▏ | 154/363 [01:04<01:14, 2.80it/s]
Loading 0: 43%|████▎ | 155/363 [01:04<01:04, 3.23it/s]
Loading 0: 43%|████▎ | 157/363 [01:05<01:17, 2.64it/s]
Loading 0: 44%|████▎ | 158/363 [01:06<01:43, 1.98it/s]
Loading 0: 44%|████▍ | 159/363 [01:07<02:09, 1.58it/s]
Loading 0: 45%|████▍ | 162/363 [01:08<01:15, 2.67it/s]
Loading 0: 45%|████▍ | 163/363 [01:08<01:11, 2.81it/s]
Loading 0: 45%|████▌ | 164/363 [01:08<01:01, 3.25it/s]
Loading 0: 46%|████▌ | 166/363 [01:09<01:14, 2.65it/s]
Loading 0: 46%|████▌ | 167/363 [01:10<01:38, 1.98it/s]
Loading 0: 46%|████▋ | 168/363 [01:11<02:02, 1.59it/s]
Loading 0: 47%|████▋ | 171/363 [01:11<01:11, 2.68it/s]
Loading 0: 47%|████▋ | 172/363 [01:12<01:07, 2.82it/s]
Loading 0: 48%|████▊ | 173/363 [01:12<00:58, 3.23it/s]
Loading 0: 48%|████▊ | 175/363 [01:13<01:11, 2.64it/s]
Loading 0: 48%|████▊ | 176/363 [01:14<01:34, 1.98it/s]
Loading 0: 49%|████▉ | 177/363 [01:15<01:57, 1.58it/s]
Loading 0: 50%|████▉ | 180/363 [01:15<01:08, 2.68it/s]
Loading 0: 50%|████▉ | 181/363 [01:15<01:04, 2.81it/s]
Loading 0: 50%|█████ | 182/363 [01:16<00:55, 3.25it/s]
Loading 0: 51%|█████ | 184/363 [01:17<01:07, 2.65it/s]
Loading 0: 51%|█████ | 185/363 [01:18<01:29, 1.98it/s]
Loading 0: 51%|█████ | 186/363 [01:19<01:51, 1.59it/s]
Loading 0: 52%|█████▏ | 189/363 [01:19<01:04, 2.69it/s]
Loading 0: 52%|█████▏ | 190/363 [01:19<01:01, 2.82it/s]
Loading 0: 53%|█████▎ | 191/363 [01:19<00:52, 3.26it/s]
Loading 0: 53%|█████▎ | 193/363 [01:20<01:03, 2.66it/s]
Loading 0: 53%|█████▎ | 194/363 [01:21<01:25, 1.99it/s]
Loading 0: 54%|█████▎ | 195/363 [01:22<01:45, 1.59it/s]
Loading 0: 55%|█████▍ | 198/363 [01:23<01:01, 2.68it/s]
Loading 0: 55%|█████▍ | 199/363 [01:23<00:58, 2.82it/s]
Loading 0: 55%|█████▌ | 200/363 [01:23<00:50, 3.24it/s]
Loading 0: 56%|█████▌ | 202/363 [01:24<01:01, 2.64it/s]
Loading 0: 56%|█████▌ | 203/363 [01:25<01:20, 1.98it/s]
Loading 0: 56%|█████▌ | 204/363 [01:26<01:40, 1.58it/s]
Loading 0: 57%|█████▋ | 207/363 [01:26<00:58, 2.68it/s]
Loading 0: 57%|█████▋ | 208/363 [01:27<00:55, 2.81it/s]
Loading 0: 58%|█████▊ | 209/363 [01:27<00:47, 3.26it/s]
Loading 0: 58%|█████▊ | 211/363 [01:28<00:57, 2.65it/s]
Loading 0: 58%|█████▊ | 212/363 [01:29<01:16, 1.98it/s]
Loading 0: 59%|█████▊ | 213/363 [01:30<01:34, 1.58it/s]
Loading 0: 60%|█████▉ | 216/363 [01:30<00:54, 2.68it/s]
Loading 0: 60%|█████▉ | 217/363 [01:30<00:51, 2.81it/s]
Loading 0: 60%|██████ | 218/363 [01:31<00:44, 3.25it/s]
Loading 0: 61%|██████ | 220/363 [01:32<00:53, 2.66it/s]
Loading 0: 61%|██████ | 221/363 [01:33<01:11, 1.99it/s]
Loading 0: 61%|██████ | 222/363 [01:34<01:28, 1.59it/s]
Loading 0: 62%|██████▏ | 225/363 [01:34<00:51, 2.69it/s]
Loading 0: 62%|██████▏ | 226/363 [01:34<00:48, 2.82it/s]
Loading 0: 63%|██████▎ | 227/363 [01:34<00:41, 3.26it/s]
Loading 0: 63%|██████▎ | 229/363 [01:35<00:50, 2.66it/s]
Loading 0: 63%|██████▎ | 230/363 [01:36<01:07, 1.98it/s]
Loading 0: 64%|██████▎ | 231/363 [01:37<01:23, 1.58it/s]
Loading 0: 64%|██████▍ | 234/363 [01:38<00:48, 2.67it/s]
Loading 0: 65%|██████▍ | 235/363 [01:38<00:45, 2.81it/s]
Loading 0: 65%|██████▌ | 236/363 [01:38<00:39, 3.25it/s]
Loading 0: 66%|██████▌ | 238/363 [01:39<00:47, 2.65it/s]
Loading 0: 66%|██████▌ | 239/363 [01:40<01:02, 1.97it/s]
Loading 0: 66%|██████▌ | 240/363 [01:41<01:17, 1.58it/s]
Loading 0: 67%|██████▋ | 243/363 [01:41<00:44, 2.67it/s]
Loading 0: 67%|██████▋ | 244/363 [01:42<00:42, 2.81it/s]
Loading 0: 67%|██████▋ | 245/363 [01:42<00:36, 3.25it/s]
Loading 0: 68%|██████▊ | 247/363 [01:43<00:43, 2.65it/s]
Loading 0: 68%|██████▊ | 248/363 [01:44<00:58, 1.98it/s]
Loading 0: 69%|██████▊ | 249/363 [01:45<01:11, 1.59it/s]
Loading 0: 69%|██████▉ | 252/363 [01:45<00:41, 2.69it/s]
Loading 0: 70%|██████▉ | 253/363 [01:46<00:38, 2.82it/s]
Loading 0: 70%|██████▉ | 254/363 [01:46<00:33, 3.25it/s]
Loading 0: 71%|███████ | 256/363 [01:47<00:40, 2.65it/s]
Loading 0: 71%|███████ | 257/363 [01:48<00:53, 1.98it/s]
Loading 0: 71%|███████ | 258/363 [01:49<01:06, 1.57it/s]
Loading 0: 72%|███████▏ | 261/363 [01:49<00:38, 2.66it/s]
Loading 0: 72%|███████▏ | 262/363 [01:49<00:36, 2.80it/s]
Loading 0: 72%|███████▏ | 263/363 [01:49<00:30, 3.24it/s]
Loading 0: 73%|███████▎ | 265/363 [01:50<00:37, 2.64it/s]
Loading 0: 73%|███████▎ | 266/363 [01:51<00:49, 1.97it/s]
Loading 0: 74%|███████▎ | 267/363 [01:52<01:00, 1.58it/s]
Loading 0: 74%|███████▍ | 270/363 [01:53<00:34, 2.68it/s]
Loading 0: 75%|███████▍ | 271/363 [01:53<00:32, 2.81it/s]
Loading 0: 75%|███████▍ | 272/363 [01:53<00:27, 3.25it/s]
Loading 0: 75%|███████▌ | 274/363 [01:54<00:33, 2.65it/s]
Loading 0: 76%|███████▌ | 275/363 [01:55<00:44, 1.98it/s]
Loading 0: 76%|███████▌ | 276/363 [01:56<00:55, 1.58it/s]
Loading 0: 77%|███████▋ | 279/363 [01:57<00:31, 2.67it/s]
Loading 0: 77%|███████▋ | 280/363 [01:57<00:29, 2.81it/s]
Loading 0: 77%|███████▋ | 281/363 [01:57<00:25, 3.24it/s]
Loading 0: 78%|███████▊ | 283/363 [01:58<00:30, 2.64it/s]
Loading 0: 78%|███████▊ | 284/363 [01:59<00:40, 1.97it/s]
Loading 0: 79%|███████▊ | 285/363 [02:00<00:49, 1.58it/s]
Loading 0: 79%|███████▉ | 288/363 [02:00<00:28, 2.66it/s]
Loading 0: 80%|███████▉ | 289/363 [02:01<00:26, 2.80it/s]
Loading 0: 80%|███████▉ | 290/363 [02:01<00:22, 3.21it/s]
Loading 0: 80%|████████ | 292/363 [02:02<00:27, 2.62it/s]
Loading 0: 81%|████████ | 293/363 [02:03<00:35, 1.97it/s]
Loading 0: 81%|████████ | 294/363 [02:04<00:43, 1.58it/s]
Loading 0: 82%|████████▏ | 297/363 [02:04<00:24, 2.67it/s]
Loading 0: 82%|████████▏ | 298/363 [02:04<00:23, 2.80it/s]
Loading 0: 82%|████████▏ | 299/363 [02:05<00:19, 3.24it/s]
Loading 0: 83%|████████▎ | 301/363 [02:06<00:23, 2.61it/s]
Loading 0: 83%|████████▎ | 302/363 [02:07<00:31, 1.96it/s]
Loading 0: 83%|████████▎ | 303/363 [02:08<00:38, 1.57it/s]
Loading 0: 84%|████████▍ | 306/363 [02:08<00:21, 2.66it/s]
Loading 0: 85%|████████▍ | 307/363 [02:08<00:20, 2.79it/s]
Loading 0: 85%|████████▍ | 308/363 [02:08<00:17, 3.21it/s]
Loading 0: 85%|████████▌ | 310/363 [02:10<00:22, 2.31it/s]
Loading 0: 86%|████████▌ | 311/363 [02:11<00:28, 1.82it/s]
Loading 0: 86%|████████▌ | 312/363 [02:12<00:33, 1.51it/s]
Loading 0: 87%|████████▋ | 315/363 [02:12<00:18, 2.57it/s]
Loading 0: 87%|████████▋ | 316/363 [02:12<00:17, 2.72it/s]
Loading 0: 87%|████████▋ | 317/363 [02:12<00:14, 3.16it/s]
Loading 0: 88%|████████▊ | 319/363 [02:13<00:16, 2.60it/s]
Loading 0: 88%|████████▊ | 320/363 [02:14<00:21, 1.95it/s]
Loading 0: 88%|████████▊ | 321/363 [02:15<00:26, 1.56it/s]
Loading 0: 89%|████████▉ | 324/363 [02:16<00:14, 2.64it/s]
Loading 0: 90%|████████▉ | 325/363 [02:16<00:13, 2.78it/s]
Loading 0: 90%|████████▉ | 326/363 [02:16<00:11, 3.22it/s]
Loading 0: 90%|█████████ | 328/363 [02:17<00:13, 2.63it/s]
Loading 0: 91%|█████████ | 329/363 [02:18<00:17, 1.97it/s]
Loading 0: 91%|█████████ | 330/363 [02:19<00:20, 1.58it/s]
Loading 0: 92%|█████████▏| 333/363 [02:20<00:11, 2.67it/s]
Loading 0: 92%|█████████▏| 334/363 [02:20<00:10, 2.81it/s]
Loading 0: 92%|█████████▏| 335/363 [02:20<00:08, 3.24it/s]
Loading 0: 93%|█████████▎| 337/363 [02:21<00:09, 2.64it/s]
Loading 0: 93%|█████████▎| 338/363 [02:22<00:12, 1.97it/s]
Loading 0: 93%|█████████▎| 339/363 [02:23<00:15, 1.57it/s]
Loading 0: 94%|█████████▍| 342/363 [02:23<00:07, 2.67it/s]
Loading 0: 94%|█████████▍| 343/363 [02:24<00:07, 2.80it/s]
Loading 0: 95%|█████████▍| 344/363 [02:24<00:05, 3.24it/s]
Loading 0: 95%|█████████▌| 346/363 [02:25<00:06, 2.64it/s]
Loading 0: 96%|█████████▌| 347/363 [02:26<00:08, 1.97it/s]
Loading 0: 96%|█████████▌| 348/363 [02:27<00:09, 1.58it/s]
Loading 0: 97%|█████████▋| 351/363 [02:27<00:04, 2.67it/s]
Loading 0: 97%|█████████▋| 352/363 [02:27<00:03, 2.81it/s]
Loading 0: 97%|█████████▋| 353/363 [02:28<00:03, 3.25it/s]
Loading 0: 98%|█████████▊| 355/363 [02:28<00:03, 2.64it/s]
Loading 0: 98%|█████████▊| 356/363 [02:29<00:03, 1.97it/s]
Loading 0: 98%|█████████▊| 357/363 [02:31<00:03, 1.58it/s]
Loading 0: 99%|█████████▉| 360/363 [02:31<00:01, 2.67it/s]
Loading 0: 99%|█████████▉| 361/363 [02:31<00:00, 2.81it/s]
Loading 0: 100%|█████████▉| 362/363 [02:31<00:00, 3.25it/s]
Job chaiml-rirv938-0sw-96p-18713-v10-mkmlizer completed after 229.24s with status: succeeded
Stopping job with name chaiml-rirv938-0sw-96p-18713-v10-mkmlizer
Pipeline stage MKMLizer completed in 230.85s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.38s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-rirv938-0sw-96p-18713-v10
Waiting for inference service chaiml-rirv938-0sw-96p-18713-v10 to be ready
Inference service chaiml-rirv938-0sw-96p-18713-v10 ready after 50.93461275100708s
Pipeline stage MKMLDeployer completed in 52.69s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.8096914291381836s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.9096276760101318s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.9118058681488037s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.940983772277832s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.272380828857422s
5 requests
0 failed requests
5th percentile: 1.9100633144378663
10th percentile: 1.9104989528656007
20th percentile: 1.9113702297210693
30th percentile: 1.9176414489746094
40th percentile: 1.9293126106262206
50th percentile: 1.940983772277832
60th percentile: 2.073542594909668
70th percentile: 2.206101417541504
80th percentile: 2.3798429489135744
90th percentile: 2.594767189025879
95th percentile: 2.702229309082031
99th percentile: 2.788199005126953
mean time: 2.1688979148864744
Pipeline stage StressChecker completed in 15.90s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.40s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.43s
Shutdown handler de-registered
chaiml-rirv938-0sw-96p_18713_v10 status is now deployed due to DeploymentManager action
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: quantized model in 148.421s
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Processed model ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 in 183.623s
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/nvidia
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/nvidia/flywheel_model.0.safetensors
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 3/363 [00:00<01:52, 3.19it/s]
Loading 0: 1%| | 4/363 [00:01<02:59, 2.00it/s]
Loading 0: 1%|▏ | 5/363 [00:02<03:51, 1.54it/s]
Loading 0: 2%|▏ | 8/363 [00:03<02:00, 2.96it/s]
Loading 0: 2%|▏ | 9/363 [00:03<01:55, 3.06it/s]
Loading 0: 3%|▎ | 10/363 [00:03<01:39, 3.56it/s]
Loading 0: 3%|▎ | 12/363 [00:04<02:00, 2.90it/s]
Loading 0: 4%|▎ | 13/363 [00:05<02:41, 2.16it/s]
Loading 0: 4%|▍ | 14/363 [00:06<03:23, 1.71it/s]
Loading 0: 5%|▍ | 17/363 [00:06<01:57, 2.94it/s]
Loading 0: 5%|▍ | 18/363 [00:06<01:51, 3.09it/s]
Loading 0: 5%|▌ | 19/363 [00:06<01:35, 3.59it/s]
Loading 0: 6%|▌ | 21/363 [00:07<01:56, 2.94it/s]
Loading 0: 6%|▌ | 22/363 [00:08<02:35, 2.20it/s]
Loading 0: 6%|▋ | 23/363 [00:09<03:14, 1.75it/s]
Loading 0: 7%|▋ | 26/363 [00:09<01:53, 2.97it/s]
Loading 0: 7%|▋ | 27/363 [00:10<01:47, 3.12it/s]
Loading 0: 8%|▊ | 28/363 [00:10<01:34, 3.56it/s]
Loading 0: 8%|▊ | 30/363 [00:11<01:53, 2.93it/s]
Loading 0: 9%|▊ | 31/363 [00:12<02:31, 2.19it/s]
Loading 0: 9%|▉ | 32/363 [00:13<03:09, 1.74it/s]
Loading 0: 10%|▉ | 35/363 [00:13<01:50, 2.96it/s]
Loading 0: 10%|▉ | 36/363 [00:13<01:45, 3.11it/s]
Loading 0: 10%|█ | 37/363 [00:13<01:30, 3.59it/s]
Loading 0: 11%|█ | 39/363 [00:14<01:50, 2.94it/s]
Loading 0: 11%|█ | 40/363 [00:15<02:26, 2.20it/s]
Loading 0: 11%|█▏ | 41/363 [00:16<03:03, 1.76it/s]
Loading 0: 12%|█▏ | 44/363 [00:16<01:47, 2.98it/s]
Loading 0: 12%|█▏ | 45/363 [00:17<01:41, 3.13it/s]
Loading 0: 13%|█▎ | 46/363 [00:17<01:28, 3.57it/s]
Loading 0: 13%|█▎ | 48/363 [00:18<01:48, 2.91it/s]
Loading 0: 13%|█▎ | 49/363 [00:18<02:24, 2.18it/s]
Loading 0: 14%|█▍ | 50/363 [00:19<03:00, 1.74it/s]
Loading 0: 15%|█▍ | 53/363 [00:20<01:45, 2.94it/s]
Loading 0: 15%|█▍ | 54/363 [00:20<01:39, 3.09it/s]
Loading 0: 15%|█▌ | 55/363 [00:20<01:26, 3.54it/s]
Loading 0: 15%|█▌ | 56/363 [00:21<02:09, 2.36it/s]
Loading 0: 16%|█▌ | 58/363 [00:21<01:36, 3.18it/s]
Loading 0: 16%|█▋ | 59/363 [00:22<01:31, 3.32it/s]
Loading 0: 17%|█▋ | 60/363 [00:22<01:17, 3.89it/s]
Loading 0: 17%|█▋ | 61/363 [00:22<01:08, 4.38it/s]
Loading 0: 17%|█▋ | 62/363 [00:23<02:01, 2.47it/s]
Loading 0: 17%|█▋ | 63/363 [00:24<02:43, 1.83it/s]
Loading 0: 18%|█▊ | 64/363 [00:25<03:19, 1.50it/s]
Loading 0: 18%|█▊ | 67/363 [00:25<01:45, 2.79it/s]
Loading 0: 19%|█▊ | 68/363 [00:25<01:39, 2.97it/s]
Loading 0: 19%|█▉ | 69/363 [00:25<01:24, 3.47it/s]
Loading 0: 20%|█▉ | 71/363 [00:26<01:41, 2.86it/s]
Loading 0: 20%|█▉ | 72/363 [00:27<02:15, 2.15it/s]
Loading 0: 20%|██ | 73/363 [00:28<02:48, 1.72it/s]
Loading 0: 21%|██ | 76/363 [00:28<01:37, 2.94it/s]
Loading 0: 21%|██ | 77/363 [00:29<01:32, 3.10it/s]
Loading 0: 21%|██▏ | 78/363 [00:29<01:19, 3.58it/s]
Loading 0: 22%|██▏ | 80/363 [00:30<01:36, 2.93it/s]
Loading 0: 22%|██▏ | 81/363 [00:30<02:09, 2.19it/s]
Loading 0: 23%|██▎ | 82/363 [00:31<02:41, 1.74it/s]
Loading 0: 23%|██▎ | 85/363 [00:32<01:34, 2.95it/s]
Loading 0: 24%|██▎ | 86/363 [00:32<01:29, 3.10it/s]
Loading 0: 24%|██▍ | 87/363 [00:32<01:16, 3.59it/s]
Loading 0: 25%|██▍ | 89/363 [00:33<01:33, 2.94it/s]
Loading 0: 25%|██▍ | 90/363 [00:34<02:04, 2.19it/s]
Loading 0: 25%|██▌ | 91/363 [00:35<02:34, 1.76it/s]
Loading 0: 26%|██▌ | 94/363 [00:35<01:30, 2.97it/s]
Loading 0: 26%|██▌ | 95/363 [00:35<01:25, 3.12it/s]
Loading 0: 26%|██▋ | 96/363 [00:35<01:14, 3.58it/s]
Loading 0: 27%|██▋ | 98/363 [00:36<01:30, 2.93it/s]
Loading 0: 27%|██▋ | 99/363 [00:37<02:00, 2.19it/s]
Loading 0: 28%|██▊ | 100/363 [00:38<02:30, 1.75it/s]
Loading 0: 28%|██▊ | 103/363 [00:38<01:27, 2.96it/s]
Loading 0: 29%|██▊ | 104/363 [00:39<01:23, 3.11it/s]
Loading 0: 29%|██▉ | 105/363 [00:39<01:11, 3.59it/s]
Loading 0: 29%|██▉ | 106/363 [00:40<01:48, 2.38it/s]
Loading 0: 30%|██▉ | 108/363 [00:40<01:20, 3.18it/s]
Loading 0: 30%|███ | 109/363 [00:40<01:16, 3.33it/s]
Loading 0: 30%|███ | 110/363 [00:40<01:05, 3.89it/s]
Loading 0: 31%|███ | 112/363 [00:41<01:23, 3.00it/s]
Loading 0: 31%|███ | 113/363 [00:42<01:58, 2.11it/s]
Loading 0: 32%|███▏ | 116/363 [00:43<01:35, 2.59it/s]
Loading 0: 32%|███▏ | 117/363 [00:44<01:57, 2.09it/s]
Loading 0: 33%|███▎ | 118/363 [00:45<02:21, 1.73it/s]
Loading 0: 33%|███▎ | 121/363 [00:45<01:25, 2.84it/s]
Loading 0: 34%|███▎ | 122/363 [00:46<01:20, 3.00it/s]
Loading 0: 34%|███▍ | 123/363 [00:46<01:09, 3.46it/s]
Loading 0: 34%|███▍ | 125/363 [00:47<01:22, 2.89it/s]
Loading 0: 35%|███▍ | 126/363 [00:47<01:48, 2.18it/s]
Loading 0: 35%|███▍ | 127/363 [00:48<02:14, 1.75it/s]
Loading 0: 36%|███▌ | 130/363 [00:49<01:19, 2.95it/s]
Loading 0: 36%|███▌ | 131/363 [00:49<01:14, 3.10it/s]
Loading 0: 36%|███▋ | 132/363 [00:49<01:04, 3.58it/s]
Loading 0: 37%|███▋ | 134/363 [00:50<01:18, 2.92it/s]
Loading 0: 37%|███▋ | 135/363 [00:51<01:44, 2.19it/s]
Loading 0: 37%|███▋ | 136/363 [00:52<02:09, 1.75it/s]
Loading 0: 38%|███▊ | 139/363 [00:52<01:15, 2.96it/s]
Loading 0: 39%|███▊ | 140/363 [00:52<01:11, 3.11it/s]
Loading 0: 39%|███▉ | 141/363 [00:52<01:02, 3.58it/s]
Loading 0: 39%|███▉ | 142/363 [00:53<00:54, 4.06it/s]
Loading 0: 39%|███▉ | 143/363 [00:53<01:29, 2.46it/s]
Loading 0: 40%|███▉ | 144/363 [00:54<02:00, 1.82it/s]
Loading 0: 40%|████ | 147/363 [00:55<01:29, 2.41it/s]
Loading 0: 41%|████ | 148/363 [00:56<01:49, 1.96it/s]
Loading 0: 41%|████ | 149/363 [00:57<02:09, 1.65it/s]
Loading 0: 42%|████▏ | 152/363 [00:57<01:16, 2.77it/s]
Loading 0: 42%|████▏ | 153/363 [00:58<01:11, 2.94it/s]
Loading 0: 42%|████▏ | 154/363 [00:58<01:01, 3.40it/s]
Loading 0: 43%|████▎ | 156/363 [00:59<01:12, 2.85it/s]
Loading 0: 43%|████▎ | 157/363 [01:00<01:35, 2.16it/s]
Loading 0: 44%|████▎ | 158/363 [01:01<01:58, 1.73it/s]
Loading 0: 44%|████▍ | 161/363 [01:01<01:08, 2.93it/s]
Loading 0: 45%|████▍ | 162/363 [01:01<01:05, 3.09it/s]
Loading 0: 45%|████▍ | 163/363 [01:01<00:56, 3.56it/s]
Loading 0: 45%|████▌ | 165/363 [01:02<01:07, 2.92it/s]
Loading 0: 46%|████▌ | 166/363 [01:03<01:30, 2.18it/s]
Loading 0: 46%|████▌ | 167/363 [01:04<01:51, 1.75it/s]
Loading 0: 47%|████▋ | 170/363 [01:04<01:05, 2.96it/s]
Loading 0: 47%|████▋ | 171/363 [01:05<01:01, 3.12it/s]
Loading 0: 47%|████▋ | 172/363 [01:05<00:53, 3.60it/s]
Loading 0: 48%|████▊ | 174/363 [01:06<01:04, 2.94it/s]
Loading 0: 48%|████▊ | 175/363 [01:06<01:25, 2.20it/s]
Loading 0: 48%|████▊ | 176/363 [01:07<01:46, 1.75it/s]
Loading 0: 49%|████▉ | 179/363 [01:08<01:02, 2.96it/s]
Loading 0: 50%|████▉ | 180/363 [01:08<00:58, 3.12it/s]
Loading 0: 50%|████▉ | 181/363 [01:08<00:50, 3.60it/s]
Loading 0: 50%|█████ | 183/363 [01:09<01:01, 2.94it/s]
Loading 0: 51%|█████ | 184/363 [01:10<01:21, 2.20it/s]
Loading 0: 51%|█████ | 185/363 [01:11<01:40, 1.76it/s]
Loading 0: 52%|█████▏ | 188/363 [01:11<00:58, 2.98it/s]
Loading 0: 52%|█████▏ | 189/363 [01:11<00:55, 3.13it/s]
Loading 0: 52%|█████▏ | 190/363 [01:11<00:47, 3.61it/s]
Loading 0: 53%|█████▎ | 192/363 [01:12<00:57, 2.95it/s]
Loading 0: 53%|█████▎ | 193/363 [01:13<01:17, 2.20it/s]
Loading 0: 53%|█████▎ | 194/363 [01:14<01:36, 1.76it/s]
Loading 0: 54%|█████▍ | 197/363 [01:14<00:55, 2.98it/s]
Loading 0: 55%|█████▍ | 198/363 [01:15<00:52, 3.13it/s]
Loading 0: 55%|█████▍ | 199/363 [01:15<00:45, 3.58it/s]
Loading 0: 55%|█████▌ | 201/363 [01:16<00:55, 2.93it/s]
Loading 0: 56%|█████▌ | 202/363 [01:17<01:13, 2.19it/s]
Loading 0: 56%|█████▌ | 203/363 [01:18<01:30, 1.76it/s]
Loading 0: 57%|█████▋ | 206/363 [01:18<00:52, 2.98it/s]
Loading 0: 57%|█████▋ | 207/363 [01:18<00:49, 3.13it/s]
Loading 0: 57%|█████▋ | 208/363 [01:18<00:43, 3.60it/s]
Loading 0: 58%|█████▊ | 210/363 [01:19<00:51, 2.94it/s]
Loading 0: 58%|█████▊ | 211/363 [01:20<01:09, 2.20it/s]
Loading 0: 58%|█████▊ | 212/363 [01:21<01:25, 1.76it/s]
Loading 0: 59%|█████▉ | 215/363 [01:21<00:49, 2.98it/s]
Loading 0: 60%|█████▉ | 216/363 [01:21<00:46, 3.13it/s]
Loading 0: 60%|█████▉ | 217/363 [01:22<00:40, 3.62it/s]
Loading 0: 60%|██████ | 218/363 [01:22<01:00, 2.39it/s]
Loading 0: 61%|██████ | 220/363 [01:23<00:44, 3.19it/s]
Loading 0: 61%|██████ | 221/363 [01:23<00:42, 3.34it/s]
Loading 0: 61%|██████ | 222/363 [01:23<00:36, 3.85it/s]
Loading 0: 61%|██████▏ | 223/363 [01:23<00:31, 4.38it/s]
Loading 0: 62%|██████▏ | 224/363 [01:24<00:55, 2.48it/s]
Loading 0: 62%|██████▏ | 225/363 [01:25<01:16, 1.81it/s]
Loading 0: 63%|██████▎ | 228/363 [01:26<00:55, 2.43it/s]
Loading 0: 63%|██████▎ | 229/363 [01:27<01:07, 1.97it/s]
Loading 0: 63%|██████▎ | 230/363 [01:28<01:20, 1.65it/s]
Loading 0: 64%|██████▍ | 233/363 [01:28<00:46, 2.79it/s]
Loading 0: 64%|██████▍ | 234/363 [01:28<00:43, 2.96it/s]
Loading 0: 65%|██████▍ | 235/363 [01:29<00:37, 3.42it/s]
Loading 0: 65%|██████▌ | 237/363 [01:29<00:43, 2.88it/s]
Loading 0: 66%|██████▌ | 238/363 [01:30<00:57, 2.17it/s]
Loading 0: 66%|██████▌ | 239/363 [01:31<01:11, 1.74it/s]
Loading 0: 67%|██████▋ | 242/363 [01:32<00:41, 2.94it/s]
Loading 0: 67%|██████▋ | 243/363 [01:32<00:38, 3.10it/s]
Loading 0: 67%|██████▋ | 244/363 [01:32<00:33, 3.55it/s]
Loading 0: 68%|██████▊ | 246/363 [01:33<00:39, 2.93it/s]
Loading 0: 68%|██████▊ | 247/363 [01:34<00:52, 2.20it/s]
Loading 0: 68%|██████▊ | 248/363 [01:35<01:05, 1.76it/s]
Loading 0: 69%|██████▉ | 251/363 [01:35<00:37, 2.98it/s]
Loading 0: 69%|██████▉ | 252/363 [01:35<00:35, 3.14it/s]
Loading 0: 70%|██████▉ | 253/363 [01:35<00:30, 3.61it/s]
Loading 0: 70%|███████ | 255/363 [01:36<00:36, 2.96it/s]
Loading 0: 71%|███████ | 256/363 [01:37<00:48, 2.21it/s]
Loading 0: 71%|███████ | 257/363 [01:38<01:00, 1.77it/s]
Loading 0: 72%|███████▏ | 260/363 [01:38<00:34, 2.99it/s]
Loading 0: 72%|███████▏ | 261/363 [01:39<00:32, 3.13it/s]
Loading 0: 72%|███████▏ | 262/363 [01:39<00:28, 3.58it/s]
Loading 0: 73%|███████▎ | 264/363 [01:40<00:33, 2.94it/s]
Loading 0: 73%|███████▎ | 265/363 [01:40<00:44, 2.20it/s]
Loading 0: 73%|███████▎ | 266/363 [01:41<00:55, 1.75it/s]
Loading 0: 74%|███████▍ | 269/363 [01:42<00:31, 2.97it/s]
Loading 0: 74%|███████▍ | 270/363 [01:42<00:29, 3.12it/s]
Loading 0: 75%|███████▍ | 271/363 [01:42<00:25, 3.60it/s]
Loading 0: 75%|███████▌ | 273/363 [01:43<00:30, 2.95it/s]
Loading 0: 75%|███████▌ | 274/363 [01:44<00:40, 2.21it/s]
Loading 0: 76%|███████▌ | 275/363 [01:45<00:49, 1.76it/s]
Loading 0: 77%|███████▋ | 278/363 [01:45<00:28, 2.98it/s]
Loading 0: 77%|███████▋ | 279/363 [01:45<00:26, 3.13it/s]
Loading 0: 77%|███████▋ | 280/363 [01:45<00:23, 3.59it/s]
Loading 0: 78%|███████▊ | 282/363 [01:46<00:27, 2.94it/s]
Loading 0: 78%|███████▊ | 283/363 [01:47<00:36, 2.20it/s]
Loading 0: 78%|███████▊ | 284/363 [01:48<00:44, 1.76it/s]
Loading 0: 79%|███████▉ | 287/363 [01:48<00:25, 2.99it/s]
Loading 0: 79%|███████▉ | 288/363 [01:49<00:23, 3.14it/s]
Loading 0: 80%|███████▉ | 289/363 [01:49<00:20, 3.61it/s]
Loading 0: 80%|████████ | 291/363 [01:50<00:24, 2.95it/s]
Loading 0: 80%|████████ | 292/363 [01:51<00:32, 2.21it/s]
Loading 0: 81%|████████ | 293/363 [01:52<00:39, 1.76it/s]
Loading 0: 82%|████████▏ | 296/363 [01:52<00:22, 2.98it/s]
Loading 0: 82%|████████▏ | 297/363 [01:52<00:21, 3.13it/s]
Loading 0: 82%|████████▏ | 298/363 [01:52<00:18, 3.59it/s]
Loading 0: 82%|████████▏ | 299/363 [01:53<00:26, 2.38it/s]
Loading 0: 83%|████████▎ | 301/363 [01:53<00:19, 3.18it/s]
Loading 0: 83%|████████▎ | 302/363 [01:54<00:18, 3.33it/s]
Loading 0: 83%|████████▎ | 303/363 [01:54<00:15, 3.84it/s]
Loading 0: 84%|████████▎ | 304/363 [01:54<00:16, 3.67it/s]
Loading 0: 84%|████████▍ | 306/363 [01:55<00:19, 2.91it/s]
Loading 0: 85%|████████▍ | 307/363 [01:56<00:27, 2.07it/s]
Loading 0: 85%|████████▌ | 310/363 [01:57<00:20, 2.58it/s]
Loading 0: 86%|████████▌ | 311/363 [01:58<00:25, 2.08it/s]
Loading 0: 86%|████████▌ | 312/363 [01:59<00:29, 1.73it/s]
Loading 0: 87%|████████▋ | 315/363 [01:59<00:16, 2.85it/s]
Loading 0: 87%|████████▋ | 316/363 [01:59<00:15, 3.00it/s]
Loading 0: 87%|████████▋ | 317/363 [01:59<00:13, 3.44it/s]
Loading 0: 88%|████████▊ | 319/363 [02:00<00:15, 2.89it/s]
Loading 0: 88%|████████▊ | 320/363 [02:01<00:19, 2.19it/s]
Loading 0: 88%|████████▊ | 321/363 [02:02<00:23, 1.76it/s]
Loading 0: 89%|████████▉ | 324/363 [02:02<00:13, 2.97it/s]
Loading 0: 90%|████████▉ | 325/363 [02:03<00:12, 3.13it/s]
Loading 0: 90%|████████▉ | 326/363 [02:03<00:10, 3.61it/s]
Loading 0: 90%|█████████ | 328/363 [02:04<00:11, 2.96it/s]
Loading 0: 91%|█████████ | 329/363 [02:04<00:15, 2.21it/s]
Loading 0: 91%|█████████ | 330/363 [02:05<00:18, 1.76it/s]
Loading 0: 92%|█████████▏| 333/363 [02:06<00:10, 2.98it/s]
Loading 0: 92%|█████████▏| 334/363 [02:06<00:09, 3.13it/s]
Loading 0: 92%|█████████▏| 335/363 [02:06<00:07, 3.62it/s]
Loading 0: 93%|█████████▎| 337/363 [02:07<00:08, 2.95it/s]
Loading 0: 93%|█████████▎| 338/363 [02:08<00:11, 2.21it/s]
Loading 0: 93%|█████████▎| 339/363 [02:09<00:13, 1.77it/s]
Loading 0: 94%|█████████▍| 342/363 [02:09<00:07, 2.99it/s]
Loading 0: 94%|█████████▍| 343/363 [02:09<00:06, 3.14it/s]
Loading 0: 95%|█████████▍| 344/363 [02:09<00:05, 3.62it/s]
Loading 0: 95%|█████████▌| 346/363 [02:10<00:05, 2.95it/s]
Loading 0: 96%|█████████▌| 347/363 [02:11<00:07, 2.20it/s]
Loading 0: 96%|█████████▌| 348/363 [02:12<00:08, 1.76it/s]
Loading 0: 97%|█████████▋| 351/363 [02:12<00:04, 2.97it/s]
Loading 0: 97%|█████████▋| 352/363 [02:13<00:03, 3.12it/s]
Loading 0: 97%|█████████▋| 353/363 [02:13<00:02, 3.61it/s]
Loading 0: 98%|█████████▊| 355/363 [02:14<00:02, 2.92it/s]
Loading 0: 98%|█████████▊| 356/363 [02:15<00:03, 2.19it/s]
Loading 0: 98%|█████████▊| 357/363 [02:16<00:03, 1.76it/s]
Loading 0: 99%|█████████▉| 360/363 [02:16<00:01, 2.97it/s]
Loading 0: 99%|█████████▉| 361/363 [02:16<00:00, 3.13it/s]
Loading 0: 100%|█████████▉| 362/363 [02:16<00:00, 3.56it/s]
Job chaiml-nitral-dpos1-bet-5231-v12-mkmlizer completed after 209.86s with status: succeeded
Stopping job with name chaiml-nitral-dpos1-bet-5231-v12-mkmlizer
Pipeline stage MKMLizer completed in 211.25s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.29s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-nitral-dpos1-bet-5231-v12
Waiting for inference service chaiml-nitral-dpos1-bet-5231-v12 to be ready
Inference service chaiml-nitral-dpos1-bet-5231-v12 ready after 40.80884838104248s
Pipeline stage MKMLDeployer completed in 41.97s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.6090946197509766s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.4168453216552734s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.887711763381958s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.116246223449707s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.9218113422393799s
5 requests
0 failed requests
5th percentile: 1.8945316791534423
10th percentile: 1.9013515949249267
20th percentile: 1.9149914264678956
30th percentile: 1.9606983184814453
40th percentile: 2.0384722709655763
50th percentile: 2.116246223449707
60th percentile: 2.2364858627319335
70th percentile: 2.35672550201416
80th percentile: 2.455295181274414
90th percentile: 2.532194900512695
95th percentile: 2.570644760131836
99th percentile: 2.6014046478271484
mean time: 2.190341854095459
Pipeline stage StressChecker completed in 13.50s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.30s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.27s
Shutdown handler de-registered
chaiml-nitral-dpos1-bet_5231_v12 status is now deployed due to DeploymentManager action
chaiml-nitral-dpos1-bet_5231_v12 status is now inactive due to auto deactivation removed underperforming models
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage AMDMKMLizer
Starting job with name chaiml-nitral-dpos1-bet-5231-v12-mkmlizer
Waiting for job on chaiml-nitral-dpos1-bet-5231-v12-mkmlizer to finish
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: bash: no job control in this shell
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Version: 0.29.16+gfx942 ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Features: FLYWHEEL, CUDA, ROCM ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ https://mk1.ai ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ belonging to: ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Chai Research Corp. ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ Expiration: 2028-03-31 00:00:00 ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ║ ║
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Downloaded to shared memory in 58.203s
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Checking if ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 already exists in ChaiML
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmp9zitp7l5, device:0
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer:
Loading 0: 0%| | 0.00/363 [00:00<?, ?it/s]
Loading 0: 5%|▌ | 19.0/363 [00:01<00:21, 15.9it/s]
Loading 0: 5%|▌ | 19.0/363 [00:01<00:21, 15.9it/s]
Loading 0: 10%|█ | 37.0/363 [00:02<00:20, 15.7it/s]
Loading 0: 10%|█ | 37.0/363 [00:02<00:20, 15.7it/s]
Loading 0: 15%|█▌ | 55.0/363 [00:03<00:19, 15.7it/s]
Loading 0: 15%|█▌ | 55.0/363 [00:03<00:19, 15.7it/s]
Loading 0: 17%|█▋ | 63.0/363 [00:04<00:23, 12.8it/s]
Loading 0: 17%|█▋ | 63.0/363 [00:04<00:23, 12.8it/s]
Loading 0: 23%|██▎ | 82.0/363 [00:05<00:19, 14.2it/s]
Loading 0: 23%|██▎ | 82.0/363 [00:05<00:19, 14.2it/s]
Loading 0: 29%|██▉ | 105/363 [00:06<00:15, 16.3it/s]
Loading 0: 29%|██▉ | 105/363 [00:06<00:15, 16.3it/s]
Loading 0: 35%|███▍ | 126/363 [00:07<00:13, 17.5it/s]
Loading 0: 35%|███▍ | 126/363 [00:07<00:13, 17.5it/s]
Loading 0: 39%|███▉ | 142/363 [00:08<00:13, 16.3it/s]
Loading 0: 39%|███▉ | 142/363 [00:08<00:13, 16.3it/s]
Loading 0: 45%|████▍ | 162/363 [00:10<00:11, 17.3it/s]
Loading 0: 45%|████▍ | 162/363 [00:10<00:11, 17.3it/s]
Loading 0: 50%|█████ | 183/363 [00:11<00:09, 18.1it/s]
Loading 0: 50%|█████ | 183/363 [00:11<00:09, 18.1it/s]
Loading 0: 56%|█████▌ | 203/363 [00:12<00:08, 18.3it/s]
Loading 0: 56%|█████▌ | 203/363 [00:12<00:08, 18.3it/s]
Loading 0: 61%|██████▏ | 223/363 [00:13<00:07, 17.8it/s]
Loading 0: 61%|██████▏ | 223/363 [00:13<00:07, 17.8it/s]
Loading 0: 66%|██████▌ | 239/363 [00:14<00:07, 17.1it/s]
Loading 0: 66%|██████▌ | 239/363 [00:14<00:07, 17.1it/s]
Loading 0: 72%|███████▏ | 262/363 [00:15<00:05, 18.4it/s]
Loading 0: 72%|███████▏ | 262/363 [00:15<00:05, 18.4it/s]
Loading 0: 78%|███████▊ | 283/363 [00:16<00:04, 19.1it/s]
Loading 0: 78%|███████▊ | 283/363 [00:16<00:04, 19.1it/s]
Loading 0: 83%|████████▎ | 303/363 [00:17<00:03, 19.3it/s]
Loading 0: 83%|████████▎ | 303/363 [00:17<00:03, 19.3it/s]
Loading 0: 85%|████████▍ | 307/363 [00:18<00:03, 14.2it/s]
Loading 0: 85%|████████▍ | 307/363 [00:18<00:03, 14.2it/s]
Loading 0: 91%|█████████ | 330/363 [00:19<00:02, 15.9it/s]
Loading 0: 91%|█████████ | 330/363 [00:19<00:02, 15.9it/s]
Loading 0: 97%|█████████▋| 353/363 [00:20<00:00, 17.8it/s]
Loading 0: 97%|█████████▋| 353/363 [00:20<00:00, 17.8it/s]
Loading 0: 100%|██████████| 363/363 [00:21<00:00, 17.8it/s]
Loading 0: 100%|██████████| 363/363 [00:21<00:00, 17.8it/s]
Loading 0: 100%|██████████| 363/363 [00:21<00:00, 17.0it/s]
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: quantized model in 38.009s
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Processed model ChaiML/nitral-dpos1-beta06-exp1-10kpref-1800-lowlr-2e-2-1 in 96.212s
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: creating bucket guanaco-mkml-models
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd/config.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd/special_tokens_map.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd/tokenizer_config.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd/tokenizer.json
chaiml-nitral-dpos1-bet-5231-v12-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-nitral-dpos1-bet-5231-v12/amd/flywheel_model.0.safetensors
Job chaiml-nitral-dpos1-bet-5231-v12-mkmlizer completed after 138.43s with status: succeeded
Stopping job with name chaiml-nitral-dpos1-bet-5231-v12-mkmlizer
Pipeline stage AMDMKMLizer completed in 139.61s
Shutdown handler de-registered
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem
chaiml-nitral-dpos1-bet_5231_v12 status is now protected due to ABTestQueueItem