Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-20250611-retune-48497-v3-mkmlizer
Waiting for job on chaiml-20250611-retune-48497-v3-mkmlizer to finish
chaiml-20250611-retune-48497-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ https://mk1.ai ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ belonging to: ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Chai Research Corp. ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-20250611-retune-48497-v3-mkmlizer: ║ ║
chaiml-20250611-retune-48497-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-20250611-retune-48497-v3-mkmlizer: Downloaded to shared memory in 27.172s
chaiml-20250611-retune-48497-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp6axveotb, device:0
chaiml-20250611-retune-48497-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-20250611-retune-48497-v3-mkmlizer: quantized model in 19.769s
chaiml-20250611-retune-48497-v3-mkmlizer: Processed model ChaiML/20250611_retune_subscribed in 46.942s
chaiml-20250611-retune-48497-v3-mkmlizer: creating bucket guanaco-mkml-models
chaiml-20250611-retune-48497-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-20250611-retune-48497-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-20250611-retune-48497-v3
chaiml-20250611-retune-48497-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-20250611-retune-48497-v3/config.json
chaiml-20250611-retune-48497-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-20250611-retune-48497-v3/special_tokens_map.json
chaiml-20250611-retune-48497-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-20250611-retune-48497-v3/tokenizer_config.json
chaiml-20250611-retune-48497-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-20250611-retune-48497-v3/tokenizer.json
chaiml-20250611-retune-48497-v3-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 2%|▏ | 5/291 [00:00<00:09, 31.38it/s]
Loading 0: 4%|▍ | 11/291 [00:00<00:06, 44.84it/s]
Loading 0: 5%|▌ | 16/291 [00:00<00:06, 40.26it/s]
Loading 0: 7%|▋ | 21/291 [00:00<00:06, 41.28it/s]
Loading 0: 9%|▉ | 26/291 [00:00<00:06, 39.45it/s]
Loading 0: 11%|█ | 31/291 [00:00<00:06, 39.52it/s]
Loading 0: 12%|█▏ | 36/291 [00:00<00:06, 40.68it/s]
Loading 0: 14%|█▍ | 41/291 [00:01<00:07, 31.98it/s]
Loading 0: 16%|█▋ | 48/291 [00:01<00:06, 39.85it/s]
Loading 0: 18%|█▊ | 53/291 [00:01<00:05, 39.95it/s]
Loading 0: 20%|█▉ | 58/291 [00:01<00:05, 40.35it/s]
Loading 0: 22%|██▏ | 63/291 [00:01<00:05, 41.79it/s]
Loading 0: 23%|██▎ | 68/291 [00:01<00:06, 32.96it/s]
Loading 0: 26%|██▌ | 75/291 [00:01<00:05, 40.13it/s]
Loading 0: 27%|██▋ | 80/291 [00:02<00:05, 40.01it/s]
Loading 0: 29%|██▉ | 85/291 [00:02<00:08, 24.98it/s]
Loading 0: 31%|███ | 90/291 [00:02<00:06, 28.91it/s]
Loading 0: 32%|███▏ | 94/291 [00:02<00:06, 30.37it/s]
Loading 0: 34%|███▍ | 99/291 [00:02<00:05, 34.46it/s]
Loading 0: 36%|███▌ | 104/291 [00:02<00:05, 32.37it/s]
Loading 0: 38%|███▊ | 112/291 [00:03<00:04, 41.91it/s]
Loading 0: 41%|████ | 118/291 [00:03<00:04, 41.26it/s]
Loading 0: 42%|████▏ | 123/291 [00:03<00:04, 41.57it/s]
Loading 0: 45%|████▍ | 130/291 [00:03<00:03, 47.11it/s]
Loading 0: 47%|████▋ | 136/291 [00:03<00:03, 45.20it/s]
Loading 0: 48%|████▊ | 141/291 [00:03<00:03, 44.20it/s]
Loading 0: 51%|█████ | 148/291 [00:03<00:02, 50.11it/s]
Loading 0: 53%|█████▎ | 154/291 [00:03<00:02, 47.37it/s]
Loading 0: 55%|█████▍ | 159/291 [00:04<00:02, 45.96it/s]
Loading 0: 57%|█████▋ | 166/291 [00:04<00:02, 51.51it/s]
Loading 0: 59%|█████▉ | 172/291 [00:04<00:02, 48.51it/s]
Loading 0: 61%|██████ | 177/291 [00:04<00:02, 48.64it/s]
Loading 0: 63%|██████▎ | 182/291 [00:04<00:02, 45.02it/s]
Loading 0: 64%|██████▍ | 187/291 [00:04<00:03, 30.94it/s]
Loading 0: 66%|██████▌ | 191/291 [00:04<00:03, 32.01it/s]
Loading 0: 67%|██████▋ | 195/291 [00:05<00:02, 32.51it/s]
Loading 0: 69%|██████▉ | 202/291 [00:05<00:02, 39.68it/s]
Loading 0: 71%|███████▏ | 208/291 [00:05<00:02, 39.56it/s]
Loading 0: 73%|███████▎ | 213/291 [00:05<00:01, 40.37it/s]
Loading 0: 76%|███████▌ | 220/291 [00:05<00:01, 46.63it/s]
Loading 0: 78%|███████▊ | 226/291 [00:05<00:01, 44.92it/s]
Loading 0: 79%|███████▉ | 231/291 [00:05<00:01, 43.70it/s]
Loading 0: 82%|████████▏ | 238/291 [00:05<00:01, 49.22it/s]
Loading 0: 84%|████████▍ | 244/291 [00:06<00:01, 46.22it/s]
Loading 0: 86%|████████▌ | 249/291 [00:06<00:00, 45.90it/s]
Loading 0: 88%|████████▊ | 256/291 [00:06<00:00, 51.06it/s]
Loading 0: 90%|█████████ | 262/291 [00:06<00:00, 48.12it/s]
Loading 0: 92%|█████████▏| 267/291 [00:06<00:00, 46.97it/s]
Loading 0: 94%|█████████▍| 274/291 [00:06<00:00, 51.59it/s]
Loading 0: 96%|█████████▌| 280/291 [00:06<00:00, 47.85it/s]
Loading 0: 98%|█████████▊| 285/291 [00:06<00:00, 47.22it/s]
Loading 0: 100%|█████████▉| 290/291 [00:07<00:00, 33.43it/s]
Job chaiml-20250611-retune-48497-v3-mkmlizer completed after 74.1s with status: succeeded
Stopping job with name chaiml-20250611-retune-48497-v3-mkmlizer
Pipeline stage MKMLizer completed in 74.64s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-20250611-retune-48497-v3
Waiting for inference service chaiml-20250611-retune-48497-v3 to be ready
Inference service chaiml-20250611-retune-48497-v3 ready after 100.45642852783203s
Pipeline stage MKMLDeployer completed in 101.06s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.3367602825164795s
Received healthy response to inference request in 2.6872611045837402s
Received healthy response to inference request in 4.264543771743774s
Received healthy response to inference request in 2.719316005706787s
Received healthy response to inference request in 5.716990232467651s
5 requests
0 failed requests
5th percentile: 2.6936720848083495
10th percentile: 2.700083065032959
20th percentile: 2.712905025482178
30th percentile: 2.8428048610687258
40th percentile: 3.0897825717926026
50th percentile: 3.3367602825164795
60th percentile: 3.7078736782073975
70th percentile: 4.078987073898316
80th percentile: 4.55503306388855
90th percentile: 5.1360116481781
95th percentile: 5.426500940322875
99th percentile: 5.658892374038696
mean time: 3.7449742794036864
%s, retrying in %s seconds...
Received healthy response to inference request in 4.086052179336548s
Received healthy response to inference request in 4.618359804153442s
Received healthy response to inference request in 4.649899482727051s
Received healthy response to inference request in 3.7273502349853516s
Received healthy response to inference request in 2.5720348358154297s
5 requests
0 failed requests
5th percentile: 2.8030979156494142
10th percentile: 3.0341609954833983
20th percentile: 3.496287155151367
30th percentile: 3.799090623855591
40th percentile: 3.9425714015960693
50th percentile: 4.086052179336548
60th percentile: 4.298975229263306
70th percentile: 4.5118982791900635
80th percentile: 4.624667739868164
90th percentile: 4.637283611297607
95th percentile: 4.6435915470123295
99th percentile: 4.648637895584106
mean time: 3.9307393074035644
%s, retrying in %s seconds...
Received healthy response to inference request in 4.0719945430755615s
Received healthy response to inference request in 2.919313669204712s
Received healthy response to inference request in 4.070603370666504s
Received healthy response to inference request in 3.5447444915771484s
Received healthy response to inference request in 4.121107578277588s
5 requests
0 failed requests
5th percentile: 3.0443998336791993
10th percentile: 3.1694859981536867
20th percentile: 3.419658327102661
30th percentile: 3.6499162673950196
40th percentile: 3.8602598190307615
50th percentile: 4.070603370666504
60th percentile: 4.071159839630127
70th percentile: 4.07171630859375
80th percentile: 4.081817150115967
90th percentile: 4.101462364196777
95th percentile: 4.111284971237183
99th percentile: 4.119143056869507
mean time: 3.745552730560303
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.07171630859375s')
Shutdown handler de-registered
chaiml-20250611-retune-_48497_v3 status is now failed due to DeploymentManager action