Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nem-93303-v318-mkmlizer
Waiting for job on mistralai-mistral-nem-93303-v318-mkmlizer to finish
mistralai-mistral-nem-93303-v318-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nem-93303-v318-mkmlizer: ║ _____ __ __ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ /___/ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ Version: 0.11.12 ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ https://mk1.ai ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ The license key for the current software has been verified as ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ belonging to: ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ Chai Research Corp. ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
mistralai-mistral-nem-93303-v318-mkmlizer: ║ ║
mistralai-mistral-nem-93303-v318-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nem-93303-v318-mkmlizer: Downloaded to shared memory in 55.312s
mistralai-mistral-nem-93303-v318-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp6a20e__d, device:0
mistralai-mistral-nem-93303-v318-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nem-93303-v318-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nem-93303-v318-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nem-93303-v318-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318
mistralai-mistral-nem-93303-v318-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318/config.json
mistralai-mistral-nem-93303-v318-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318/special_tokens_map.json
mistralai-mistral-nem-93303-v318-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318/tokenizer_config.json
mistralai-mistral-nem-93303-v318-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318/tokenizer.json
mistralai-mistral-nem-93303-v318-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nem-93303-v318/flywheel_model.0.safetensors
mistralai-mistral-nem-93303-v318-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:13, 26.40it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:08, 43.82it/s]
Loading 0: 5%|▍ | 17/363 [00:00<00:08, 41.98it/s]
Loading 0: 6%|▌ | 22/363 [00:00<00:08, 41.14it/s]
Loading 0: 7%|▋ | 27/363 [00:00<00:07, 42.50it/s]
Loading 0: 9%|▉ | 32/363 [00:00<00:10, 30.98it/s]
Loading 0: 11%|█ | 39/363 [00:01<00:08, 38.80it/s]
Loading 0: 12%|█▏ | 44/363 [00:01<00:08, 39.80it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:07, 40.73it/s]
Loading 0: 15%|█▍ | 54/363 [00:01<00:07, 42.31it/s]
Loading 0: 16%|█▋ | 59/363 [00:01<00:06, 44.31it/s]
Loading 0: 18%|█▊ | 64/363 [00:01<00:12, 23.54it/s]
Loading 0: 20%|█▉ | 71/363 [00:02<00:09, 30.90it/s]
Loading 0: 21%|██ | 76/363 [00:02<00:09, 31.84it/s]
Loading 0: 22%|██▏ | 81/363 [00:02<00:09, 31.11it/s]
Loading 0: 24%|██▎ | 86/363 [00:02<00:08, 34.54it/s]
Loading 0: 25%|██▌ | 91/363 [00:02<00:08, 31.05it/s]
Loading 0: 27%|██▋ | 98/363 [00:02<00:07, 36.82it/s]
Loading 0: 28%|██▊ | 103/363 [00:02<00:06, 38.17it/s]
Loading 0: 30%|███ | 109/363 [00:02<00:05, 43.09it/s]
Loading 0: 31%|███▏ | 114/363 [00:03<00:06, 37.52it/s]
Loading 0: 33%|███▎ | 119/363 [00:03<00:06, 36.31it/s]
Loading 0: 34%|███▍ | 125/363 [00:03<00:06, 39.55it/s]
Loading 0: 36%|███▌ | 130/363 [00:03<00:06, 38.32it/s]
Loading 0: 37%|███▋ | 134/363 [00:03<00:06, 33.54it/s]
Loading 0: 38%|███▊ | 138/363 [00:03<00:06, 33.65it/s]
Loading 0: 39%|███▉ | 142/363 [00:04<00:09, 22.16it/s]
Loading 0: 40%|███▉ | 145/363 [00:04<00:09, 22.86it/s]
Loading 0: 41%|████ | 149/363 [00:04<00:09, 23.73it/s]
Loading 0: 42%|████▏ | 154/363 [00:04<00:07, 27.28it/s]
Loading 0: 44%|████▎ | 158/363 [00:04<00:07, 25.71it/s]
Loading 0: 45%|████▍ | 163/363 [00:04<00:06, 30.60it/s]
Loading 0: 46%|████▌ | 167/363 [00:05<00:06, 29.26it/s]
Loading 0: 47%|████▋ | 172/363 [00:05<00:05, 33.32it/s]
Loading 0: 48%|████▊ | 176/363 [00:05<00:06, 30.09it/s]
Loading 0: 50%|████▉ | 181/363 [00:05<00:05, 32.92it/s]
Loading 0: 51%|█████ | 185/363 [00:05<00:05, 29.75it/s]
Loading 0: 53%|█████▎ | 192/363 [00:05<00:04, 36.28it/s]
Loading 0: 54%|█████▍ | 196/363 [00:05<00:04, 35.51it/s]
Loading 0: 55%|█████▌ | 201/363 [00:05<00:04, 36.97it/s]
Loading 0: 56%|█████▋ | 205/363 [00:06<00:04, 36.08it/s]
Loading 0: 58%|█████▊ | 210/363 [00:06<00:04, 36.40it/s]
Loading 0: 59%|█████▉ | 214/363 [00:06<00:04, 31.50it/s]
Loading 0: 60%|██████ | 218/363 [00:06<00:04, 31.29it/s]
Loading 0: 61%|██████▏ | 223/363 [00:06<00:05, 25.50it/s]
Loading 0: 62%|██████▏ | 226/363 [00:06<00:05, 25.09it/s]
Loading 0: 63%|██████▎ | 230/363 [00:07<00:05, 25.98it/s]
Loading 0: 65%|██████▍ | 235/363 [00:07<00:04, 30.87it/s]
Loading 0: 66%|██████▌ | 239/363 [00:07<00:04, 29.27it/s]
Loading 0: 68%|██████▊ | 246/363 [00:07<00:03, 37.04it/s]
Loading 0: 69%|██████▉ | 250/363 [00:07<00:03, 37.27it/s]
Loading 0: 71%|███████ | 256/363 [00:07<00:02, 41.05it/s]
Loading 0: 72%|███████▏ | 262/363 [00:07<00:02, 39.50it/s]
Loading 0: 74%|███████▎ | 267/363 [00:07<00:02, 38.72it/s]
Loading 0: 75%|███████▌ | 273/363 [00:08<00:02, 43.45it/s]
Loading 0: 77%|███████▋ | 278/363 [00:08<00:01, 44.00it/s]
Loading 0: 78%|███████▊ | 283/363 [00:08<00:01, 43.32it/s]
Loading 0: 79%|███████▉ | 288/363 [00:08<00:01, 44.60it/s]
Loading 0: 81%|████████ | 293/363 [00:08<00:01, 37.59it/s]
Loading 0: 82%|████████▏ | 299/363 [00:08<00:01, 42.58it/s]
Loading 0: 84%|████████▎ | 304/363 [00:16<00:25, 2.31it/s]
Loading 0: 85%|████████▍ | 308/363 [00:16<00:18, 2.98it/s]
Loading 0: 86%|████████▌ | 312/363 [00:16<00:13, 3.90it/s]
Loading 0: 88%|████████▊ | 319/363 [00:16<00:07, 6.18it/s]
Loading 0: 89%|████████▉ | 324/363 [00:16<00:04, 8.13it/s]
Loading 0: 90%|█████████ | 328/363 [00:16<00:03, 9.78it/s]
Loading 0: 91%|█████████▏| 332/363 [00:16<00:02, 11.75it/s]
Loading 0: 93%|█████████▎| 337/363 [00:16<00:01, 15.45it/s]
Loading 0: 94%|█████████▍| 341/363 [00:17<00:01, 18.08it/s]
Loading 0: 95%|█████████▌| 346/363 [00:17<00:00, 22.76it/s]
Loading 0: 97%|█████████▋| 351/363 [00:17<00:00, 26.65it/s]
Loading 0: 98%|█████████▊| 356/363 [00:17<00:00, 30.00it/s]
Loading 0: 99%|█████████▉| 361/363 [00:17<00:00, 32.97it/s]
Job mistralai-mistral-nem-93303-v318-mkmlizer completed after 144.12s with status: succeeded
Stopping job with name mistralai-mistral-nem-93303-v318-mkmlizer
Pipeline stage MKMLizer completed in 144.57s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service mistralai-mistral-nem-93303-v318
Waiting for inference service mistralai-mistral-nem-93303-v318 to be ready
Failed to get response for submission randomforest1024-v25_v8: HTTPConnectionPool(host='randomforest1024-v25-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service mistralai-mistral-nem-93303-v318 ready after 201.29503774642944s
Pipeline stage MKMLDeployer completed in 201.81s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
Failed to get response for submission randomforest1024-v25_v8: HTTPConnectionPool(host='randomforest1024-v25-v8-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.18384981155395508
10th percentile: 0.18393564224243164
20th percentile: 0.18410730361938477
30th percentile: 0.18419408798217773
40th percentile: 0.18419599533081055
50th percentile: 0.18419790267944336
60th percentile: 0.19449834823608397
70th percentile: 0.20479879379272461
80th percentile: 0.2103729724884033
90th percentile: 0.2112208843231201
95th percentile: 0.21164484024047853
99th percentile: 0.21198400497436523
mean time: 0.194834566116333
%s, retrying in %s seconds...
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.16644158363342285
10th percentile: 0.16777758598327636
20th percentile: 0.1704495906829834
30th percentile: 0.17258987426757813
40th percentile: 0.17419843673706054
50th percentile: 0.17580699920654297
60th percentile: 0.17590246200561524
70th percentile: 0.1759979248046875
80th percentile: 0.1774156093597412
90th percentile: 0.18015551567077637
95th percentile: 0.18152546882629395
99th percentile: 0.18262143135070802
mean time: 0.17432785034179688
%s, retrying in %s seconds...
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
{"detail":"Single '}' encountered in format string"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 0.19604167938232422
10th percentile: 0.19789977073669435
20th percentile: 0.20161595344543456
30th percentile: 0.2036156177520752
40th percentile: 0.20389876365661622
50th percentile: 0.20418190956115723
60th percentile: 0.23887009620666502
70th percentile: 0.2735582828521728
80th percentile: 0.33569521903991706
90th percentile: 0.4252809047698975
95th percentile: 0.47007374763488763
99th percentile: 0.5059080219268799
mean time: 0.28152170181274416
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
mistralai-mistral-nem_93303_v318 status is now failed due to DeploymentManager action