Running pipeline stage MKMLizer
Starting job with name zonemercy-acute-nemo-v0-5087-v4-mkmlizer
Waiting for job on zonemercy-acute-nemo-v0-5087-v4-mkmlizer to finish
Stopping job with name zonemercy-acute-nemo-v0-5087-v4-mkmlizer
%s, retrying in %s seconds...
Starting job with name zonemercy-acute-nemo-v0-5087-v4-mkmlizer
Waiting for job on zonemercy-acute-nemo-v0-5087-v4-mkmlizer to finish
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ _____ __ __ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ /___/ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ Version: 0.9.11 ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ https://mk1.ai ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ belonging to: ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ Chai Research Corp. ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ║ ║
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission neversleep-noromaid-v0_8068_v133: ('http://neversleep-noromaid-v0-8068-v133-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:52216->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: Downloaded to shared memory in 76.879s
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpmu4dqj2l, device:0
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: quantized model in 41.871s
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: Processed model zonemercy/Acute-Nemo-v0-5e6ep1 in 118.750s
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4/config.json
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4/special_tokens_map.json
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4/tokenizer_config.json
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4/tokenizer.json
zonemercy-acute-nemo-v0-5087-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-acute-nemo-v0-5087-v4/flywheel_model.0.safetensors
zonemercy-acute-nemo-v0-5087-v4-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:15, 23.36it/s]
Loading 0: 3%|▎ | 10/363 [00:00<00:12, 27.96it/s]
Loading 0: 4%|▍ | 14/363 [00:00<00:14, 24.36it/s]
Loading 0: 5%|▌ | 19/363 [00:00<00:11, 30.65it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:14, 23.07it/s]
Loading 0: 7%|▋ | 26/363 [00:01<00:16, 20.52it/s]
Loading 0: 9%|▊ | 31/363 [00:01<00:12, 26.14it/s]
Loading 0: 10%|▉ | 35/363 [00:01<00:12, 27.19it/s]
Loading 0: 11%|█ | 39/363 [00:01<00:11, 27.94it/s]
Loading 0: 12%|█▏ | 43/363 [00:01<00:12, 26.64it/s]
Loading 0: 13%|█▎ | 48/363 [00:01<00:10, 28.89it/s]
Loading 0: 14%|█▍ | 52/363 [00:01<00:11, 27.83it/s]
Loading 0: 15%|█▌ | 56/363 [00:02<00:11, 27.87it/s]
Loading 0: 17%|█▋ | 60/363 [00:02<00:09, 30.54it/s]
Loading 0: 18%|█▊ | 64/363 [00:02<00:15, 19.90it/s]
Loading 0: 19%|█▉ | 69/363 [00:02<00:11, 24.98it/s]
Loading 0: 20%|██ | 73/363 [00:02<00:11, 25.05it/s]
Loading 0: 21%|██ | 77/363 [00:03<00:12, 23.18it/s]
Loading 0: 23%|██▎ | 82/363 [00:03<00:10, 28.00it/s]
Loading 0: 24%|██▎ | 86/363 [00:03<00:10, 25.56it/s]
Loading 0: 25%|██▌ | 91/363 [00:03<00:08, 30.33it/s]
Loading 0: 26%|██▌ | 95/363 [00:03<00:10, 26.57it/s]
Loading 0: 28%|██▊ | 100/363 [00:03<00:08, 31.20it/s]
Loading 0: 29%|██▊ | 104/363 [00:04<00:12, 20.49it/s]
Loading 0: 30%|███ | 109/363 [00:04<00:10, 25.13it/s]
Loading 0: 31%|███ | 113/363 [00:04<00:10, 23.32it/s]
Loading 0: 33%|███▎ | 118/363 [00:04<00:08, 28.10it/s]
Loading 0: 34%|███▎ | 122/363 [00:04<00:09, 25.35it/s]
Loading 0: 35%|███▍ | 127/363 [00:04<00:08, 29.45it/s]
Loading 0: 36%|███▌ | 131/363 [00:05<00:08, 26.16it/s]
Loading 0: 37%|███▋ | 136/363 [00:05<00:07, 30.68it/s]
Loading 0: 39%|███▉ | 141/363 [00:05<00:07, 31.25it/s]
Loading 0: 40%|███▉ | 145/363 [00:05<00:09, 22.10it/s]
Loading 0: 41%|████ | 149/363 [00:05<00:09, 21.47it/s]
Loading 0: 42%|████▏ | 154/363 [00:05<00:07, 26.40it/s]
Loading 0: 44%|████▎ | 158/363 [00:06<00:08, 24.47it/s]
Loading 0: 45%|████▍ | 163/363 [00:06<00:06, 29.31it/s]
Loading 0: 46%|████▌ | 167/363 [00:06<00:07, 26.06it/s]
Loading 0: 47%|████▋ | 172/363 [00:06<00:06, 30.76it/s]
Loading 0: 48%|████▊ | 176/363 [00:06<00:06, 27.10it/s]
Loading 0: 50%|████▉ | 181/363 [00:06<00:05, 30.96it/s]
Loading 0: 51%|█████ | 185/363 [00:07<00:08, 20.61it/s]
Loading 0: 52%|█████▏ | 190/363 [00:07<00:06, 25.40it/s]
Loading 0: 53%|█████▎ | 194/363 [00:07<00:07, 23.57it/s]
Loading 0: 55%|█████▍ | 199/363 [00:07<00:05, 28.33it/s]
Loading 0: 56%|█████▌ | 203/363 [00:07<00:06, 25.42it/s]
Loading 0: 58%|█████▊ | 210/363 [00:07<00:04, 31.20it/s]
Loading 0: 59%|█████▉ | 214/363 [00:08<00:05, 28.88it/s]
Loading 0: 60%|██████ | 218/363 [00:08<00:05, 28.14it/s]
Loading 0: 61%|██████▏ | 223/363 [00:08<00:05, 23.50it/s]
Loading 0: 62%|██████▏ | 226/363 [00:08<00:06, 22.32it/s]
Loading 0: 63%|██████▎ | 230/363 [00:08<00:06, 21.83it/s]
Loading 0: 65%|██████▍ | 235/363 [00:09<00:04, 26.82it/s]
Loading 0: 66%|██████▌ | 239/363 [00:09<00:05, 24.77it/s]
Loading 0: 67%|██████▋ | 244/363 [00:09<00:04, 29.70it/s]
Loading 0: 68%|██████▊ | 248/363 [00:09<00:04, 26.42it/s]
Loading 0: 70%|███████ | 255/363 [00:09<00:03, 32.26it/s]
Loading 0: 71%|███████▏ | 259/363 [00:09<00:03, 30.08it/s]
Loading 0: 72%|███████▏ | 263/363 [00:10<00:04, 24.36it/s]
Loading 0: 73%|███████▎ | 266/363 [00:10<00:04, 21.78it/s]
Loading 0: 75%|███████▍ | 271/363 [00:10<00:03, 26.82it/s]
Loading 0: 76%|███████▌ | 275/363 [00:10<00:03, 24.54it/s]
Loading 0: 77%|███████▋ | 280/363 [00:10<00:02, 29.25it/s]
Loading 0: 78%|███████▊ | 284/363 [00:10<00:03, 26.23it/s]
Loading 0: 80%|███████▉ | 289/363 [00:10<00:02, 31.01it/s]
Loading 0: 81%|████████ | 293/363 [00:11<00:02, 26.76it/s]
Loading 0: 82%|████████▏ | 298/363 [00:11<00:02, 31.56it/s]
Loading 0: 83%|████████▎ | 303/363 [00:11<00:01, 31.90it/s]
Loading 0: 85%|████████▍ | 307/363 [00:11<00:02, 22.14it/s]
Loading 0: 86%|████████▌ | 311/363 [00:11<00:02, 21.44it/s]
Loading 0: 87%|████████▋ | 316/363 [00:12<00:01, 26.23it/s]
Loading 0: 88%|████████▊ | 320/363 [00:12<00:01, 24.36it/s]
Loading 0: 90%|█████████ | 327/363 [00:12<00:01, 30.34it/s]
Loading 0: 91%|█████████ | 331/363 [00:12<00:01, 28.77it/s]
Loading 0: 92%|█████████▏| 335/363 [00:12<00:00, 30.97it/s]
Loading 0: 93%|█████████▎| 339/363 [00:12<00:00, 27.57it/s]
Loading 0: 95%|█████████▍| 344/363 [00:19<00:09, 2.08it/s]
Loading 0: 96%|█████████▌| 348/363 [00:20<00:05, 2.76it/s]
Loading 0: 97%|█████████▋| 353/363 [00:20<00:02, 3.96it/s]
Loading 0: 98%|█████████▊| 357/363 [00:20<00:01, 5.08it/s]
Job zonemercy-acute-nemo-v0-5087-v4-mkmlizer completed after 148.26s with status: succeeded
Stopping job with name zonemercy-acute-nemo-v0-5087-v4-mkmlizer
Pipeline stage MKMLizer completed in 149.95s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.09s
Running pipeline stage ISVCDeployer
Creating inference service zonemercy-acute-nemo-v0-5087-v4
Waiting for inference service zonemercy-acute-nemo-v0-5087-v4 to be ready
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v11: ('http://zonemercy-acute-nemo-v1-4488-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v11: ('http://zonemercy-acute-nemo-v1-4488-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v11: ('http://zonemercy-acute-nemo-v1-4488-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v11: ('http://zonemercy-acute-nemo-v1-4488-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v11: ('http://zonemercy-acute-nemo-v1-4488-v11-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-acute-nemo-v1_4488_v10: ('http://zonemercy-acute-nemo-v1-4488-v10-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v19: ('http://undi95-meta-llama-3-70b-6209-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-cogent-nemo-v2-5e6_v17: ('http://chaiml-llama-8b-pairwise-8189-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:38774->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission blend_berib_2024-08-16: ('http://zonemercy-graft-cogent-v-7573-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:59566->127.0.0.1:8080: read: connection reset by peer\n')
Inference service zonemercy-acute-nemo-v0-5087-v4 ready after 261.87372732162476s
Pipeline stage ISVCDeployer completed in 262.57s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.272747039794922s
Received healthy response to inference request in 3.256380319595337s
{"detail":"('http://chaiml-llama-8b-pairwise-8189-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:35284->127.0.0.1:8080: read: connection reset by peer\\n')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 2.372360944747925s
Received healthy response to inference request in 2.3803768157958984s
5 requests
1 failed requests
5th percentile: 2.2270028591156006
10th percentile: 2.2633423805236816
20th percentile: 2.3360214233398438
30th percentile: 2.3739641189575194
40th percentile: 2.377170467376709
50th percentile: 2.3803768157958984
60th percentile: 2.730778217315674
70th percentile: 3.081179618835449
80th percentile: 3.259653663635254
90th percentile: 3.2662003517150877
95th percentile: 3.269473695755005
99th percentile: 3.2720923709869383
mean time: 2.6945056915283203
%s, retrying in %s seconds...
Received healthy response to inference request in 2.5766615867614746s
Received healthy response to inference request in 2.4607627391815186s
Received healthy response to inference request in 2.6727497577667236s
Received healthy response to inference request in 2.5570623874664307s
Received healthy response to inference request in 2.563990354537964s
5 requests
0 failed requests
5th percentile: 2.480022668838501
10th percentile: 2.4992825984954834
20th percentile: 2.5378024578094482
30th percentile: 2.558447980880737
40th percentile: 2.5612191677093508
50th percentile: 2.563990354537964
60th percentile: 2.5690588474273683
70th percentile: 2.5741273403167724
80th percentile: 2.5958792209625243
90th percentile: 2.634314489364624
95th percentile: 2.653532123565674
99th percentile: 2.6689062309265137
mean time: 2.566245365142822
Pipeline stage StressChecker completed in 27.95s
zonemercy-acute-nemo-v0-_5087_v4 status is now deployed due to DeploymentManager action
zonemercy-acute-nemo-v0-_5087_v4 status is now inactive due to auto deactivation removed underperforming models
zonemercy-acute-nemo-v0-_5087_v4 status is now torndown due to DeploymentManager action