Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v13-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v13-mkmlizer to finish
Failed to get response for submission neversleep-lumimaid-mist_1884_v1: ('http://neversleep-lumimaid-mist-1884-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
mistralai-mistral-nemo-9330-v13-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ _____ __ __ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ /___/ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ Version: 0.9.6 ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ https://mk1.ai ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ The license key for the current software has been verified as ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ belonging to: ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ Chai Research Corp. ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ Expiration: 2024-10-15 23:59:59 ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ║ ║
mistralai-mistral-nemo-9330-v13-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission neversleep-lumimaid-mist_7954_v1: ('http://neversleep-lumimaid-mist-7954-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_7954_v1: ('http://neversleep-lumimaid-mist-7954-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
mistralai-mistral-nemo-9330-v13-mkmlizer: Downloaded to shared memory in 49.593s
mistralai-mistral-nemo-9330-v13-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmps7eonvsj, device:0
mistralai-mistral-nemo-9330-v13-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_7954_v1: ('http://neversleep-lumimaid-mist-7954-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1884_v1: ('http://neversleep-lumimaid-mist-1884-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1884_v1: ('http://neversleep-lumimaid-mist-1884-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
mistralai-mistral-nemo-9330-v13-mkmlizer: quantized model in 35.367s
mistralai-mistral-nemo-9330-v13-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 84.959s
mistralai-mistral-nemo-9330-v13-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v13-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v13-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13/config.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13/special_tokens_map.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13/tokenizer_config.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13/tokenizer.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v13/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v13-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
mistralai-mistral-nemo-9330-v13-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:10, 34.21it/s]
Loading 0: 4%|▎ | 13/363 [00:00<00:06, 54.08it/s]
Loading 0: 5%|▌ | 19/363 [00:00<00:07, 47.76it/s]
Loading 0: 7%|▋ | 24/363 [00:00<00:07, 45.66it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:06, 51.33it/s]
Loading 0: 10%|█ | 37/363 [00:00<00:06, 47.97it/s]
Loading 0: 12%|█▏ | 42/363 [00:00<00:06, 46.00it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:06, 50.83it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:06, 48.01it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:08, 36.78it/s]
Loading 0: 18%|█▊ | 66/363 [00:01<00:07, 38.08it/s]
Loading 0: 20%|█▉ | 72/363 [00:01<00:06, 42.50it/s]
Loading 0: 21%|██▏ | 78/363 [00:01<00:06, 42.08it/s]
Loading 0: 23%|██▎ | 83/363 [00:01<00:06, 42.05it/s]
Loading 0: 25%|██▍ | 90/363 [00:02<00:05, 47.21it/s]
Loading 0: 26%|██▌ | 95/363 [00:02<00:05, 47.77it/s]
Loading 0: 28%|██▊ | 100/363 [00:02<00:06, 41.00it/s]
Loading 0: 30%|██▉ | 108/363 [00:02<00:05, 50.32it/s]
Loading 0: 31%|███▏ | 114/363 [00:02<00:05, 45.42it/s]
Loading 0: 33%|███▎ | 119/363 [00:02<00:05, 44.31it/s]
Loading 0: 34%|███▍ | 125/363 [00:02<00:04, 47.75it/s]
Loading 0: 36%|███▌ | 131/363 [00:02<00:04, 49.04it/s]
Loading 0: 38%|███▊ | 137/363 [00:03<00:05, 42.70it/s]
Loading 0: 39%|███▉ | 142/363 [00:03<00:06, 33.46it/s]
Loading 0: 40%|████ | 146/363 [00:03<00:06, 34.75it/s]
Loading 0: 41%|████▏ | 150/363 [00:03<00:06, 34.15it/s]
Loading 0: 43%|████▎ | 156/363 [00:03<00:05, 39.66it/s]
Loading 0: 44%|████▍ | 161/363 [00:03<00:04, 41.43it/s]
Loading 0: 46%|████▌ | 166/363 [00:03<00:04, 42.91it/s]
Loading 0: 47%|████▋ | 171/363 [00:03<00:04, 44.74it/s]
Loading 0: 48%|████▊ | 176/363 [00:04<00:04, 37.41it/s]
Loading 0: 51%|█████ | 184/363 [00:04<00:03, 45.46it/s]
Loading 0: 52%|█████▏ | 190/363 [00:04<00:03, 44.18it/s]
Loading 0: 54%|█████▎ | 195/363 [00:04<00:03, 43.16it/s]
Loading 0: 56%|█████▌ | 202/363 [00:04<00:03, 47.59it/s]
Loading 0: 57%|█████▋ | 208/363 [00:04<00:03, 45.49it/s]
Loading 0: 59%|█████▊ | 213/363 [00:04<00:03, 44.16it/s]
Loading 0: 60%|██████ | 218/363 [00:05<00:03, 45.34it/s]
Loading 0: 61%|██████▏ | 223/363 [00:05<00:04, 34.76it/s]
Loading 0: 63%|██████▎ | 228/363 [00:05<00:03, 36.12it/s]
Loading 0: 64%|██████▍ | 232/363 [00:05<00:03, 36.81it/s]
Loading 0: 66%|██████▌ | 238/363 [00:05<00:03, 40.85it/s]
Loading 0: 67%|██████▋ | 244/363 [00:05<00:02, 40.42it/s]
Loading 0: 69%|██████▊ | 249/363 [00:05<00:02, 40.58it/s]
Loading 0: 71%|███████ | 256/363 [00:05<00:02, 45.60it/s]
Loading 0: 72%|███████▏ | 262/363 [00:06<00:02, 43.63it/s]
Loading 0: 74%|███████▎ | 267/363 [00:06<00:02, 42.30it/s]
Loading 0: 75%|███████▌ | 273/363 [00:06<00:01, 46.56it/s]
Loading 0: 77%|███████▋ | 278/363 [00:06<00:01, 46.09it/s]
Loading 0: 78%|███████▊ | 283/363 [00:06<00:01, 45.47it/s]
Loading 0: 79%|███████▉ | 288/363 [00:06<00:01, 46.35it/s]
Loading 0: 81%|████████ | 293/363 [00:06<00:01, 38.46it/s]
Loading 0: 83%|████████▎ | 300/363 [00:06<00:01, 46.07it/s]
Loading 0: 84%|████████▍ | 305/363 [00:13<00:22, 2.60it/s]
Loading 0: 85%|████████▌ | 309/363 [00:13<00:16, 3.34it/s]
Loading 0: 86%|████████▌ | 313/363 [00:13<00:11, 4.33it/s]
Loading 0: 88%|████████▊ | 320/363 [00:14<00:06, 6.78it/s]
Loading 0: 90%|████████▉ | 326/363 [00:14<00:04, 9.24it/s]
Loading 0: 91%|█████████ | 331/363 [00:14<00:02, 11.80it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:01, 16.70it/s]
Loading 0: 95%|█████████▍| 344/363 [00:14<00:00, 20.50it/s]
Loading 0: 96%|█████████▌| 349/363 [00:14<00:00, 23.86it/s]
Loading 0: 98%|█████████▊| 356/363 [00:14<00:00, 30.58it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 32.55it/s]
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:950: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
mistralai-mistral-nemo-9330-v13-mkmlizer: warnings.warn(
mistralai-mistral-nemo-9330-v13-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:778: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
mistralai-mistral-nemo-9330-v13-mkmlizer: warnings.warn(
mistralai-mistral-nemo-9330-v13-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:469: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
mistralai-mistral-nemo-9330-v13-mkmlizer: warnings.warn(
mistralai-mistral-nemo-9330-v13-mkmlizer:
Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]
Downloading shards: 50%|█████ | 1/2 [00:05<00:05, 5.46s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 3.86s/it]
Downloading shards: 100%|██████████| 2/2 [00:08<00:00, 4.10s/it]
mistralai-mistral-nemo-9330-v13-mkmlizer:
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 2.48it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 4.09it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.72it/s]
mistralai-mistral-nemo-9330-v13-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
mistralai-mistral-nemo-9330-v13-mkmlizer: Saving duration: 1.323s
mistralai-mistral-nemo-9330-v13-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 12.885s
mistralai-mistral-nemo-9330-v13-mkmlizer: creating bucket guanaco-reward-models
mistralai-mistral-nemo-9330-v13-mkmlizer: Bucket 's3://guanaco-reward-models/' created
mistralai-mistral-nemo-9330-v13-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/config.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/special_tokens_map.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/tokenizer_config.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/merges.txt
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/vocab.json
mistralai-mistral-nemo-9330-v13-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/mistralai-mistral-nemo-9330-v13_reward/tokenizer.json
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Job mistralai-mistral-nemo-9330-v13-mkmlizer completed after 177.54s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v13-mkmlizer
Pipeline stage MKMLizer completed in 178.49s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.11s
Running pipeline stage ISVCDeployer
Creating inference service mistralai-mistral-nemo-9330-v13
Waiting for inference service mistralai-mistral-nemo-9330-v13 to be ready
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_1884_v1: ('http://neversleep-lumimaid-mist-1884-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission undi95-meta-llama-3-70b_6209_v18: ('http://undi95-meta-llama-3-70b-6209-v18-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:37970->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission neversleep-lumimaid-mist_1893_v1: ('http://neversleep-lumimaid-mist-1893-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_7954_v1: ('http://neversleep-lumimaid-mist-7954-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission neversleep-lumimaid-mist_7954_v1: ('http://neversleep-lumimaid-mist-7954-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Inference service mistralai-mistral-nemo-9330-v13 ready after 60.506492376327515s
Pipeline stage ISVCDeployer completed in 62.12s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.2467095851898193s
Received healthy response to inference request in 0.6460261344909668s
Received healthy response to inference request in 0.35091376304626465s
Received healthy response to inference request in 0.6618633270263672s
Received healthy response to inference request in 0.49150538444519043s
5 requests
0 failed requests
5th percentile: 0.3790320873260498
10th percentile: 0.407150411605835
20th percentile: 0.46338706016540526
30th percentile: 0.5224095344543457
40th percentile: 0.5842178344726563
50th percentile: 0.6460261344909668
60th percentile: 0.6523610115051269
70th percentile: 0.6586958885192871
80th percentile: 0.9788325786590579
90th percentile: 1.6127710819244387
95th percentile: 1.9297403335571286
99th percentile: 2.1833157348632812
mean time: 0.8794036388397217
Pipeline stage StressChecker completed in 5.29s
mistralai-mistral-nemo-_9330_v13 status is now deployed due to DeploymentManager action
mistralai-mistral-nemo-_9330_v13 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of mistralai-mistral-nemo-_9330_v13
Running pipeline stage ISVCDeleter
Checking if service mistralai-mistral-nemo-9330-v13 is running
Tearing down inference service mistralai-mistral-nemo-9330-v13
Service mistralai-mistral-nemo-9330-v13 has been torndown
Pipeline stage ISVCDeleter completed in 4.34s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key mistralai-mistral-nemo-9330-v13/config.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v13/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v13/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v13/tokenizer.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v13/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key mistralai-mistral-nemo-9330-v13_reward/config.json from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/merges.txt from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/reward.tensors from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key mistralai-mistral-nemo-9330-v13_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 7.04s
mistralai-mistral-nemo-_9330_v13 status is now torndown due to DeploymentManager action