Running pipeline stage MKMLizer
Starting job with name chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer
Waiting for job on chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer to finish
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ _____ __ __ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ /___/ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ Version: 0.8.14 ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ https://mk1.ai ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ The license key for the current software has been verified as ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ belonging to: ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ Chai Research Corp. ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ║ ║
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Downloaded to shared memory in 25.343s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: quantizing model to /dev/shm/model_cache
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer:
Loading 0: 0%| | 0/291 [00:00<?, ?it/s]
Loading 0: 1%| | 2/291 [00:04<11:20, 2.35s/it]
Loading 0: 4%|▍ | 13/291 [00:04<01:16, 3.65it/s]
Loading 0: 8%|▊ | 23/291 [00:04<00:35, 7.53it/s]
Loading 0: 11%|█▏ | 33/291 [00:05<00:20, 12.54it/s]
Loading 0: 17%|█▋ | 49/291 [00:05<00:10, 22.87it/s]
Loading 0: 21%|██ | 60/291 [00:05<00:09, 23.83it/s]
Loading 0: 26%|██▌ | 76/291 [00:05<00:06, 35.09it/s]
Loading 0: 30%|██▉ | 87/291 [00:05<00:04, 42.70it/s]
Loading 0: 35%|███▌ | 103/291 [00:05<00:03, 57.79it/s]
Loading 0: 39%|███▉ | 114/291 [00:06<00:02, 64.41it/s]
Loading 0: 45%|████▍ | 130/291 [00:06<00:02, 79.96it/s]
Loading 0: 49%|████▉ | 142/291 [00:06<00:01, 84.40it/s]
Loading 0: 54%|█████▍ | 157/291 [00:06<00:01, 95.79it/s]
Loading 0: 58%|█████▊ | 169/291 [00:06<00:01, 62.43it/s]
Loading 0: 63%|██████▎ | 184/291 [00:06<00:01, 76.39it/s]
Loading 0: 67%|██████▋ | 195/291 [00:06<00:01, 80.16it/s]
Loading 0: 73%|███████▎ | 211/291 [00:07<00:00, 95.11it/s]
Loading 0: 77%|███████▋ | 223/291 [00:07<00:00, 99.36it/s]
Loading 0: 82%|████████▏ | 239/291 [00:07<00:00, 109.75it/s]
Loading 0: 87%|████████▋ | 253/291 [00:07<00:00, 116.55it/s]
Loading 0: 91%|█████████▏| 266/291 [00:07<00:00, 70.80it/s]
Loading 0: 95%|█████████▍| 276/291 [00:07<00:00, 75.50it/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: quantized model in 24.186s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Processed model ChaiML/sao10k-l3-rp-v3-3 in 49.530s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: creating bucket guanaco-mkml-models
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/config.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/special_tokens_map.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/tokenizer_config.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/tokenizer.json
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-sao10k-l3-rp-v3-3-v6/flywheel_model.0.safetensors
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: loading reward model from Jellywibble/CHAI_alignment_reward_model
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:919: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:769: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: warnings.warn(
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Saving duration: 0.150s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: Processed model Jellywibble/CHAI_alignment_reward_model in 6.222s
chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer: creating bucket guanaco-reward-models
Job chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer completed after 134.07s with status: succeeded
Stopping job with name chaiml-sao10k-l3-rp-v3-3-v6-mkmlizer
Pipeline stage MKMLizer completed in 136.11s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.42s
Running pipeline stage ISVCDeployer
Creating inference service chaiml-sao10k-l3-rp-v3-3-v6
Waiting for inference service chaiml-sao10k-l3-rp-v3-3-v6 to be ready
Inference service chaiml-sao10k-l3-rp-v3-3-v6 ready after 51.05707311630249s
Pipeline stage ISVCDeployer completed in 53.34s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.3435118198394775s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.618872880935669s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.6351609230041504s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.694105863571167s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5980210304260254s
5 requests
0 failed requests
5th percentile: 1.602191400527954
10th percentile: 1.6063617706298827
20th percentile: 1.6147025108337403
30th percentile: 1.6221304893493653
40th percentile: 1.6286457061767579
50th percentile: 1.6351609230041504
60th percentile: 1.658738899230957
70th percentile: 1.6823168754577638
80th percentile: 1.8239870548248291
90th percentile: 2.0837494373321532
95th percentile: 2.213630628585815
99th percentile: 2.317535581588745
mean time: 1.7779345035552978
Pipeline stage StressChecker completed in 12.72s
chaiml-sao10k-l3-rp-v3-3_v6 status is now deployed due to DeploymentManager action
chaiml-sao10k-l3-rp-v3-3_v6 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-sao10k-l3-rp-v3-3_v6
Running pipeline stage ISVCDeleter
Checking if service chaiml-sao10k-l3-rp-v3-3-v6 is running
Tearing down inference service chaiml-sao10k-l3-rp-v3-3-v6
Service chaiml-sao10k-l3-rp-v3-3-v6 has been torndown
Pipeline stage ISVCDeleter completed in 6.27s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/config.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/tokenizer.json from bucket guanaco-mkml-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6/tokenizer_config.json from bucket guanaco-mkml-models
Cleaning model data from model cache
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/merges.txt from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/reward.tensors from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/special_tokens_map.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/tokenizer.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/tokenizer_config.json from bucket guanaco-reward-models
Deleting key chaiml-sao10k-l3-rp-v3-3-v6_reward/vocab.json from bucket guanaco-reward-models
Pipeline stage MKMLModelDeleter completed in 5.51s
chaiml-sao10k-l3-rp-v3-3_v6 status is now torndown due to DeploymentManager action