Running pipeline stage MKMLizer
Starting job with name neversleep-noromaid-v0-8068-v101-mkmlizer
Waiting for job on neversleep-noromaid-v0-8068-v101-mkmlizer to finish
neversleep-noromaid-v0-8068-v101-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ _____ __ __ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ /___/ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ Version: 0.8.14 ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ https://mk1.ai ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ The license key for the current software has been verified as ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ belonging to: ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ Chai Research Corp. ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ Expiration: 2024-07-15 23:59:59 ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ║ ║
neversleep-noromaid-v0-8068-v101-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
neversleep-noromaid-v0-8068-v101-mkmlizer: Downloaded to shared memory in 117.363s
neversleep-noromaid-v0-8068-v101-mkmlizer: quantizing model to /dev/shm/model_cache
neversleep-noromaid-v0-8068-v101-mkmlizer: Saving flywheel model at /dev/shm/model_cache
neversleep-noromaid-v0-8068-v101-mkmlizer: quantized model in 139.340s
neversleep-noromaid-v0-8068-v101-mkmlizer: Processed model NeverSleep/Noromaid-v0.1-mixtral-8x7b-Instruct-v3 in 256.703s
neversleep-noromaid-v0-8068-v101-mkmlizer: creating bucket guanaco-mkml-models
neversleep-noromaid-v0-8068-v101-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
neversleep-noromaid-v0-8068-v101-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/config.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/tokenizer_config.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/tokenizer.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/tokenizer.model s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/tokenizer.model
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/special_tokens_map.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/flywheel_model.3.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/flywheel_model.3.safetensors
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/flywheel_model.1.safetensors
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/flywheel_model.2.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/flywheel_model.2.safetensors
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/neversleep-noromaid-v0-8068-v101/flywheel_model.0.safetensors
neversleep-noromaid-v0-8068-v101-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
neversleep-noromaid-v0-8068-v101-mkmlizer:
Loading 0: 0%| | 0/995 [00:00<?, ?it/s]
Loading 0: 5%|▌ | 52/995 [00:01<00:21, 43.28it/s]
Loading 0: 11%|█ | 107/995 [00:02<00:19, 45.40it/s]
Loading 0: 16%|█▋ | 162/995 [00:03<00:17, 46.39it/s]
Loading 0: 21%|██ | 210/995 [00:11<00:57, 13.74it/s]
Loading 0: 27%|██▋ | 265/995 [00:12<00:38, 18.93it/s]
Loading 0: 28%|██▊ | 278/995 [00:29<02:19, 5.15it/s]
Loading 0: 32%|███▏ | 320/995 [00:30<01:33, 7.22it/s]
Loading 0: 37%|███▋ | 368/995 [00:31<01:01, 10.25it/s]
Loading 0: 43%|████▎ | 423/995 [00:33<00:39, 14.43it/s]
Loading 0: 48%|████▊ | 478/995 [00:34<00:27, 19.09it/s]
Loading 0: 53%|█████▎ | 526/995 [00:35<00:20, 22.92it/s]
Loading 0: 57%|█████▋ | 564/995 [00:52<00:18, 22.92it/s]
Loading 0: 57%|█████▋ | 565/995 [00:52<01:02, 6.86it/s]
Loading 0: 58%|█████▊ | 581/995 [00:53<00:57, 7.24it/s]
Loading 0: 64%|██████▍ | 636/995 [00:54<00:33, 10.81it/s]
Loading 0: 69%|██████▉ | 691/995 [00:56<00:20, 15.05it/s]
Loading 0: 74%|███████▍ | 739/995 [00:57<00:13, 18.85it/s]
Loading 0: 80%|███████▉ | 794/995 [00:58<00:08, 23.74it/s]
Loading 0: 85%|████████▍ | 845/995 [01:14<00:06, 23.74it/s]
Loading 0: 85%|████████▌ | 846/995 [01:14<00:19, 7.81it/s]
Loading 0: 85%|████████▌ | 849/995 [01:19<00:23, 6.14it/s]
Loading 0: 90%|█████████ | 897/995 [01:20<00:11, 8.90it/s]
Loading 0: 96%|█████████▌| 952/995 [01:21<00:03, 12.77it/s]
Loading 0: 100%|██████████| 995/995 [01:22<00:00, 15.67it/s]
/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:919: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v101-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v101-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
neversleep-noromaid-v0-8068-v101-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v101-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:769: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v101-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v101-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
neversleep-noromaid-v0-8068-v101-mkmlizer: warnings.warn(
neversleep-noromaid-v0-8068-v101-mkmlizer:
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:01<00:01, 1.10s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00, 1.71it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00, 1.51it/s]
neversleep-noromaid-v0-8068-v101-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
neversleep-noromaid-v0-8068-v101-mkmlizer: Saving duration: 1.708s
neversleep-noromaid-v0-8068-v101-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 13.087s
neversleep-noromaid-v0-8068-v101-mkmlizer: creating bucket guanaco-reward-models
neversleep-noromaid-v0-8068-v101-mkmlizer: Bucket 's3://guanaco-reward-models/' created
neversleep-noromaid-v0-8068-v101-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/tokenizer_config.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/special_tokens_map.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/config.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/vocab.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/merges.txt
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/tokenizer.json
neversleep-noromaid-v0-8068-v101-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/neversleep-noromaid-v0-8068-v101_reward/reward.tensors
Job neversleep-noromaid-v0-8068-v101-mkmlizer completed after 318.51s with status: succeeded
Stopping job with name neversleep-noromaid-v0-8068-v101-mkmlizer
Pipeline stage MKMLizer completed in 320.02s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.45s
Running pipeline stage ISVCDeployer
Creating inference service neversleep-noromaid-v0-8068-v101
Waiting for inference service neversleep-noromaid-v0-8068-v101 to be ready
Tearing down inference service neversleep-noromaid-v0-8068-v101
%s, retrying in %s seconds...
Creating inference service neversleep-noromaid-v0-8068-v101
Waiting for inference service neversleep-noromaid-v0-8068-v101 to be ready
Tearing down inference service neversleep-noromaid-v0-8068-v101
%s, retrying in %s seconds...
Creating inference service neversleep-noromaid-v0-8068-v101
Waiting for inference service neversleep-noromaid-v0-8068-v101 to be ready
Tearing down inference service neversleep-noromaid-v0-8068-v101
DeploymentError('404\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Date\': \'Tue, 02 Jul 2024 01:14:40 GMT\', \'Content-Type\': \'application/json\', \'Content-Length\': \'310\', \'Connection\': \'keep-alive\', \'Cache-Control\': \'no-cache, private\', \'x-kubernetes-pf-flowschema-uid\': \'95e685fd-6059-4b39-a2ec-40803f81f66b\', \'x-kubernetes-pf-prioritylevel-uid\': \'559b8561-d805-4cc8-9f38-ae50b54f7b32\', \'CF-Cache-Status\': \'DYNAMIC\', \'Set-Cookie\': \'__cf_bm=7Djrcwmk85TpWyjxP47rEfhGC5r0tdM_HHAh4YxrP6g-1719882880-1.0.1.1-izf9bgj4l48Cuw5rsWfg5z2ntsDNAANoRRshjcUSQFg; path=/; expires=Tue, 02-Jul-24 01:44:40 GMT; domain=.coreweave.com; HttpOnly; Secure; SameSite=None\', \'Server\': \'cloudflare\', \'CF-RAY\': \'89cac7c31d6f54d6-MAN\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"inferenceservices.serving.kubeflow.org \\\\"neversleep-noromaid-v0-8068-v101\\\\" not found","reason":"NotFound","details":{"name":"neversleep-noromaid-v0-8068-v101","group":"serving.kubeflow.org","kind":"inferenceservices"},"code":404}\\n\'\nOriginal traceback: \n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 55, in inner\n resp = func(self, *args, **kwargs)\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/dynamic/client.py", line 270, in request\n api_response = self.client.call_api(\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api\n return self.__call_api(resource_path, method,\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api\n response_data = self.request(\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 415, in request\n return self.rest_client.DELETE(url,\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 266, in DELETE\n return self.request("DELETE", url,\n\n File "/home/alex/.local/lib/python3.8/site-packages/kubernetes/client/rest.py", line 234, in request\n raise ApiException(http_resp=r)\n')
neversleep-noromaid-v0_8068_v101 status is now failed due to DeploymentManager action
admin requested tearing down of neversleep-noromaid-v0_8068_v101
Running pipeline stage ISVCDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage ISVCDeleter completed in 0.13s
Running pipeline stage MKMLModelDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLModelDeleter completed in 0.05s
neversleep-noromaid-v0_8068_v101 status is now torndown due to DeploymentManager action