nousresearch-meta-llama_4941

submission_id: nousresearch-meta-llama_4941_v61
developer_uid: alexdaoud
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}
model_name: nousresearch-meta-llama_4941_v61
model_repo: NousResearch/Meta-Llama-3-8B-Instruct
reward_formatter: {'bot_template': '{bot_name}: {message}\n', 'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'response_template': '{bot_name}:', 'truncate_by_message': False, 'user_template': '{user_name}: {message}\n'}
reward_repo: ChaiML/gpt2_xl_pairwise_89m_step_347634
status: torndown
timestamp: 2024-06-27T20:20:21+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name nousresearch-meta-llama-4941-v61-mkmlizer
Waiting for job on nousresearch-meta-llama-4941-v61-mkmlizer to finish
nousresearch-meta-llama-4941-v61-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4941-v61-mkmlizer: ║     _____            __           __                                ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║       /___/                                                         ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  Version: 0.8.14                                                    ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  https://mk1.ai                                                     ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  The license key for the current software has been verified as      ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  belonging to:                                                      ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  Chai Research Corp.                                                ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║  Expiration: 2024-07-15 23:59:59                                    ║
nousresearch-meta-llama-4941-v61-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4941-v61-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4941-v61-mkmlizer: /opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:131: FutureWarning: 'list_files_info' (from 'huggingface_hub.hf_api') is deprecated and will be removed from version '0.23'. Use `list_repo_tree` and `get_paths_info` instead.
nousresearch-meta-llama-4941-v61-mkmlizer:   warnings.warn(warning_message, FutureWarning)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
nousresearch-meta-llama-4941-v61-mkmlizer: Downloaded to shared memory in 46.045s
nousresearch-meta-llama-4941-v61-mkmlizer: quantizing model to /dev/shm/model_cache
nousresearch-meta-llama-4941-v61-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission blend_dukel_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_furit_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_furit_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
nousresearch-meta-llama-4941-v61-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:02, 119.04it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:02, 107.05it/s]
Loading 0:  13%|█▎        | 39/291 [00:00<00:02, 119.85it/s]
Loading 0:  18%|█▊        | 52/291 [00:00<00:02, 115.59it/s]
Loading 0:  23%|██▎       | 66/291 [00:00<00:01, 120.19it/s]
Loading 0:  27%|██▋       | 79/291 [00:00<00:01, 116.88it/s]
Loading 0:  31%|███▏      | 91/291 [00:01<00:02, 67.74it/s] 
Loading 0:  35%|███▌      | 103/291 [00:01<00:02, 75.59it/s]
Loading 0:  40%|███▉      | 116/291 [00:01<00:02, 86.56it/s]
Loading 0:  44%|████▍     | 129/291 [00:01<00:01, 96.62it/s]
Loading 0:  48%|████▊     | 141/291 [00:01<00:01, 98.18it/s]
Loading 0:  54%|█████▎    | 156/291 [00:01<00:01, 108.66it/s]
Loading 0:  58%|█████▊    | 168/291 [00:01<00:01, 106.74it/s]
Loading 0:  62%|██████▏   | 181/291 [00:01<00:00, 110.42it/s]
Loading 0:  66%|██████▋   | 193/291 [00:02<00:01, 67.99it/s] 
Loading 0:  70%|███████   | 205/291 [00:02<00:01, 77.66it/s]
Loading 0:  75%|███████▌  | 219/291 [00:02<00:00, 89.57it/s]
Loading 0:  79%|███████▉  | 230/291 [00:02<00:00, 91.32it/s]
Loading 0:  85%|████████▍ | 246/291 [00:02<00:00, 104.96it/s]
Loading 0:  89%|████████▊ | 258/291 [00:02<00:00, 103.71it/s]
Loading 0:  94%|█████████▍| 273/291 [00:02<00:00, 112.48it/s]
Loading 0:  98%|█████████▊| 285/291 [00:02<00:00, 111.22it/s]
                                                             
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
nousresearch-meta-llama-4941-v61-mkmlizer: quantized model in 24.361s
nousresearch-meta-llama-4941-v61-mkmlizer: Processed model NousResearch/Meta-Llama-3-8B-Instruct in 72.944s
nousresearch-meta-llama-4941-v61-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4941-v61-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4941-v61-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61
nousresearch-meta-llama-4941-v61-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61/config.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61/tokenizer_config.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61/special_tokens_map.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61/tokenizer.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61/flywheel_model.0.safetensors
nousresearch-meta-llama-4941-v61-mkmlizer: loading reward model from ChaiML/gpt2_xl_pairwise_89m_step_347634
nousresearch-meta-llama-4941-v61-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py:913: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v61-mkmlizer:   warnings.warn(
nousresearch-meta-llama-4941-v61-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:757: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v61-mkmlizer:   warnings.warn(
nousresearch-meta-llama-4941-v61-mkmlizer: /opt/conda/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:468: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
nousresearch-meta-llama-4941-v61-mkmlizer:   warnings.warn(
nousresearch-meta-llama-4941-v61-mkmlizer: 
Downloading shards:   0%|          | 0/2 [00:00<?, ?it/s]
Downloading shards:  50%|█████     | 1/2 [00:10<00:10, 10.41s/it]
Downloading shards: 100%|██████████| 2/2 [00:20<00:00, 10.49s/it]
Downloading shards: 100%|██████████| 2/2 [00:20<00:00, 10.48s/it]
nousresearch-meta-llama-4941-v61-mkmlizer: 
Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.72it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.90it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.63it/s]
nousresearch-meta-llama-4941-v61-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
nousresearch-meta-llama-4941-v61-mkmlizer: Saving duration: 1.620s
nousresearch-meta-llama-4941-v61-mkmlizer: Processed model ChaiML/gpt2_xl_pairwise_89m_step_347634 in 25.156s
nousresearch-meta-llama-4941-v61-mkmlizer: creating bucket guanaco-reward-models
nousresearch-meta-llama-4941-v61-mkmlizer: Bucket 's3://guanaco-reward-models/' created
nousresearch-meta-llama-4941-v61-mkmlizer: uploading /tmp/reward_cache to s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/config.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/special_tokens_map.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/special_tokens_map.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/tokenizer_config.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/tokenizer_config.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/vocab.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/vocab.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/tokenizer.json s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/tokenizer.json
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/merges.txt s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/merges.txt
nousresearch-meta-llama-4941-v61-mkmlizer: cp /tmp/reward_cache/reward.tensors s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward/reward.tensors
Job nousresearch-meta-llama-4941-v61-mkmlizer completed after 124.84s with status: succeeded
Stopping job with name nousresearch-meta-llama-4941-v61-mkmlizer
Pipeline stage MKMLizer completed in 125.72s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.12s
Running pipeline stage ISVCDeployer
Creating inference service nousresearch-meta-llama-4941-v61
Waiting for inference service nousresearch-meta-llama-4941-v61 to be ready
Failed to get response for submission blend_jokus_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_furit_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_dukel_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_jokus_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_dukel_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_dukel_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_midur_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission blend_subet_2024-06-27: ('http://mistralai-mixtral-8x7b-3473-v47-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service nousresearch-meta-llama-4941-v61
%s, retrying in %s seconds...
Creating inference service nousresearch-meta-llama-4941-v61
Waiting for inference service nousresearch-meta-llama-4941-v61 to be ready
Tearing down inference service nousresearch-meta-llama-4941-v61
%s, retrying in %s seconds...
Creating inference service nousresearch-meta-llama-4941-v61
Waiting for inference service nousresearch-meta-llama-4941-v61 to be ready
Tearing down inference service nousresearch-meta-llama-4941-v61
DeploymentError('Timeout to start the InferenceService nousresearch-meta-llama-4941-v61.                                The InferenceService is as following: {\'apiVersion\': \'serving.kubeflow.org/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'kpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'2\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'networking.knative.dev/ingress-class\': \'kourier.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2024-06-27T20:42:52Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'kourier.ingress.networking.knative.dev\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kubeflow.org/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {\'.\': {}, \'f:nodeSelectorTerms\': {}}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2024-06-27T20:42:47Z\'}, {\'apiVersion\': \'serving.kubeflow.org/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {}}, \'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2024-06-27T20:42:58Z\'}], \'name\': \'nousresearch-meta-llama-4941-v61\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'60060969435\', \'uid\': \'91ebe87f-bce2-445b-bed0-120f5496bffa\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'topology.kubernetes.io/region\', \'operator\': \'In\', \'values\': [\'ORD1\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}]}}}, \'containerConcurrency\': 4, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'512\'}, {\'name\': \'BEST_OF\', \'value\': \'16\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/nousresearch-meta-llama-4941-v61\'}, {\'name\': \'REWARD_TENSORIZER_URI\', \'value\': \'s3://guanaco-reward-models/nousresearch-meta-llama-4941-v61_reward\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:min_p_support_v2\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kfserving-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'12Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'12Gi\', \'nvidia.com/gpu\': \'1\'}}}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 500, \'minReplicas\': 0, \'timeout\': 20}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'nousresearch-meta-llama-4941-v61-predictor-default-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2024-06-27T20:42:58Z\', \'message\': \'Configuration "nousresearch-meta-llama-4941-v61-predictor-default" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2024-06-27T20:42:58Z\', \'message\': \'Configuration "nousresearch-meta-llama-4941-v61-predictor-default" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'status\': \'Unknown\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2024-06-27T20:42:58Z\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2024-06-27T20:42:58Z\', \'message\': \'Configuration "nousresearch-meta-llama-4941-v61-predictor-default" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'status\': \'Unknown\', \'type\': \'Ready\'}]}}')
nousresearch-meta-llama_4941_v61 status is now failed due to DeploymentManager action
admin requested tearing down of nousresearch-meta-llama_4941_v61
Running pipeline stage ISVCDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage ISVCDeleter completed in 0.09s
Running pipeline stage MKMLModelDeleter
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLModelDeleter completed in 0.08s
nousresearch-meta-llama_4941_v61 status is now torndown due to DeploymentManager action