submission_id: mistralai-mistral-nemo-_9330_v60
developer_uid: albert_chai
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.1, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
model_name: nemo-dpo-bo8x1024_T11
model_repo: mistralai/Mistral-Nemo-Instruct-2407
status: torndown
timestamp: 2024-08-24T05:55:24+00:00
Resubmit model
Running pipeline stage MKMLizer
Starting job with name mistralai-mistral-nemo-9330-v60-mkmlizer
Waiting for job on mistralai-mistral-nemo-9330-v60-mkmlizer to finish
mistralai-mistral-nemo-9330-v60-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
mistralai-mistral-nemo-9330-v60-mkmlizer: ║     _____            __           __                                ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║       /___/                                                         ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  Version: 0.10.1                                                    ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  https://mk1.ai                                                     ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  The license key for the current software has been verified as      ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  belonging to:                                                      ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  Chai Research Corp.                                                ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ║                                                                     ║
mistralai-mistral-nemo-9330-v60-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
mistralai-mistral-nemo-9330-v60-mkmlizer: Downloaded to shared memory in 46.505s
mistralai-mistral-nemo-9330-v60-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpkto9vo4u, device:0
mistralai-mistral-nemo-9330-v60-mkmlizer: Saving flywheel model at /dev/shm/model_cache
mistralai-mistral-nemo-9330-v60-mkmlizer: quantized model in 38.047s
mistralai-mistral-nemo-9330-v60-mkmlizer: Processed model mistralai/Mistral-Nemo-Instruct-2407 in 84.553s
mistralai-mistral-nemo-9330-v60-mkmlizer: creating bucket guanaco-mkml-models
mistralai-mistral-nemo-9330-v60-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
mistralai-mistral-nemo-9330-v60-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60
mistralai-mistral-nemo-9330-v60-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60/config.json
mistralai-mistral-nemo-9330-v60-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60/special_tokens_map.json
mistralai-mistral-nemo-9330-v60-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60/tokenizer_config.json
mistralai-mistral-nemo-9330-v60-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60/tokenizer.json
mistralai-mistral-nemo-9330-v60-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60/flywheel_model.0.safetensors
mistralai-mistral-nemo-9330-v60-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.79it/s]
Loading 0:   3%|▎         | 12/363 [00:00<00:07, 48.53it/s]
Loading 0:   5%|▍         | 18/363 [00:00<00:07, 48.61it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 38.92it/s]
Loading 0:   8%|▊         | 30/363 [00:00<00:07, 43.48it/s]
Loading 0:  10%|▉         | 35/363 [00:00<00:07, 42.72it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:07, 42.50it/s]
Loading 0:  12%|█▏        | 45/363 [00:01<00:07, 44.23it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:08, 35.80it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:07, 40.88it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:10, 29.99it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 30.04it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:08, 35.63it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 36.68it/s]
Loading 0:  22%|██▏       | 81/363 [00:02<00:07, 37.08it/s]
Loading 0:  23%|██▎       | 85/363 [00:02<00:07, 37.00it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:07, 37.52it/s]
Loading 0:  26%|██▌       | 93/363 [00:02<00:07, 36.41it/s]
Loading 0:  27%|██▋       | 98/363 [00:02<00:06, 38.74it/s]
Loading 0:  28%|██▊       | 102/363 [00:02<00:07, 36.88it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:06, 36.97it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:06, 40.53it/s]
Loading 0:  32%|███▏      | 117/363 [00:03<00:06, 38.29it/s]
Loading 0:  33%|███▎      | 121/363 [00:03<00:06, 38.24it/s]
Loading 0:  34%|███▍      | 125/363 [00:03<00:06, 37.93it/s]
Loading 0:  36%|███▌      | 129/363 [00:03<00:06, 36.31it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:05, 38.18it/s]
Loading 0:  38%|███▊      | 138/363 [00:03<00:06, 36.07it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:08, 25.22it/s]
Loading 0:  40%|███▉      | 145/363 [00:04<00:08, 24.97it/s]
Loading 0:  41%|████      | 149/363 [00:04<00:08, 25.53it/s]
Loading 0:  43%|████▎     | 156/363 [00:04<00:06, 33.57it/s]
Loading 0:  44%|████▍     | 160/363 [00:04<00:06, 33.75it/s]
Loading 0:  45%|████▌     | 165/363 [00:04<00:05, 35.97it/s]
Loading 0:  47%|████▋     | 169/363 [00:04<00:05, 34.84it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:05, 37.30it/s]
Loading 0:  49%|████▉     | 178/363 [00:04<00:05, 36.17it/s]
Loading 0:  50%|█████     | 183/363 [00:05<00:04, 38.20it/s]
Loading 0:  52%|█████▏    | 187/363 [00:05<00:04, 36.62it/s]
Loading 0:  53%|█████▎    | 192/363 [00:05<00:04, 39.13it/s]
Loading 0:  54%|█████▍    | 196/363 [00:05<00:04, 37.63it/s]
Loading 0:  55%|█████▌    | 201/363 [00:05<00:04, 39.39it/s]
Loading 0:  56%|█████▋    | 205/363 [00:05<00:04, 37.66it/s]
Loading 0:  58%|█████▊    | 210/363 [00:05<00:03, 39.13it/s]
Loading 0:  59%|█████▉    | 214/363 [00:05<00:03, 37.58it/s]
Loading 0:  60%|██████    | 218/363 [00:05<00:03, 36.97it/s]
Loading 0:  61%|██████▏   | 223/363 [00:06<00:05, 27.21it/s]
Loading 0:  63%|██████▎   | 227/363 [00:06<00:04, 28.81it/s]
Loading 0:  64%|██████▎   | 231/363 [00:06<00:04, 28.51it/s]
Loading 0:  65%|██████▍   | 235/363 [00:06<00:04, 30.90it/s]
Loading 0:  66%|██████▌   | 239/363 [00:06<00:04, 29.97it/s]
Loading 0:  68%|██████▊   | 246/363 [00:06<00:03, 37.47it/s]
Loading 0:  69%|██████▉   | 250/363 [00:06<00:03, 36.46it/s]
Loading 0:  70%|███████   | 255/363 [00:07<00:02, 38.43it/s]
Loading 0:  71%|███████▏  | 259/363 [00:07<00:02, 37.27it/s]
Loading 0:  73%|███████▎  | 264/363 [00:07<00:02, 39.45it/s]
Loading 0:  74%|███████▍  | 269/363 [00:07<00:02, 39.24it/s]
Loading 0:  75%|███████▌  | 273/363 [00:07<00:02, 38.57it/s]
Loading 0:  76%|███████▋  | 277/363 [00:07<00:02, 37.18it/s]
Loading 0:  78%|███████▊  | 282/363 [00:07<00:02, 39.57it/s]
Loading 0:  79%|███████▉  | 286/363 [00:07<00:02, 37.37it/s]
Loading 0:  80%|████████  | 291/363 [00:08<00:01, 38.73it/s]
Loading 0:  81%|████████▏ | 295/363 [00:08<00:01, 37.41it/s]
Loading 0:  82%|████████▏ | 299/363 [00:08<00:01, 36.94it/s]
Loading 0:  84%|████████▎ | 304/363 [00:15<00:29,  2.02it/s]
Loading 0:  85%|████████▍ | 307/363 [00:15<00:22,  2.54it/s]
Loading 0:  86%|████████▌ | 312/363 [00:15<00:13,  3.76it/s]
Loading 0:  88%|████████▊ | 320/363 [00:15<00:06,  6.48it/s]
Loading 0:  90%|████████▉ | 326/363 [00:15<00:04,  8.90it/s]
Loading 0:  91%|█████████ | 331/363 [00:16<00:02, 11.42it/s]
Loading 0:  93%|█████████▎| 338/363 [00:16<00:01, 16.18it/s]
Loading 0:  95%|█████████▍| 344/363 [00:16<00:00, 19.92it/s]
Loading 0:  96%|█████████▌| 349/363 [00:16<00:00, 23.29it/s]
Loading 0:  98%|█████████▊| 356/363 [00:16<00:00, 29.97it/s]
Loading 0: 100%|█████████▉| 362/363 [00:16<00:00, 32.37it/s]
                                                            

Job mistralai-mistral-nemo-9330-v60-mkmlizer completed after 136.48s with status: succeeded
Stopping job with name mistralai-mistral-nemo-9330-v60-mkmlizer
Pipeline stage MKMLizer completed in 137.31s
Running pipeline stage MKMLKubeTemplater
Pipeline stage MKMLKubeTemplater completed in 0.39s
Running pipeline stage ISVCDeployer
Creating inference service mistralai-mistral-nemo-9330-v60
Waiting for inference service mistralai-mistral-nemo-9330-v60 to be ready
Tearing down inference service mistralai-mistral-nemo-9330-v60
%s, retrying in %s seconds...
Creating inference service mistralai-mistral-nemo-9330-v60
Ignoring service mistralai-mistral-nemo-9330-v60 already deployed
Waiting for inference service mistralai-mistral-nemo-9330-v60 to be ready
Tearing down inference service mistralai-mistral-nemo-9330-v60
%s, retrying in %s seconds...
Creating inference service mistralai-mistral-nemo-9330-v60
Waiting for inference service mistralai-mistral-nemo-9330-v60 to be ready
Failed to get response for submission blend_fedek_2024-08-24: ('http://mistralai-mixtral-8x7b-3473-v131-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Failed to get response for submission neversleep-noromaid-v0_8068_v133: ('http://neversleep-noromaid-v0-8068-v133-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:55912->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission chaiml-albert-0823-dpo30_6942_v1: ('http://chaiml-albert-0823-dpo30-6942-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_berib_2024-08-16: ('http://zonemercy-graft-cogent-v-7573-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission blend_remul_2024-08-22: ('http://zonemercy-graft-cogent-v-7573-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission mistralai-mistral-nemo-_9330_v59: ('http://mistralai-mistral-nemo-9330-v59-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-nemo-_9330_v59: ('http://mistralai-mistral-nemo-9330-v59-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission mistralai-mistral-nemo-_9330_v57: ('http://mistralai-mistral-nemo-9330-v57-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission mistralai-mistral-nemo-_9330_v59: ('http://mistralai-mistral-nemo-9330-v59-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission mistralai-mistral-nemo-_9330_v59: ('http://mistralai-mistral-nemo-9330-v59-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission mistralai-mistral-nemo-_9330_v58: ('http://mistralai-mistral-nemo-9330-v58-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission blend_dedat_2024-08-16: ('http://zonemercy-graft-cogent-v-7573-v6-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission chaiml-albert-0823-dpo30_3614_v1: ('http://chaiml-albert-0823-dpo30-3614-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'EOF\n')
Failed to get response for submission chaiml-albert-0823-dpo30_6942_v1: ('http://chaiml-albert-0823-dpo30-6942-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Tearing down inference service mistralai-mistral-nemo-9330-v60
DeploymentError('Timeout to start the InferenceService mistralai-mistral-nemo-9330-v60.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'3700\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2024-08-24T06:08:01Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2024-08-24T06:08:01Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2024-08-24T06:08:01Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2024-08-24T06:18:06Z\'}], \'name\': \'mistralai-mistral-nemo-9330-v60\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'60159061\', \'uid\': \'d681f9ea-ff64-4051-99fa-0c78ab030bc9\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'topology.kubernetes.io/region\', \'operator\': \'In\', \'values\': [\'ORD1\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.1\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/mistralai-mistral-nemo-9330-v60\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:mkml_v0.10.1\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 500, \'minReplicas\': 0, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'mistralai-mistral-nemo-9330-v60-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2024-08-24T06:08:07Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2024-08-24T06:18:06Z\', \'message\': \'Revision "mistralai-mistral-nemo-9330-v60-predictor-00001" failed with message: 0/4529 nodes are available: 1 node(s) had taint {node.coreweave.cloud/reserved: 068123c732583ca97229d877d4556e1e1f4ca50d}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 075ba5f3157f94ecc7e8099d5841259a894bd8e3}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 07b2baa5117bf3dd29052fdc67f601965171b005}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 1a595a58a4eeff882120a1e1b0d1010e09698d99}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 33920511c252dd3a880d5284bd117a96124ca978}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 3a45a883af800a19f23cadb8f85db86d74b5a84f}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 58762d02b028c950f16f692fde0a10956cc6b043}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 6ba4a623b9ed926b4f18ba472b1c7eee1e0e85b5}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 6c7fa72bb0e687df2f2a055b49e2e7687c0dc25e}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 6f8c6fc295765e2b1fc0e66bc7317ef1349a72e7}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 7409618228c514a32d04288f2195480ae03ba023}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 94e0405b286360fb1b34a7fa704f0d5951462ef5}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: afdef22bd9179253164b37cabb64f3e40675acb2}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: d9a4805717baccf21a30a19f47cb010767a4f67b}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: eb044b4b7b054c04154357b3cb76c9f802ff9c2d}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: fb5beb3113ffb96ddf1b1bcb694dbc0afa65bfc6}, that the pod didn\\\'t tolerate, 1092 node(s) didn\\\'t match Pod\\\'s node affinity, 1479 node(s) had taint {is_cpu_compute: true}, that the pod didn\\\'t tolerate, 150 node(s) had taint {node.coreweave.cloud/reserved: 7c04e42959dba3824ee072e2d1083485eab9fa7a}, that the pod didn\\\'t tolerate, 183 Insufficient nvidia.com/gpu, 19 node(s) had taint {node.coreweave.cloud/reserved: 04688a0d6a3e07a42ec8266db6b2253d1faf71fc}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 007b3cc3da717eac69ab0559a137e7f3b606c461}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 08c1468d7e24e3b0938e976db9cc5cd234ce0b06}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 2f7a0a7b992bfe7d029beeb48b22e6e66048044d}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 3b3525672b1d07bb0e82ed6aa93fb7b63151b984}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 41cc1990ce793067e6e5eb8c7630a6b2866eb4ed}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: a524fb38e86e111d860e8720ba166a0adade52a3}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: ab6668f5db9960265cd5619120217d08181b955e}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: f7a44a72a965b42d74feced14310079a26b3230d}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved_for_prometheus: true}, that the pod didn\\\'t tolerate, 3 node(s) had taint {node.coreweave.cloud/reserved: 960b232b8c784c940161614146fd165b5bd0be0a}, that the pod didn\\\'t tolerate, 35 node(s) had taint {node.coreweave.cloud/reserved: 86269dd9a6b0a4932485d7d6e07571590bb2cb05}, that the pod didn\\\'t tolerate, 4 node(s) had taint {node.coreweave.cloud/reserved: 1d765a2e3fdfad1ed54881e9e7d00732704d2290}, that the pod didn\\\'t tolerate, 429 node(s) were unschedulable, 43 node(s) had taint {node.coreweave.cloud/reserved: mimir}, that the pod didn\\\'t tolerate, 5 node(s) had taint {node.coreweave.cloud/reserved: 3c12ff0391e67bef9ec6b130d6f40d5eed238ade}, that the pod didn\\\'t tolerate, 5 node(s) had taint {node.coreweave.cloud/reserved: 54552d502f78950a4ba54f20164f0a8fa5e0ff52}, that the pod didn\\\'t tolerate, 5 node(s) had taint {node.coreweave.cloud/reserved: 9dedfac275b2b916c916a8f5b21f7e7efa369b15}, that the pod didn\\\'t tolerate, 50 node(s) had taint {node.coreweave.cloud/reserved: 7bb0910539250d61afdb118ca00f35954c4c65a3}, that the pod didn\\\'t tolerate, 6 node(s) had taint {node.coreweave.cloud/troubleshoot: augment}, that the pod didn\\\'t tolerate, 60 node(s) had taint {node.coreweave.cloud/reserved: 54606afe8f3a00542920fdd9dc3130149c8794db}, that the pod didn\\\'t tolerate, 7 node(s) had taint {node.coreweave.cloud/reserved: b528168f007b9294330e209e0ff29d083c6363c1}, that the pod didn\\\'t tolerate, 78 node(s) had taint {node.coreweave.cloud/reserved: 5a7af8c200facc8160037a2d7a2d134b61e5f9a2}, that the pod didn\\\'t tolerate, 8 node(s) had taint {node.coreweave.cloud/reservation-policy: local}, that the pod didn\\\'t tolerate, 824 node(s) had taint {node.coreweave.cloud/hypervisor: true}, that the pod didn\\\'t tolerate, 9 node(s) had taint {node.coreweave.cloud/reserved: c251f4dd13d1a2d79cc745b5749552f06e3fa3cc}, that the pod didn\\\'t tolerate..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2024-08-24T06:08:07Z\', \'message\': \'Configuration "mistralai-mistral-nemo-9330-v60-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2024-08-24T06:08:07Z\', \'message\': \'Configuration "mistralai-mistral-nemo-9330-v60-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2024-08-24T06:08:07Z\', \'message\': \'Configuration "mistralai-mistral-nemo-9330-v60-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2024-08-24T06:08:07Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
mistralai-mistral-nemo-_9330_v60 status is now failed due to DeploymentManager action
admin requested tearing down of mistralai-mistral-nemo-_9330_v60
Running pipeline stage ISVCDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage ISVCDeleter completed in 0.19s
Running pipeline stage MKMLModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Failed to get response for submission mistralai-mistral-nemo-_9330_v58: ('http://mistralai-mistral-nemo-9330-v58-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Deleting key mistralai-mistral-nemo-9330-v60/config.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v60/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v60/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v60/tokenizer.json from bucket guanaco-mkml-models
Deleting key mistralai-mistral-nemo-9330-v60/tokenizer_config.json from bucket guanaco-mkml-models
Pipeline stage MKMLModelDeleter completed in 2.77s
mistralai-mistral-nemo-_9330_v60 status is now torndown due to DeploymentManager action

Usage Metrics

Latency Metrics