Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rica40325-fliter65kv1-v1-mkmlizer
Waiting for job on rica40325-fliter65kv1-v1-mkmlizer to finish
rica40325-fliter65kv1-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rica40325-fliter65kv1-v1-mkmlizer: ║ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
rica40325-fliter65kv1-v1-mkmlizer: ║ https://mk1.ai ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ The license key for the current software has been verified as ║
rica40325-fliter65kv1-v1-mkmlizer: ║ belonging to: ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Chai Research Corp. ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
rica40325-fliter65kv1-v1-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
rica40325-fliter65kv1-v1-mkmlizer: ║ ║
rica40325-fliter65kv1-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
rica40325-fliter65kv1-v1-mkmlizer: Downloaded to shared memory in 37.856s
rica40325-fliter65kv1-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp9bz4c1nl, device:0
rica40325-fliter65kv1-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
rica40325-fliter65kv1-v1-mkmlizer: quantized model in 30.472s
rica40325-fliter65kv1-v1-mkmlizer: Processed model rica40325/fliter65kv1 in 68.328s
rica40325-fliter65kv1-v1-mkmlizer: creating bucket guanaco-mkml-models
rica40325-fliter65kv1-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rica40325-fliter65kv1-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rica40325-fliter65kv1-v1
rica40325-fliter65kv1-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rica40325-fliter65kv1-v1/special_tokens_map.json
rica40325-fliter65kv1-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rica40325-fliter65kv1-v1/config.json
rica40325-fliter65kv1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rica40325-fliter65kv1-v1/tokenizer_config.json
rica40325-fliter65kv1-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rica40325-fliter65kv1-v1/tokenizer.json
rica40325-fliter65kv1-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:12, 29.56it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:07, 47.12it/s]
Loading 0: 5%|▍ | 18/363 [00:00<00:07, 46.76it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:09, 36.81it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 46.02it/s]
Loading 0: 10%|▉ | 36/363 [00:00<00:07, 46.39it/s]
Loading 0: 11%|█▏ | 41/363 [00:01<00:08, 38.15it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:06, 46.29it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:07, 42.89it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:09, 32.28it/s]
Loading 0: 18%|█▊ | 65/363 [00:01<00:09, 31.44it/s]
Loading 0: 20%|█▉ | 71/363 [00:01<00:07, 37.04it/s]
Loading 0: 21%|██ | 76/363 [00:01<00:07, 38.19it/s]
Loading 0: 22%|██▏ | 81/363 [00:02<00:07, 40.22it/s]
Loading 0: 24%|██▍ | 87/363 [00:02<00:06, 39.99it/s]
Loading 0: 25%|██▌ | 92/363 [00:02<00:06, 39.90it/s]
Loading 0: 27%|██▋ | 99/363 [00:02<00:05, 44.97it/s]
Loading 0: 29%|██▉ | 105/363 [00:02<00:05, 43.03it/s]
Loading 0: 31%|███ | 112/363 [00:02<00:05, 46.76it/s]
Loading 0: 32%|███▏ | 117/363 [00:02<00:05, 43.09it/s]
Loading 0: 34%|███▎ | 122/363 [00:02<00:05, 43.30it/s]
Loading 0: 35%|███▍ | 127/363 [00:03<00:06, 36.96it/s]
Loading 0: 37%|███▋ | 134/363 [00:03<00:05, 43.68it/s]
Loading 0: 38%|███▊ | 139/363 [00:03<00:05, 42.63it/s]
Loading 0: 40%|███▉ | 144/363 [00:03<00:08, 26.98it/s]
Loading 0: 41%|████ | 149/363 [00:03<00:07, 28.96it/s]
Loading 0: 43%|████▎ | 156/363 [00:04<00:05, 36.05it/s]
Loading 0: 44%|████▍ | 161/363 [00:04<00:05, 36.63it/s]
Loading 0: 46%|████▌ | 166/363 [00:04<00:05, 38.23it/s]
Loading 0: 47%|████▋ | 171/363 [00:04<00:04, 39.80it/s]
Loading 0: 48%|████▊ | 176/363 [00:04<00:05, 33.91it/s]
Loading 0: 50%|█████ | 183/363 [00:04<00:04, 40.55it/s]
Loading 0: 52%|█████▏ | 188/363 [00:04<00:04, 40.01it/s]
Loading 0: 53%|█████▎ | 193/363 [00:04<00:04, 40.24it/s]
Loading 0: 55%|█████▍ | 198/363 [00:05<00:03, 42.26it/s]
Loading 0: 56%|█████▌ | 203/363 [00:05<00:04, 35.70it/s]
Loading 0: 58%|█████▊ | 210/363 [00:05<00:03, 42.87it/s]
Loading 0: 59%|█████▉ | 215/363 [00:05<00:03, 42.10it/s]
Loading 0: 61%|██████ | 221/363 [00:05<00:03, 45.83it/s]
Loading 0: 62%|██████▏ | 226/363 [00:05<00:04, 28.13it/s]
Loading 0: 63%|██████▎ | 230/363 [00:06<00:04, 28.27it/s]
Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 35.30it/s]
Loading 0: 67%|██████▋ | 242/363 [00:06<00:03, 36.14it/s]
Loading 0: 68%|██████▊ | 247/363 [00:06<00:03, 37.06it/s]
Loading 0: 69%|██████▉ | 252/363 [00:06<00:02, 38.79it/s]
Loading 0: 71%|███████ | 257/363 [00:06<00:03, 33.65it/s]
Loading 0: 73%|███████▎ | 264/363 [00:06<00:02, 40.15it/s]
Loading 0: 74%|███████▍ | 269/363 [00:06<00:02, 40.42it/s]
Loading 0: 75%|███████▌ | 274/363 [00:07<00:02, 40.92it/s]
Loading 0: 77%|███████▋ | 279/363 [00:07<00:02, 41.50it/s]
Loading 0: 78%|███████▊ | 284/363 [00:07<00:02, 33.79it/s]
Loading 0: 80%|████████ | 291/363 [00:07<00:01, 41.17it/s]
Loading 0: 82%|████████▏ | 296/363 [00:07<00:01, 40.56it/s]
Loading 0: 83%|████████▎ | 301/363 [00:07<00:01, 41.37it/s]
Loading 0: 84%|████████▍ | 306/363 [00:08<00:02, 22.98it/s]
Loading 0: 85%|████████▌ | 310/363 [00:08<00:02, 24.01it/s]
Loading 0: 87%|████████▋ | 314/363 [00:08<00:01, 26.17it/s]
Loading 0: 88%|████████▊ | 320/363 [00:08<00:01, 31.92it/s]
Loading 0: 90%|████████▉ | 326/363 [00:08<00:01, 33.38it/s]
Loading 0: 91%|█████████ | 330/363 [00:08<00:01, 32.53it/s]
Loading 0: 93%|█████████▎| 338/363 [00:09<00:00, 41.26it/s]
Loading 0: 95%|█████████▍| 344/363 [00:09<00:00, 40.47it/s]
Loading 0: 96%|█████████▌| 349/363 [00:09<00:00, 39.83it/s]
Loading 0: 98%|█████████▊| 355/363 [00:09<00:00, 43.53it/s]
Loading 0: 99%|█████████▉| 360/363 [00:09<00:00, 42.99it/s]
Job rica40325-fliter65kv1-v1-mkmlizer completed after 95.64s with status: succeeded
Stopping job with name rica40325-fliter65kv1-v1-mkmlizer
Pipeline stage MKMLizer completed in 96.17s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rica40325-fliter65kv1-v1
Waiting for inference service rica40325-fliter65kv1-v1 to be ready
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service rica40325-fliter65kv1-v1
%s, retrying in %s seconds...
Creating inference service rica40325-fliter65kv1-v1
Waiting for inference service rica40325-fliter65kv1-v1 to be ready
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service rica40325-fliter65kv1-v1
%s, retrying in %s seconds...
Creating inference service rica40325-fliter65kv1-v1
Waiting for inference service rica40325-fliter65kv1-v1 to be ready
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-cyndonia24b-cpos_11400_v1: ('http://chaiml-cyndonia24b-cpos-11400-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service rica40325-fliter65kv1-v1
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService rica40325-fliter65kv1-v1. The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2025-06-10T09:38:36Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2025-06-10T09:38:36Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2025-06-10T09:38:37Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2025-06-10T09:38:42Z\'}], \'name\': \'rica40325-fliter65kv1-v1\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'415923180\', \'uid\': \'c94906bd-b94b-4e8d-9851-e9b6a4a098fd\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'MAX_CACHED_RESPONSES\', \'value\': \'-1\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/rica40325-fliter65kv1-v1\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:v1.18.35\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 500, \'minReplicas\': 0, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'rica40325-fliter65kv1-v1-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2025-06-10T09:38:38Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2025-06-10T09:38:38Z\', \'message\': \'Revision "rica40325-fliter65kv1-v1-predictor-00001" failed with message: 0/4225 nodes are available: 1 node(s) had taint {node.coreweave.cloud/reserved: 007b3cc3da717eac69ab0559a137e7f3b606c461}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 0916d1e497c1c48dbbfe549139a7a5898c3cc36c}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 1a595a58a4eeff882120a1e1b0d1010e09698d99}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 1ab2bafe389d143597c5335bc6baa73e7629eb1c}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 326155b28370d67b46839f7a77fb42d24e633355}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 3a45a883af800a19f23cadb8f85db86d74b5a84f}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 3b3525672b1d07bb0e82ed6aa93fb7b63151b984}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 4ad14ec6a7b860f9abd125bc6f682c4f86c03bab}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 8475d6203fe60d2b8cb5af41ca53d6d5a4777694}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 85bb0ba151c747754c81f7c9f79197511768624a}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 960b232b8c784c940161614146fd165b5bd0be0a}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 9dedfac275b2b916c916a8f5b21f7e7efa369b15}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 9f636c92f7d8da46b07c40d851caf4968df86237}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: a5120afadb4b45cbd040add46dc20e5015a987bf}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: ab6668f5db9960265cd5619120217d08181b955e}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: afdef22bd9179253164b37cabb64f3e40675acb2}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: c155caf3468ba5b86882781ef1b4b1d508c91f3d}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: c34219f02015e9725e0e83dae49cbb2bc89dbac3}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: d9a4805717baccf21a30a19f47cb010767a4f67b}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: e239a1e1f1cd3fefe8276e3d58646858101fa194}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: e603c298c138ac32e157f8d732075ff63d7af158}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: fb5beb3113ffb96ddf1b1bcb694dbc0afa65bfc6}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: ffb2607e06ed03796809e5e2c78fd2c1c2d8ec65}, that the pod didn\\\'t tolerate, 1 node(s) had taint {test.foo.com/thing: }, that the pod didn\\\'t tolerate, 12 node(s) had taint {node.coreweave.cloud/reserved: 7bb0910539250d61afdb118ca00f35954c4c65a3}, that the pod didn\\\'t tolerate, 13 node(s) had taint {node.coreweave.cloud/reserved: 0e588f0c71cf3df28088e9c13ae1fd11a80165e7}, that the pod didn\\\'t tolerate, 14 node(s) had taint {node.coreweave.cloud/reserved: b30d62b659c7adc22a11354c4debfa194e7fb193}, that the pod didn\\\'t tolerate, 1400 node(s) were unschedulable, 152 node(s) had taint {node.coreweave.cloud/reserved: mimir}, that the pod didn\\\'t tolerate, 1744 node(s) didn\\\'t match Pod\\\'s node affinity, 2 node(s) had taint {node.coreweave.cloud/reservation-policy: local}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 047edcf4a982e8f4954be13f0346f48956e44b61}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 068123c732583ca97229d877d4556e1e1f4ca50d}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 07b2baa5117bf3dd29052fdc67f601965171b005}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 08c1468d7e24e3b0938e976db9cc5cd234ce0b06}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 2f70572f9f29c093e947a8e5963e95291e1dcb9b}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 6c3c6b209d62ed4178811e7d043d6bbcc1a9e43a}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: b528168f007b9294330e209e0ff29d083c6363c1}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: ddfd41dda0413d090ac08ed7cab356be13b4c10c}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved_for_prometheus: true}, that the pod didn\\\'t tolerate, 24 node(s) had taint {node.coreweave.cloud/reserved: 9d310b2299204b884162349bd9e1c6ba8269dbc5}, that the pod didn\\\'t tolerate, 3 node(s) had taint {node.coreweave.cloud/reserved: bb01192a8ff7186ad7285ee0b54492896962197f}, that the pod didn\\\'t tolerate, 3 node(s) had taint {node.coreweave.cloud/reserved: f7a44a72a965b42d74feced14310079a26b3230d}, that the pod didn\\\'t tolerate, 50 node(s) had taint {node.coreweave.cloud/reserved: 6c7fa72bb0e687df2f2a055b49e2e7687c0dc25e}, that the pod didn\\\'t tolerate, 655 node(s) had taint {node.coreweave.cloud/hypervisor: true}, that the pod didn\\\'t tolerate, 7 node(s) had taint {node.coreweave.cloud/reserved: 04688a0d6a3e07a42ec8266db6b2253d1faf71fc}, that the pod didn\\\'t tolerate, 9 node(s) had taint {node.coreweave.cloud/reserved: d9d52a7cb8a5be4cf618a4f0417eac9e6df4dd57}, that the pod didn\\\'t tolerate, 94 Insufficient nvidia.com/gpu..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2025-06-10T09:38:42Z\', \'message\': \'Configuration "rica40325-fliter65kv1-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2025-06-10T09:38:42Z\', \'message\': \'Configuration "rica40325-fliter65kv1-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2025-06-10T09:38:42Z\', \'message\': \'Configuration "rica40325-fliter65kv1-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2025-06-10T09:38:42Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
Shutdown handler de-registered
rica40325-fliter65kv1_v1 status is now failed due to DeploymentManager action