Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-checkpoint-1068-v1-mkmlizer
Waiting for job on riverise-checkpoint-1068-v1-mkmlizer to finish
riverise-checkpoint-1068-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-checkpoint-1068-v1-mkmlizer: ║ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ██████ ██████ █████ ████ ████ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░███░█████░███ ░███ ███ ░███ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░███░░███ ░███ ░███████ ░███ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░███ ░███ ░███ ░░███ ░███ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ █████ █████ █████ ░░████ █████ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Version: 0.27.1+vampire_v3 ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Features: FLYWHEEL, CUDA ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
riverise-checkpoint-1068-v1-mkmlizer: ║ https://mk1.ai ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ The license key for the current software has been verified as ║
riverise-checkpoint-1068-v1-mkmlizer: ║ belonging to: ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Chai Research Corp. ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
riverise-checkpoint-1068-v1-mkmlizer: ║ Expiration: 2028-03-31 23:59:59 ║
riverise-checkpoint-1068-v1-mkmlizer: ║ ║
riverise-checkpoint-1068-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
riverise-checkpoint-1068-v1-mkmlizer: Downloaded to shared memory in 47.788s
riverise-checkpoint-1068-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp25eoz8by, device:0
riverise-checkpoint-1068-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
riverise-checkpoint-1068-v1-mkmlizer: quantized model in 30.607s
riverise-checkpoint-1068-v1-mkmlizer: Processed model Riverise/checkpoint-1068 in 78.396s
riverise-checkpoint-1068-v1-mkmlizer: creating bucket guanaco-mkml-models
riverise-checkpoint-1068-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-checkpoint-1068-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-checkpoint-1068-v1
riverise-checkpoint-1068-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-checkpoint-1068-v1/config.json
riverise-checkpoint-1068-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-checkpoint-1068-v1/special_tokens_map.json
riverise-checkpoint-1068-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-checkpoint-1068-v1/tokenizer_config.json
riverise-checkpoint-1068-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-checkpoint-1068-v1/tokenizer.json
riverise-checkpoint-1068-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/riverise-checkpoint-1068-v1/flywheel_model.0.safetensors
riverise-checkpoint-1068-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:12, 29.18it/s]
Loading 0: 3%|▎ | 12/363 [00:00<00:07, 47.02it/s]
Loading 0: 5%|▍ | 18/363 [00:00<00:07, 46.59it/s]
Loading 0: 6%|▋ | 23/363 [00:00<00:09, 34.79it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:07, 44.18it/s]
Loading 0: 10%|█ | 37/363 [00:00<00:07, 42.20it/s]
Loading 0: 12%|█▏ | 42/363 [00:01<00:07, 41.18it/s]
Loading 0: 13%|█▎ | 49/363 [00:01<00:06, 47.18it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:06, 46.35it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:08, 36.35it/s]
Loading 0: 18%|█▊ | 66/363 [00:01<00:08, 36.18it/s]
Loading 0: 20%|█▉ | 72/363 [00:01<00:07, 39.89it/s]
Loading 0: 21%|██ | 77/363 [00:01<00:07, 40.54it/s]
Loading 0: 23%|██▎ | 82/363 [00:02<00:07, 35.27it/s]
Loading 0: 25%|██▍ | 89/363 [00:02<00:06, 42.08it/s]
Loading 0: 26%|██▌ | 94/363 [00:02<00:06, 42.48it/s]
Loading 0: 27%|██▋ | 99/363 [00:02<00:06, 41.32it/s]
Loading 0: 29%|██▉ | 105/363 [00:02<00:06, 40.41it/s]
Loading 0: 31%|███ | 112/363 [00:02<00:05, 45.08it/s]
Loading 0: 32%|███▏ | 117/363 [00:02<00:05, 44.01it/s]
Loading 0: 34%|███▎ | 122/363 [00:02<00:05, 44.96it/s]
Loading 0: 35%|███▍ | 127/363 [00:03<00:06, 36.07it/s]
Loading 0: 37%|███▋ | 134/363 [00:03<00:05, 43.26it/s]
Loading 0: 38%|███▊ | 139/363 [00:03<00:05, 43.90it/s]
Loading 0: 40%|███▉ | 144/363 [00:03<00:07, 27.96it/s]
Loading 0: 41%|████ | 149/363 [00:03<00:07, 30.52it/s]
Loading 0: 43%|████▎ | 156/363 [00:03<00:05, 36.38it/s]
Loading 0: 44%|████▍ | 161/363 [00:04<00:05, 35.84it/s]
Loading 0: 46%|████▌ | 166/363 [00:04<00:05, 36.95it/s]
Loading 0: 47%|████▋ | 172/363 [00:04<00:05, 37.33it/s]
Loading 0: 48%|████▊ | 176/363 [00:04<00:05, 34.48it/s]
Loading 0: 50%|█████ | 183/363 [00:04<00:04, 41.04it/s]
Loading 0: 52%|█████▏ | 188/363 [00:04<00:04, 41.42it/s]
Loading 0: 53%|█████▎ | 193/363 [00:04<00:04, 39.63it/s]
Loading 0: 55%|█████▍ | 198/363 [00:05<00:04, 40.87it/s]
Loading 0: 56%|█████▌ | 203/363 [00:05<00:04, 34.33it/s]
Loading 0: 58%|█████▊ | 210/363 [00:05<00:03, 40.66it/s]
Loading 0: 59%|█████▉ | 215/363 [00:05<00:03, 39.90it/s]
Loading 0: 61%|██████ | 220/363 [00:05<00:03, 41.24it/s]
Loading 0: 62%|██████▏ | 225/363 [00:05<00:05, 26.25it/s]
Loading 0: 63%|██████▎ | 230/363 [00:06<00:04, 28.68it/s]
Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 35.58it/s]
Loading 0: 67%|██████▋ | 242/363 [00:06<00:03, 37.50it/s]
Loading 0: 68%|██████▊ | 247/363 [00:06<00:02, 39.58it/s]
Loading 0: 70%|██████▉ | 253/363 [00:06<00:02, 39.15it/s]
Loading 0: 71%|███████ | 258/363 [00:06<00:02, 38.49it/s]
Loading 0: 73%|███████▎ | 265/363 [00:06<00:02, 43.51it/s]
Loading 0: 75%|███████▍ | 271/363 [00:06<00:02, 42.78it/s]
Loading 0: 76%|███████▌ | 276/363 [00:07<00:02, 41.09it/s]
Loading 0: 78%|███████▊ | 283/363 [00:07<00:01, 45.61it/s]
Loading 0: 79%|███████▉ | 288/363 [00:07<00:01, 45.70it/s]
Loading 0: 81%|████████ | 293/363 [00:07<00:01, 38.57it/s]
Loading 0: 82%|████████▏ | 299/363 [00:07<00:01, 42.96it/s]
Loading 0: 84%|████████▎ | 304/363 [00:08<00:02, 25.18it/s]
Loading 0: 85%|████████▍ | 308/363 [00:08<00:02, 26.91it/s]
Loading 0: 86%|████████▌ | 312/363 [00:08<00:01, 26.21it/s]
Loading 0: 88%|████████▊ | 320/363 [00:08<00:01, 35.70it/s]
Loading 0: 90%|████████▉ | 325/363 [00:08<00:01, 37.22it/s]
Loading 0: 91%|█████████ | 330/363 [00:08<00:01, 31.31it/s]
Loading 0: 93%|█████████▎| 337/363 [00:08<00:00, 38.40it/s]
Loading 0: 94%|█████████▍| 342/363 [00:09<00:00, 38.49it/s]
Loading 0: 96%|█████████▌| 347/363 [00:09<00:00, 40.00it/s]
Loading 0: 97%|█████████▋| 352/363 [00:09<00:00, 41.85it/s]
Loading 0: 98%|█████████▊| 357/363 [00:09<00:00, 35.06it/s]
Job riverise-checkpoint-1068-v1-mkmlizer completed after 106.46s with status: succeeded
Stopping job with name riverise-checkpoint-1068-v1-mkmlizer
Pipeline stage MKMLizer completed in 107.03s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-checkpoint-1068-v1
Waiting for inference service riverise-checkpoint-1068-v1 to be ready
Tearing down inference service riverise-checkpoint-1068-v1
%s, retrying in %s seconds...
Creating inference service riverise-checkpoint-1068-v1
Waiting for inference service riverise-checkpoint-1068-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Tearing down inference service riverise-checkpoint-1068-v1
%s, retrying in %s seconds...
Creating inference service riverise-checkpoint-1068-v1
Waiting for inference service riverise-checkpoint-1068-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Tearing down inference service riverise-checkpoint-1068-v1
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService riverise-checkpoint-1068-v1. The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2025-06-09T02:43:29Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2025-06-09T02:43:29Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2025-06-09T02:43:29Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2025-06-09T02:43:31Z\'}], \'name\': \'riverise-checkpoint-1068-v1\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'414053580\', \'uid\': \'6bc75c77-d311-45d7-9a6e-3f5b367ac1ca\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'0.95\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'MAX_CACHED_RESPONSES\', \'value\': \'-1\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/riverise-checkpoint-1068-v1\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:v1.18.35\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 500, \'minReplicas\': 0, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'riverise-checkpoint-1068-v1-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'message\': \'Revision "riverise-checkpoint-1068-v1-predictor-00001" failed with message: 0/4225 nodes are available: 1 node(s) had taint {node.coreweave.cloud/reserved: 007b3cc3da717eac69ab0559a137e7f3b606c461}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 0916d1e497c1c48dbbfe549139a7a5898c3cc36c}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 1a595a58a4eeff882120a1e1b0d1010e09698d99}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 1ab2bafe389d143597c5335bc6baa73e7629eb1c}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 326155b28370d67b46839f7a77fb42d24e633355}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 3a45a883af800a19f23cadb8f85db86d74b5a84f}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 3b3525672b1d07bb0e82ed6aa93fb7b63151b984}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 4ad14ec6a7b860f9abd125bc6f682c4f86c03bab}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 8475d6203fe60d2b8cb5af41ca53d6d5a4777694}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 85bb0ba151c747754c81f7c9f79197511768624a}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 960b232b8c784c940161614146fd165b5bd0be0a}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 9dedfac275b2b916c916a8f5b21f7e7efa369b15}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: 9f636c92f7d8da46b07c40d851caf4968df86237}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: a23e6272a875746a522968abe77c4ff953358e92}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: a5120afadb4b45cbd040add46dc20e5015a987bf}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: ab6668f5db9960265cd5619120217d08181b955e}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: afdef22bd9179253164b37cabb64f3e40675acb2}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: c155caf3468ba5b86882781ef1b4b1d508c91f3d}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: c34219f02015e9725e0e83dae49cbb2bc89dbac3}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: d9a4805717baccf21a30a19f47cb010767a4f67b}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: e239a1e1f1cd3fefe8276e3d58646858101fa194}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: e603c298c138ac32e157f8d732075ff63d7af158}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: f7a44a72a965b42d74feced14310079a26b3230d}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: fb5beb3113ffb96ddf1b1bcb694dbc0afa65bfc6}, that the pod didn\\\'t tolerate, 1 node(s) had taint {node.coreweave.cloud/reserved: ffb2607e06ed03796809e5e2c78fd2c1c2d8ec65}, that the pod didn\\\'t tolerate, 1 node(s) had taint {test.foo.com/thing: }, that the pod didn\\\'t tolerate, 13 node(s) had taint {node.coreweave.cloud/reserved: 0e588f0c71cf3df28088e9c13ae1fd11a80165e7}, that the pod didn\\\'t tolerate, 13 node(s) had taint {node.coreweave.cloud/reserved: 7bb0910539250d61afdb118ca00f35954c4c65a3}, that the pod didn\\\'t tolerate, 14 node(s) had taint {node.coreweave.cloud/reserved: b30d62b659c7adc22a11354c4debfa194e7fb193}, that the pod didn\\\'t tolerate, 1404 node(s) were unschedulable, 152 node(s) had taint {node.coreweave.cloud/reserved: mimir}, that the pod didn\\\'t tolerate, 1739 node(s) didn\\\'t match Pod\\\'s node affinity, 2 node(s) had taint {node.coreweave.cloud/reservation-policy: local}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 047edcf4a982e8f4954be13f0346f48956e44b61}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 068123c732583ca97229d877d4556e1e1f4ca50d}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 07b2baa5117bf3dd29052fdc67f601965171b005}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 08c1468d7e24e3b0938e976db9cc5cd234ce0b06}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 2f70572f9f29c093e947a8e5963e95291e1dcb9b}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: 6c3c6b209d62ed4178811e7d043d6bbcc1a9e43a}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: b528168f007b9294330e209e0ff29d083c6363c1}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved: ddfd41dda0413d090ac08ed7cab356be13b4c10c}, that the pod didn\\\'t tolerate, 2 node(s) had taint {node.coreweave.cloud/reserved_for_prometheus: true}, that the pod didn\\\'t tolerate, 24 node(s) had taint {node.coreweave.cloud/reserved: 9d310b2299204b884162349bd9e1c6ba8269dbc5}, that the pod didn\\\'t tolerate, 3 node(s) had taint {node.coreweave.cloud/reserved: bb01192a8ff7186ad7285ee0b54492896962197f}, that the pod didn\\\'t tolerate, 51 node(s) had taint {node.coreweave.cloud/reserved: 6c7fa72bb0e687df2f2a055b49e2e7687c0dc25e}, that the pod didn\\\'t tolerate, 655 node(s) had taint {node.coreweave.cloud/hypervisor: true}, that the pod didn\\\'t tolerate, 7 node(s) had taint {node.coreweave.cloud/reserved: 04688a0d6a3e07a42ec8266db6b2253d1faf71fc}, that the pod didn\\\'t tolerate, 9 node(s) had taint {node.coreweave.cloud/reserved: d9d52a7cb8a5be4cf618a4f0417eac9e6df4dd57}, that the pod didn\\\'t tolerate, 95 Insufficient nvidia.com/gpu..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'message\': \'Configuration "riverise-checkpoint-1068-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'message\': \'Configuration "riverise-checkpoint-1068-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'message\': \'Configuration "riverise-checkpoint-1068-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2025-06-09T02:43:31Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
Shutdown handler de-registered
riverise-checkpoint-1068_v1 status is now failed due to DeploymentManager action