A Kubernetes workload can remain Pending even though its YAML looks valid and the cluster has free CPU and memory. If the Pod references a PersistentVolumeClaim (PVC) that never becomes Bound, the scheduler may be unable to place the workload or mount its data. The right fix depends on the storage path: the claim’s requested size and access mode, the selected StorageClass, the CSI provisioner, topology, an existing PersistentVolume, and the backend itself.
This guide is a diagnostic runbook for a PVC that stays Pending. It focuses on evidence-first troubleshooting and safe recovery. The commands are read-only unless explicitly identified as a change. Replace the sample namespace and claim names with your own values.
Symptoms: confirm which object is Pending
Start by separating a Pending PVC from a Pending Pod. A Pod can be Pending for CPU, memory, taints, affinity, image pulls, or storage. A PVC can be Pending because no matching volume exists or provisioning has not completed. Check both:
kubectl -n app get pods -o wide
kubectl -n app get pvc
kubectl -n app describe pvc data-store
kubectl -n app describe pod app-0In the PVC description, inspect Events, requested capacity, access modes, volume mode, and StorageClass. Events often reveal whether the controller cannot find a matching PV, the external provisioner is unavailable, or a backend operation failed. Capture the exact event text and timestamp before changing anything.
Root causes to investigate
1. No default or requested StorageClass
A claim without storageClassName may depend on a default StorageClass. A misspelled or nonexistent class prevents the intended provisioning path. A claim with storageClassName: "" explicitly requests no class, which is different from omitting the field. Inspect classes and the claim specification:
kubectl get storageclass
kubectl -n app get pvc data-store -o yaml2. No matching static PersistentVolume
For static provisioning, the cluster needs an available PV whose capacity, access modes, volume mode, and class satisfy the claim. A 100Gi request cannot bind to a 50Gi volume. A PV already bound to another claim cannot be reused just because it appears in the cluster list.
kubectl get pv -o wide
kubectl describe pv pv-name3. CSI provisioning or backend failure
Dynamic provisioning depends on a working StorageClass provisioner and its CSI components. A missing driver, failed controller, invalid provider parameter, quota limit, or storage API failure can prevent volume creation. Inspect the StorageClass provisioner and the relevant CSI controller logs using the namespace and labels supplied by your storage vendor. Avoid assuming every CSI driver uses the same Deployment name.
4. Topology and WaitForFirstConsumer
Some storage systems are constrained to zones, nodes, or topology segments. With Immediate binding, provisioning can happen before the scheduler knows where the Pod should run. WaitForFirstConsumer delays binding/provisioning until a Pod using the claim is considered for scheduling. A PVC may therefore remain Pending until a consuming Pod exists; that can be expected behavior, not a broken driver.
5. Access mode or volume mode mismatch
ReadWriteOnce, ReadOnlyMany, and ReadWriteMany describe different attachment and mount semantics; they are not interchangeable promises. Likewise, Filesystem and Block volume modes require different handling. Confirm the backend and CSI driver support the requested mode rather than weakening the claim spec to make it bind.
Diagnosis: follow the claim to the provisioner
Check the claim’s events and its StorageClass:
kubectl -n app describe pvc data-store
kubectl get storageclass fast-ssd -o yaml
kubectl get pvIf a Pod consumes the claim, inspect its events and scheduling decisions:
kubectl -n app describe pod app-0
kubectl -n app get pod app-0 -o yamlFor dynamic provisioning, identify the CSI driver from the StorageClass provisioner field. Then inspect the driver’s controller and node components in the vendor-documented namespace. For example, list Pods in a known CSI namespace before choosing logs:
kubectl -n storage-system get pods -o wide
kubectl -n storage-system get events --sort-by=.lastTimestampUse the actual namespace and selector for your driver. Controller logs may show authentication failures, quota limits, invalid parameters, API throttling, or a backend outage. Do not publish logs that include credentials, internal endpoint names, or tenant identifiers.
Fix: choose the repair that matches the evidence
If the StorageClass name is wrong, correct the workload’s PVC template or manifest in version control. If no default class exists, ask the cluster administrator to select and document the intended class; do not set a cluster-wide default casually because it changes the behavior of other claims. If the provisioner is unhealthy, restore the CSI controller or its backend permissions using the storage vendor’s supported procedure.
For a static PV mismatch, provision a PV that satisfies the claim. If the claim itself was created with incorrect parameters, plan a replacement only after confirming it is unbound, no workload depends on it, and deletion is safe; many PVC fields are immutable after creation. For topology issues, verify the StorageClass binding mode and node topology. Do not change a StorageClass used by existing workloads without understanding its effect on future claims and provisioning.
Be especially careful with deletion. Deleting a PVC can trigger volume reclamation depending on the PV reclaim policy and storage integration. A StorageClass reclaim policy of Delete can cause the underlying volume to be deleted after the claim is released. Before any delete or recreate operation, identify the bound PV, its reclaim policy, the data owner, and a tested backup. Never delete a production PVC simply to force a fresh provisioning attempt.
Validation: confirm the entire storage path
After correcting the cause, watch the claim and Pod:
kubectl -n app get pvc data-store -w
kubectl -n app get pods -o wide
kubectl -n app describe pvc data-storeOnce the claim is Bound and the Pod is Running, verify that the expected filesystem or block device is available to the application. A successful bind alone does not prove that the application can read and write its data. Use a non-destructive application health check or a controlled test file in a designated test path, not an arbitrary write into a production data directory. Check CSI attach/mount events and application logs for permission or filesystem errors.
Rollback and data safety
If a configuration change creates a new failure, revert the manifest or StorageClass change through the same deployment workflow. Restoring a previous YAML file does not reverse a volume deletion, filesystem format, or backend snapshot operation. For storage changes, rollback means restoring the configuration and, if data was altered, using a verified backup or snapshot recovery procedure. Confirm that the application is quiesced when the storage vendor requires it.
Operational checklist
- Confirm whether the PVC, Pod, or both are Pending.
- Read PVC and Pod events before editing resources.
- Check StorageClass name, provisioner, binding mode, parameters, and reclaim policy.
- For static PVs, compare capacity, access modes, volume mode, and class.
- For dynamic volumes, inspect CSI controller events and vendor-specific logs.
- Check topology and the intended node/zone placement.
- Protect data before deleting or recreating any claim or volume.
- Validate binding, mounting, and application-level reads/writes after the fix.
Primary references
Kubernetes Persistent Volumes and Claims; Kubernetes Storage Classes; Kubernetes Volumes. Related ProdCraft guides: Kubernetes OOMKilled diagnosis and CoreDNS troubleshooting.
