Interview: Kubernetes Storage

Ephemeral volumes (emptyDir, configMap, secret, projected) live and die with the Pod - data is lost when the Pod is deleted. Persistent volumes (via PVC) survive Pod restarts and rescheduling because storage is provisioned independently through PV/PVC binding. Use ephemeral volumes for scratch space and config; use PVCs for databases, file stores, and any data that must outlive a single Pod.

emptyDir is ideal for temporary data shared between containers in the same Pod (sidecar log processing, build caches). hostPath mounts a path on the node filesystem - useful only for node-level daemons (log agents, CNI plugins); avoid it for application data because Pods are not tied to nodes. A PVC is the right choice when data must persist across Pod lifecycle events and be portable across nodes via cloud or network storage.

A PersistentVolume (PV) is a cluster-level storage resource representing actual capacity (NFS share, EBS volume, Ceph RBD). A PersistentVolumeClaim (PVC) is a namespace-scoped request for storage. The control plane binds a matching PVC to an available PV. With dynamic provisioning, the StorageClass provisioner creates the PV automatically when a PVC is created - admins no longer pre-provision disks manually.

ReadWriteOnce (RWO) - one node mounts read-write; standard for block storage (EBS, Azure Disk). ReadOnlyMany (ROX) - many nodes read-only; config distribution. ReadWriteMany (RWX) - many nodes read-write; requires shared filesystems (NFS, EFS, CephFS). ReadWriteOncePod (RWOP) - exclusive to a single Pod (K8s 1.22+). Mismatch between PVC access mode and StorageClass capability is a common binding failure.

A StorageClass defines the provisioner (e.g. ebs.csi.aws.com), parameters (disk type, IOPS, encryption), reclaim policy, and binding mode. When a PVC references a StorageClass, the provisioner creates a PV and binds it automatically. Set storageclass.kubernetes.io/is-default-class: "true" so PVCs without an explicit class still get storage. allowVolumeExpansion: true enables online PVC resizing.

Immediate provisions and binds a PV as soon as the PVC is created - fast but may place storage in the wrong availability zone. WaitForFirstConsumer delays provisioning until a Pod using the PVC is scheduled, then creates storage in the same AZ as the node. This is the recommended setting for cloud block storage to avoid cross-AZ attach failures and unnecessary data transfer costs.

Delete (default for dynamic provisioning) removes the PV and underlying cloud volume when the PVC is deleted. Retain keeps the PV and data after PVC deletion - you must manually reclaim and clean up; use for production databases where accidental deletion is catastrophic. Recycle is deprecated. Always verify reclaim policy before deleting PVCs in production.

The Container Storage Interface (CSI) is a standard gRPC API for storage vendors. CSI drivers run as Pods in the cluster and implement create, attach, mount, snapshot, and resize operations. In-tree plugins (built into kubelet) are deprecated; cloud providers ship CSI drivers (e.g. ebs.csi.aws.com, pd.csi.storage.gke.io). CSI also enables VolumeSnapshots for point-in-time backups independent of application logic.

A Deployment with a single shared PVC causes all replicas to compete for one volume (only works with RWX). StatefulSets use volumeClaimTemplates to create a dedicated PVC per replica (data-postgres-0, data-postgres-1, …) with stable Pod identity. Each replica gets its own RWO volume. PVCs persist even if the StatefulSet is deleted - scale-down does not auto-delete PVCs, so you must clean up orphans manually.

Three common approaches: (1) CSI VolumeSnapshots - cluster-native, fast, storage-backend dependent; restore via PVC dataSource referencing the snapshot. (2) Application-level backup - pg_dump, Velero with restic for file-level backup across clusters. (3) Storage-array replication - async replication to another region. For disaster recovery, test restore procedures regularly; snapshots alone are not backups if they live in the same failure domain.

Check the chain: kubectl describe pvc (Bound? Capacity? Access mode?), kubectl describe pod (Events tab - attach failures, permission denied), CSI driver Pods in kube-system, and node attach limits. Common causes: PVC pending (no matching StorageClass or quota), RWO volume already attached to another node, insufficient IAM permissions for CSI driver, or AZ mismatch with Immediate binding mode.

Requirements: StorageClass with allowVolumeExpansion: true, CSI driver supporting resize, and a filesystem that supports online growth. Patch the PVC: kubectl patch pvc my-pvc -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'. The CSI driver expands the volume; you may need to restart the Pod or run resize2fs/xfs_growfs inside the container if the filesystem does not auto-expand. Verify with kubectl get pvc and df -h in the Pod.