GPU workloads need more than a pod manifest with a GPU limit. Kubernetes schedules extended resources advertised by device plugins, but real capacity also depends on GPU model, memory, topology, driver compatibility and whether a workload can share a device.
Understand how GPUs become schedulable
A vendor device plugin advertises resources such as nvidia.com/gpu on eligible nodes. The kubelet reports allocatable capacity and the scheduler places pods whose requests can be satisfied. Install and validate the driver and device plugin using the vendor’s supported instructions before debugging application manifests.
Request the resource explicitly
resources:
limits:
nvidia.com/gpu: 1
For extended resources, a limit is also the request. Do not assume a CPU or memory request reserves GPU memory. GPU memory is often not independently scheduled by the default scheduler, so model size and concurrent processes can still cause out-of-memory failures.
Separate node pools by capability
Label nodes with stable capability metadata such as accelerator family and driver generation. Use node affinity or a dedicated node pool to prevent incompatible workloads from landing together. Taints and tolerations can keep general-purpose pods away from expensive accelerator nodes.
Reduce fragmentation
Small jobs can leave unusable gaps in GPU memory or device capacity. Consider time-slicing or hardware partitioning only when supported and when isolation expectations are understood. Multi-instance GPU partitions can improve utilization but constrain the shape of workloads that fit. Track allocation and actual utilization; requested capacity alone does not show whether GPUs are busy.
Observe the full workload
Monitor device utilization, memory use, throttling, temperature, pod restarts, queue time and model throughput. Compare scheduler placement with application-level metrics such as tokens per second and batch latency. A GPU that reports high utilization may still be serving inefficiently if input pipelines or storage are bottlenecks.
Operational checklist
- Pin compatible driver and device-plugin versions.
- Validate allocatable GPU resources on every node.
- Use labels, affinity and taints for accelerator pools.
- Set CPU and memory requests alongside GPU limits.
- Plan quotas and queueing for shared clusters.
- Test node drain and recovery for long-running jobs.
GPU scheduling is a capacity-management problem as much as a Kubernetes configuration problem. Pair it with Kubernetes GPU FinOps and our vLLM production deployment guide.
Engineering considerations
GPU scheduling failures often begin below the pod specification. Confirm that the host driver is compatible with the device plugin and container runtime, then verify that the plugin registers healthy devices with kubelet. Inspect node capacity and allocatable values and review plugin logs after node upgrades. A node can be Ready while GPU resources are unavailable. Keep a tested compatibility matrix for kernel, driver, container toolkit, device plugin and workload framework, and roll upgrades through a canary node pool.
Implementation details
Different accelerator generations expose different memory sizes, compute capabilities and supported features. Label nodes with stable capability information and schedule workloads using node affinity. Taints can reserve expensive nodes for GPU workloads, while tolerations grant explicit access. Avoid labels that become stale after hardware replacement. Automate node discovery where possible and validate labels against the physical inventory before admitting a workload.
Security and reliability controls
Exclusive GPU allocation is easy to reason about but can leave capacity idle for small inference tasks. Time-slicing allows multiple workloads to share a device, but may not provide hard memory isolation or predictable latency. Hardware partitioning can create more isolated slices on supported hardware, but constrains profiles and placements. Choose a sharing model based on tenant trust, latency requirements and vendor support; do not promise isolation that the mechanism cannot provide.
Testing strategy
The default scheduler understands advertised GPU counts, not every detail of model memory or interconnect topology. A pod may receive a GPU and still fail when loading a large model. Record model memory requirements, batch sizes and runtime overhead, then validate fit with representative load tests. Distributed training may require multiple GPUs with fast interconnects on the same node or topology domain. Use topology-aware scheduling only after confirming support in the Kubernetes and device-plugin versions deployed.
Operational monitoring
Use namespace quotas and admission policies to control who can request accelerators. Define priority classes for production inference, training and experimentation. A runaway notebook should not starve a customer-facing model server. Consider queueing systems for batch jobs so users request work instead of holding an accelerator idle. Track quota usage by team and project, and include idle allocation in cost reports. Every quota should have an owner and an exception process.
Failure diagnosis
Collect device utilization, memory use, temperature, power draw and throttling metrics. Correlate these with pod placement, request queue time, batch size and tokens per second. High GPU utilization can coexist with poor throughput if CPU preprocessing, network transfer or storage is the bottleneck. Track scheduling failures and pending time as well as device metrics. Dashboards should distinguish allocated capacity from active use so teams can identify fragmentation and overprovisioning.
Rollout and maintenance
Drain a GPU node in staging and verify what happens to long-running jobs. Training workloads may need checkpointing and restart logic; inference deployments need spare capacity to maintain service during maintenance. Test device-plugin restarts, driver upgrades and node replacement. Document rescheduling and data recovery. Capacity planning must include failure reserve, not just the number of accelerator cards installed in the rack.
Operational handover
Before handing this capability to an operations team, document the configuration owner, supported versions, dependency map and escalation path. Include the commands or dashboard views needed to verify normal operation, along with the expected values and known limitations. Store configuration in version control, review changes through the normal deployment process and keep a tested rollback procedure. Schedule periodic reviews because upstream components, security requirements and workload patterns change. A short runbook should explain how to identify a failed component, what evidence to collect, which actions are safe to retry and when to escalate. Test that runbook during a maintenance exercise rather than waiting for an outage. Finally, measure the result against a baseline: reliability improvements should be visible in error rates, latency, resource consumption, recovery time or reduced manual work. This makes the implementation maintainable after the original developer has moved to another project.
