CPU and memory are fungible — Kubernetes can slice them arbitrarily thin and pack workloads tightly. GPUs aren't: a GPU is either allocated to a pod or it isn't, and until fairly recently, Kubernetes had no native concept of a GPU at all. GPU management on Kubernetes is the layer that fills that gap.

The device plugin model

Kubernetes exposes GPUs to the scheduler through the device plugin framework — a node-level component advertises available GPUs as a schedulable resource, and a pod requests a specific count of them the same way it requests CPU or memory, just as an opaque, whole-unit resource rather than something the scheduler can subdivide on its own.

Why whole-GPU allocation wastes capacity

Without further help, a pod requesting "1 GPU" gets an entire physical GPU, even if the workload only needs a fraction of its compute or memory. A cluster running many small inference workloads on full-sized GPUs this way is often paying for far more accelerator capacity than it's actually using.

MIG-aware allocation

Multi-Instance GPU (MIG) support on modern data-center GPUs lets a single physical GPU be partitioned into several fully isolated instances, each with its own dedicated compute and memory. MIG-aware scheduling means Kubernetes can allocate those partitions individually — a small inference workload gets a MIG slice instead of an entire card, and a large training job still gets a full GPU when it needs one.

Common pitfalls

  • Bin-packing blindness — without GPU-aware scheduling, small workloads can end up spread across many nodes instead of packed onto shared GPU capacity.
  • Idle GPU spend — GPUs sitting allocated to a pod that isn't actually using them are the most expensive idle resource in most clusters.
  • No visibility until the invoice — without node-level GPU visibility, utilization patterns like these are invisible until someone reviews the cloud bill.

AI workload templates

Standardized templates for common AI/ML workload shapes — training jobs, inference services — reduce how often someone hand-writes a pod spec with the wrong GPU request, and make it easier to apply consistent resource limits across a team.

How AetherVirt handles GPU scheduling

AetherVirt provides node-level GPU visibility and MIG-aware allocation as part of its platform capabilities, so accelerated and AI workloads land on the right hardware automatically instead of being hand-placed by an operator guessing at capacity.