Kubernetes 1.37: Solving AI workload preemption with KEP-5710

The August 26, 2026 release of Kubernetes 1.37 caught my attention for a very specific reason. Codenamed Garhwal and led by Dipesh Rawat, this release ships with 67 enhancements. The most interesting part is a fundamental architectural shift. Kubernetes is moving from a strictly pod-centric model to becoming natively workload-aware.

If you manage distributed AI or HPC workloads, you already know the pain of gang scheduling on Kubernetes. Historically, the kube-scheduler handled resource contention strictly pod by pod. This behavior works fine for stateless web services but fails completely for distributed training jobs that require an all-or-nothing approach.

When the scheduler evicts a single worker pod to free up space, the entire training job stalls. You end up with fragmented error streams, partial-state deadlocks, and incredibly expensive GPUs sitting idle while the system tries to figure out what to do next.

KEP-5710 and atomic preemption

That brings us to KEP-5710. Titled Workload-aware preemption, this feature just graduated to Beta in 1.37 under the WorkloadAwarePreemption feature gate. We can now treat groups of related pods as a single atomic entity during resource contention.

Digging into the code, the mechanics for parallel pod-group preemption were implemented in PR #137981 by contributor vshkrabkov. The code introduces two new abstractions called Victim and DomainVictim. These allow the scheduler to evaluate and evict entire PodGroups in parallel as a single atomic unit. This collective success or failure model finally resolves the error-stream ambiguity that has plagued gang-scheduled workloads for years.

Decoupling APIs for scale

From a scalability perspective, the Kubernetes team made a smart architectural decision leading up to this release. Under KEP-5832, they decoupled the PodGroup API and the Workload API.

By separating these boundaries, cluster operators can scale and manage them independently. This gives us a lot more flexibility when configuring complex batch scheduling and preemption policies across large node pools. It keeps the control plane from getting bogged down when evaluating massive distributed training runs.

Dynamic resource allocation at the workload level

Kubernetes 1.37 also advanced KEP-5729 to Beta, which integrates Dynamic Resource Allocation (DRA) with the new scheduling primitives.

DRA ResourceClaims can now be associated directly with Workload and PodGroup APIs. Instead of reserving specialized hardware like GPUs and TPUs on a strict per-pod basis, we can share them across an entire workload. This is exactly the kind of implementation detail that helps optimize hardware utilization without building custom controllers from scratch.

Ecosystem tools like Volcano and Kueue still have a maturity advantage for production environments right now. I expect many teams will stick with them in the short term. But KEP-5710 signals a clear shift toward native workload-aware scheduling. I am looking forward to testing these primitives in our own clusters to see how they impact idle compute time and simplify our infrastructure stack.

References

• https://kubernetes.io/blog/2026/08/26/kubernetes-v1-37-release/
• https://www.networkworld.com/article/4214824/kubernetes-1-37-advances-workload-aware-scheduling-and-cluster-networking.html
• https://palark.com/blog/kubernetes-1-36-release-features/
• https://lwkd.info/2026/20260716
• https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/5710-workload-aware-preemption/README.md
• https://hidekazu-konishi.com/entry/gang_scheduling_and_dynamic_resource_allocation_on_amazon_eks.html
• https://www.cloudoptimo.com/blog/explore-workload-aware-scheduling-with-pod-groups-in-kubernetes/