Thursday 27 August 2026, 08:06 PM
Zero-downtime ML: How GPU-aware CRIU enables lossless preemption for AI spot instances
Explore how GPU-aware CRIU integration in cloud runtimes enables lossless preemption and sub-second live migration, revolutionizing AI spot instance costs.
In my experience managing infrastructure across the Bay Area, running large language models on spot instances is usually a calculated gamble. You get compute at a steep discount, but the cloud provider can pull the plug at any moment. When that happens, everything sitting in VRAM vanishes. For years, my teams relied on manual application-level checkpointing to save training progress. On the inference side, a preempted node meant suffering through brutal cold starts and terrible Time To First Token metrics while spinning up a replacement instance.
The recent integration of GPU-aware Checkpoint/Restore In Userspace (CRIU) into OCI-compliant runtimes like containerd changes this dynamic. We can now support lossless preemption and live migration directly at the container level.
GPU contexts and VRAM allocations live outside the standard process address space. Early attempts to snapshot this data relied on API interception, which injected exponential performance overhead into every operation. I found it practically unusable in production. The underlying mechanics finally changed in April 2024 when NVIDIA released native support for checkpoint and restore functionality. They deprecated those inefficient interception methods and enabled a proper CRIU CUDA plugin. By using NVIDIA's native utility, the system transparently snapshots GPU-enabled processes.
A February 2025 paper, 'CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads' (arXiv:2502.16631), validated this architecture. Radostin Stoyanov and his team proved that CRIUgpu operates across multiple GPUs for deep learning tasks, eliminating the steady-state overhead that plagued earlier methods.
AMD engineers tackled a different bottleneck in June 2025 with a major architectural redesign to the CRIU AMDGPU plugin. Previously, restoring a state involved a sequential process where child processes had to fork and repopulate VRAM one by one. AMD introduced a parallel path for GPU restore. This reduced the latency required to get a node back online and made live migration viable on their hardware.
The mechanics of lossless preemption
Instead of terminating a preempted spot instance, Kubernetes can now pause the container, dump the exact CPU and GPU state, and migrate the workload to an available node.
In mid-2026, researchers from Shanghai Jiao Tong University and Huawei published a paper detailing a system called DeltaBox (arXiv:2605.22781). They built a copy-on-write (COW) checkpoint and rollback engine that captures process-level agent snapshots in milliseconds. By using dirty page tracking and userfaultfd, the orchestrator only moves the memory pages that actually changed. This specific optimization enables sub-second live migration for stateful AI agents.
Bandwidth realities and network congestion
Live migrating tens of gigabytes of VRAM requires substantial backend network bandwidth. If an entire rack of spot instances gets preempted simultaneously, dumping all that VRAM across the network fabric risks severe localized congestion. In my own deployments, I've had to carefully design cluster architectures to handle sudden bursts of east-west traffic. The storage backend needs enough IOPS to absorb the state dumps without choking the control plane.
Memory boundaries in multi-tenant clusters
Moving raw driver states and VRAM content around introduces concrete risks in multi-tenant environments. A clear example surfaced on June 26, 2026, with the disclosure of CVE-2026-53143. Security researchers identified a medium-severity buffer overflow vulnerability in the Linux kernel module (drm/amdkfd) for AMD GPUs.
The flaw triggered during the CRIU checkpointing of SDMA queues on AMD GFX11 hardware. It caused a memory leak that exposed adjacent Graphics Translation Table (GTT) data to userspace. When building multi-tenant clouds, strict memory boundary enforcement is non-negotiable. A vulnerability that leaks memory across tenant boundaries is a critical failure for any infrastructure provider.
We are effectively trading application-level complexity for kernel-level responsibility. Developers no longer need to write custom state-saving logic into every AI model, but platform engineers now have to treat GPU memory dumps with the same network and security rigor as bare-metal hypervisor migrations. If you are running H100s on spot instances, the cost savings are undeniable—you just have to build a network fabric capable of catching 80GB of VRAM the second AWS decides it wants that node back.