NVIDIA Snapshot: Accelerate GPU Workload Cold Starts on Kubernetes

At a Glance
| Item | Value |
|---|---|
| Publisher guide | NVIDIA: models and licenses |
| Publisher | NVIDIA Developer |
| Published | 2026-11-11 |
| License | Apache-2.0 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
NVIDIA has announced “NVIDIA Snapshot" (repository: ai-dynamo/snapshot), an open-source tool designed to save and restore the initialization state of GPU workloads in Kubernetes environments. NVIDIA Snapshot is developed in Go and is provided under the Apache-2.0 license. This tool provides the ability to capture the fully initialized CPU and GPU memory states of Pods running on Kubernetes and restore them on compatible infrastructure.
The most significant impact for users is the ability to dramatically reduce inference engine cold start times from minutes to seconds. By restoring from a fully initialized state, users can skip model weight loading, torch.compile compilation, and CUDA graph capture processes. In tested environments, restoration times ranged from 3.6 seconds to 42.7 seconds, achieving an 8x to 36x speedup compared to cold starts.
Key Changes
High-Speed Checkpoint & Restore for CPU and GPU Memory States
NVIDIA Snapshot captures the process state of fully initialized inference workers and restores them via Kubernetes custom resources. This allows users to skip time-consuming steps included in traditional startup processes, such as initialization, model weight loading, memory allocation, kernel compilation, and CUDA graph capture.
This functionality impacts platform engineers and developers operating inference infrastructure on Kubernetes. Creating a checkpoint (PodSnapshot or SnapshotJob) is a one-time preparation step per model and configuration, with processing times ranging from 15 seconds to 5.5 minutes. During restoration, simply specify nvidia.com/restore-from in the annotations of a new Pod to trigger the process. To prevent artifact bloat, the release of the Key-Value (KV) cache is automatically triggered immediately before checkpoint creation.
Support for Multi-GPU and Large MoE Models
It includes the ability to save and restore workloads spanning multiple GPUs within a single node as a single checkpoint artifact. It supports parallelization techniques such as Tensor Parallelism (TP), Data-Expert Parallelism (DEP), and Decode Context Parallelism (DCP).
This significantly impacts users operating configurations with checkpoints up to 1 TB in size, such as large Mixture-of-Experts (MoE) models using 8 GPUs. In validation using NVIDIA B200 GPUs and VAST (NFS) storage, GPU state restoration (CUDA restore) for an 8-GPU configuration took 18 to 29 seconds, and CPU process restoration using CRIU took 4.2 to 10.8 seconds, allowing even large models to complete restoration in under 43 seconds.
Improved Startup Performance for vLLM, SGLang, and TensorRT LLM
Configuration samples are provided for major inference engines including vLLM, SGLang, and NVIDIA TensorRT LLM (TRT-LLM), enabling fast startup for a wide range of models from small 600 million parameter models to ultra-large models.
- vLLM 0.31.0:
Qwen3-0.6B(TP1) restore time is 4.66 seconds (22.40x faster than the 104.43-second cold start),DeepSeek-V4.1-Flash(TEP8) restore time is 30.96 seconds (35.87x faster than the 1110.65-second cold start). - SGLang 0.5.21:
Qwen3-0.6B(TP1) restore time is 3.59 seconds (24.26x faster),DeepSeek-V4-Pro-NVFP4(TEP8) restore time is 42.69 seconds (14.56x faster). - TensorRT LLM 1.3.0:
gpt-oss-120b(TP1) restore time is 7.79 seconds (25.75x faster).
When updating existing inference processes, a small handshake process must be implemented on the workload side to stop operations before checkpointing and resume them after restoration.
Supported Models and Hardware
NVIDIA Snapshot validation environments and supported hardware, inference engines, and model configurations are as follows.
Validation Environment
- GPU: NVIDIA B200, NVIDIA HGX B300
- NVIDIA Driver: 615.71.09
- Storage Backend: VAST (NFS)
Supported Inference Engines
- vLLM: Version 0.31.0
- SGLang: Version 0.5.21
- TensorRT LLM (TRT-LLM): Version 1.3.0
Supported Models, Parallelization, and Precision
Checkpointing and restoration are possible for a wide range of architectures, from small dense models to ultra-large Mixture-of-Experts (MoE) models using NVFP4 precision. It supports configurations ranging from single-GPU (TP1) to multi-GPU configurations using 8 GPUs (TEP8).
Specific combinations of models and configurations that have been benchmarked and verified are as follows:
- Qwen3-0.6B: Tensor Parallelism TP1 (vLLM, SGLang, TRT-LLM)
- Qwen3-14B: Tensor Parallelism TP1 / TP2 (vLLM, SGLang, TRT-LLM)
- gpt-oss-120b: Tensor Parallelism TP1 / TP4 (vLLM, SGLang, TRT-LLM)
- GLM-5.3-NVFP4: Tensor-Expert Parallelism TEP8 (vLLM, SGLang, TRT-LLM)
- DeepSeek-V4.1-Flash: Tensor-Expert Parallelism TEP8 (vLLM, SGLang)
- DeepSeek-V4-Pro-NVFP4: Tensor-Expert Parallelism TEP8 (vLLM, SGLang)
For multi-GPU workloads, even when combining strategies such as Data-Expert Parallelism (DEP) or Decode Context Parallelism (DCP) alongside Tensor Parallelism (TP), the state of multiple GPUs within a single node can be saved and instantly restored as a single artifact.
How to Get It
NVIDIA Snapshot is installed as a single Helm release on a Kubernetes cluster. It consists of two components: an operator that manages the control plane and a node agent (DaemonSet) that runs with elevated privileges on GPU nodes. It is recommended to install Snapshot in a dedicated namespace and run GPU workloads in a separate namespace.
Installation Procedure Overview
Deployment is possible via the official repository using the following two methods:
- Deployment via Release Package (Recommended): Deploy to the cluster via Helm using the publicly available release images.
- Deployment from Source Code: Build container images locally and deploy them to the cluster.
Operation via Custom Resources
After installation, checkpointing and restoration are controlled via standard Kubernetes Custom Resource (CRD) operations:
PodSnapshot(namespace-scoped): A resource that requests a checkpoint for a running GPU Pod.SnapshotJob(namespace-scoped): A self-contained job that starts a Pod from a defined template and automatically captures a checkpoint once initialization is complete.PodSnapshotContent(cluster-scoped): A system resource managed by the operator that links physical checkpoint artifacts toPodSnapshot.nvidia.com/restore-from(annotation): Specified in the annotations of a newly created Pod to trigger state restoration from aPodSnapshotin the same namespace.
For details on operational procedures and configuration steps for vLLM, SGLang, and TensorRT LLM, please refer to the documentation in the NVIDIA Snapshot repository (docs/get-started/quickstart.md and docs/using-snapshot/README.md).
Related Articles
- NVIDIA Releases AICR for Validated Kubernetes Dynamo Runtimes
- Unsloth 0.1.912-beta: Multi-LoRA Batching and Faster Startup
- Kimi-K3.1-Pro 1M-Context Multimodal Agent Model: ~1128GB Memory
- Ling-3.0-flash-Sante-fp8 Text Generation Model: ~144GB Memory
What to Read Next
- Engines mentioned in this article → vLLM / SGLang
- Learn about the publisher → NVIDIA: models, licenses and articles
- Models NVIDIA has released → MetricTrail: File List / Cosmos3-H-Dreams Video Generation Model: 48GB+ VRAM, File List / NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF Text Generation Model: 8GB+ VRAM

