{"id":15766,"date":"2026-11-17T11:03:59","date_gmt":"2026-11-17T02:03:59","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=15766"},"modified":"2026-11-17T11:13:29","modified_gmt":"2026-11-17T02:13:29","slug":"nvidia-snapshot-kubernetes-gpu-checkpoint-restore","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-snapshot-kubernetes-gpu-checkpoint-restore\/","title":{"rendered":"NVIDIA Snapshot: Accelerate GPU Workload Cold Starts on Kubernetes"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Publisher guide<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-nvidia-en\/\">NVIDIA: models and licenses<\/a><\/td>\n<\/tr>\n<tr>\n<td>Publisher<\/td>\n<td>NVIDIA Developer<\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-11<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>Apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>NVIDIA has announced &#8220;NVIDIA Snapshot&#8221; (repository: <code>ai-dynamo\/snapshot<\/code>), an open-source tool designed to save and restore the initialization state of GPU workloads in Kubernetes environments. NVIDIA Snapshot is developed in Go and is provided under the Apache-2.0 license. This tool provides the ability to capture the fully initialized CPU and GPU memory states of Pods running on Kubernetes and restore them on compatible infrastructure.<\/p>\n<p>The most significant impact for users is the ability to dramatically reduce inference engine cold start times from minutes to seconds. By restoring from a fully initialized state, users can skip model weight loading, <code>torch.compile<\/code> compilation, and CUDA graph capture processes. In tested environments, restoration times ranged from 3.6 seconds to 42.7 seconds, achieving an 8x to 36x speedup compared to cold starts.<\/p>\n<h2>Key Changes<\/h2>\n<h3>High-Speed Checkpoint &amp; Restore for CPU and GPU Memory States<\/h3>\n<p>NVIDIA Snapshot captures the process state of fully initialized inference workers and restores them via Kubernetes custom resources. This allows users to skip time-consuming steps included in traditional startup processes, such as initialization, model weight loading, memory allocation, kernel compilation, and CUDA graph capture.<\/p>\n<p>This functionality impacts platform engineers and developers operating inference infrastructure on Kubernetes. Creating a checkpoint (<code>PodSnapshot<\/code> or <code>SnapshotJob<\/code>) is a one-time preparation step per model and configuration, with processing times ranging from 15 seconds to 5.5 minutes. During restoration, simply specify <code>nvidia.com\/restore-from<\/code> in the annotations of a new Pod to trigger the process. To prevent artifact bloat, the release of the Key-Value (KV) cache is automatically triggered immediately before checkpoint creation.<\/p>\n<h3>Support for Multi-GPU and Large MoE Models<\/h3>\n<p>It includes the ability to save and restore workloads spanning multiple GPUs within a single node as a single checkpoint artifact. It supports parallelization techniques such as Tensor Parallelism (TP), Data-Expert Parallelism (DEP), and Decode Context Parallelism (DCP).<\/p>\n<p>This significantly impacts users operating configurations with checkpoints up to 1 TB in size, such as large Mixture-of-Experts (MoE) models using 8 GPUs. In validation using NVIDIA B200 GPUs and VAST (NFS) storage, GPU state restoration (CUDA restore) for an 8-GPU configuration took 18 to 29 seconds, and CPU process restoration using CRIU took 4.2 to 10.8 seconds, allowing even large models to complete restoration in under 43 seconds.<\/p>\n<h3>Improved Startup Performance for vLLM, SGLang, and TensorRT LLM<\/h3>\n<p>Configuration samples are provided for major inference engines including vLLM, SGLang, and NVIDIA TensorRT LLM (TRT-LLM), enabling fast startup for a wide range of models from small 600 million parameter models to ultra-large models.<\/p>\n<ul>\n<li><strong>vLLM 0.31.0<\/strong>: <code>Qwen3-0.6B<\/code> (TP1) restore time is 4.66 seconds (22.40x faster than the 104.43-second cold start), <code>DeepSeek-V4.1-Flash<\/code> (TEP8) restore time is 30.96 seconds (35.87x faster than the 1110.65-second cold start).<\/li>\n<li><strong>SGLang 0.5.21<\/strong>: <code>Qwen3-0.6B<\/code> (TP1) restore time is 3.59 seconds (24.26x faster), <code>DeepSeek-V4-Pro-NVFP4<\/code> (TEP8) restore time is 42.69 seconds (14.56x faster).<\/li>\n<li><strong>TensorRT LLM 1.3.0<\/strong>: <code>gpt-oss-120b<\/code> (TP1) restore time is 7.79 seconds (25.75x faster).<\/li>\n<\/ul>\n<p>When updating existing inference processes, a small handshake process must be implemented on the workload side to stop operations before checkpointing and resume them after restoration.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>NVIDIA Snapshot validation environments and supported hardware, inference engines, and model configurations are as follows.<\/p>\n<h3>Validation Environment<\/h3>\n<ul>\n<li><strong>GPU<\/strong>: NVIDIA B200, NVIDIA HGX B300<\/li>\n<li><strong>NVIDIA Driver<\/strong>: 615.71.09<\/li>\n<li><strong>Storage Backend<\/strong>: VAST (NFS)<\/li>\n<\/ul>\n<h3>Supported Inference Engines<\/h3>\n<ul>\n<li><strong>vLLM<\/strong>: Version 0.31.0<\/li>\n<li><strong>SGLang<\/strong>: Version 0.5.21<\/li>\n<li><strong>TensorRT LLM (TRT-LLM)<\/strong>: Version 1.3.0<\/li>\n<\/ul>\n<h3>Supported Models, Parallelization, and Precision<\/h3>\n<p>Checkpointing and restoration are possible for a wide range of architectures, from small dense models to ultra-large Mixture-of-Experts (MoE) models using NVFP4 precision. It supports configurations ranging from single-GPU (TP1) to multi-GPU configurations using 8 GPUs (TEP8).<\/p>\n<p>Specific combinations of models and configurations that have been benchmarked and verified are as follows:<\/p>\n<ul>\n<li><strong>Qwen3-0.6B<\/strong>: Tensor Parallelism TP1 (vLLM, SGLang, TRT-LLM)<\/li>\n<li><strong>Qwen3-14B<\/strong>: Tensor Parallelism TP1 \/ TP2 (vLLM, SGLang, TRT-LLM)<\/li>\n<li><strong>gpt-oss-120b<\/strong>: Tensor Parallelism TP1 \/ TP4 (vLLM, SGLang, TRT-LLM)<\/li>\n<li><strong>GLM-5.3-NVFP4<\/strong>: Tensor-Expert Parallelism TEP8 (vLLM, SGLang, TRT-LLM)<\/li>\n<li><strong>DeepSeek-V4.1-Flash<\/strong>: Tensor-Expert Parallelism TEP8 (vLLM, SGLang)<\/li>\n<li><strong>DeepSeek-V4-Pro-NVFP4<\/strong>: Tensor-Expert Parallelism TEP8 (vLLM, SGLang)<\/li>\n<\/ul>\n<p>For multi-GPU workloads, even when combining strategies such as Data-Expert Parallelism (DEP) or Decode Context Parallelism (DCP) alongside Tensor Parallelism (TP), the state of multiple GPUs within a single node can be saved and instantly restored as a single artifact.<\/p>\n<h2>How to Get It<\/h2>\n<p>NVIDIA Snapshot is installed as a single Helm release on a Kubernetes cluster. It consists of two components: an operator that manages the control plane and a node agent (DaemonSet) that runs with elevated privileges on GPU nodes. It is recommended to install Snapshot in a dedicated namespace and run GPU workloads in a separate namespace.<\/p>\n<h3>Installation Procedure Overview<\/h3>\n<p>Deployment is possible via the official repository using the following two methods:<\/p>\n<ol>\n<li><strong>Deployment via Release Package (Recommended)<\/strong>: Deploy to the cluster via Helm using the publicly available release images.<\/li>\n<li><strong>Deployment from Source Code<\/strong>: Build container images locally and deploy them to the cluster.<\/li>\n<\/ol>\n<h3>Operation via Custom Resources<\/h3>\n<p>After installation, checkpointing and restoration are controlled via standard Kubernetes Custom Resource (CRD) operations:<\/p>\n<ul>\n<li><strong><code>PodSnapshot<\/code> (namespace-scoped)<\/strong>: A resource that requests a checkpoint for a running GPU Pod.<\/li>\n<li><strong><code>SnapshotJob<\/code> (namespace-scoped)<\/strong>: A self-contained job that starts a Pod from a defined template and automatically captures a checkpoint once initialization is complete.<\/li>\n<li><strong><code>PodSnapshotContent<\/code> (cluster-scoped)<\/strong>: A system resource managed by the operator that links physical checkpoint artifacts to <code>PodSnapshot<\/code>.<\/li>\n<li><strong><code>nvidia.com\/restore-from<\/code> (annotation)<\/strong>: Specified in the annotations of a newly created Pod to trigger state restoration from a <code>PodSnapshot<\/code> in the same namespace.<\/li>\n<\/ul>\n<p>For details on operational procedures and configuration steps for vLLM, SGLang, and TensorRT LLM, please refer to the documentation in the NVIDIA Snapshot repository (<code>docs\/get-started\/quickstart.md<\/code> and <code>docs\/using-snapshot\/README.md<\/code>).<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-aicr-validated-kubernetes-dynamo-runtimes\/\">NVIDIA Releases AICR for Validated Kubernetes Dynamo Runtimes<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/unsloth-0-1-912-beta-updates\/\">Unsloth 0.1.912-beta: Multi-LoRA Batching and Faster Startup<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/kimi-k3-1-pro\/\">Kimi-K3.1-Pro 1M-Context Multimodal Agent Model: ~1128GB Memory<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ling-3-0-flash-sante-fp8-released\/\">Ling-3.0-flash-Sante-fp8 Text Generation Model: ~144GB Memory<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Engines mentioned in this article<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-sglang-en\/\">SGLang<\/a><\/li>\n<li><strong>Learn about the publisher<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-nvidia-en\/\">NVIDIA: models, licenses and articles<\/a><\/li>\n<li><strong>Models NVIDIA has released<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-metrictrail-release\/\">MetricTrail: File List<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-cosmos3-h-dreams-surgical-world-model\/\">Cosmos3-H-Dreams Video Generation Model: 48GB+ VRAM, File List<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/nvidia-nemotron-labs-dory1-4b-gguf\/\">NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF Text Generation Model: 8GB+ VRAM<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/developer.nvidia.com\/blog\/start-and-restore-gpu-workloads-in-seconds-with-nvidia-snapshot\/\">Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>NVIDIA Snapshot is an open-source tool for Kubernetes that saves and restores GPU memory states, reducing inference cold starts to just seconds.<\/p>\n","protected":false},"author":1,"featured_media":15765,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[2023,165,4832,169,2006,1547,592],"class_list":["post-15766","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-kubernetes-en","tag-moe-en","tag-nvidia-snapshot-en","tag-sglang-en","tag-tensorrt-llm-en","tag-verified","tag-vllm-en"],"lang":"en","translations":{"en":15766,"ja":15764},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15766","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=15766"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15766\/revisions"}],"predecessor-version":[{"id":18082,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15766\/revisions\/18082"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/15765"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=15766"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=15766"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=15766"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}