NVIDIA Releases AICR for Validated Kubernetes Dynamo Runtimes

NVIDIA Releases AICR for Validated Kubernetes Dynamo Runtimes

At a Glance

Item Value
Publisher guide NVIDIA: models and licenses
Publisher NVIDIA Developer
Published 2026-10-31
License NOASSERTION
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

NVIDIA has published a guide on how to build validated runtimes using NVIDIA AI Cluster Runtime (AICR), a tool designed to ensure the stable operation of the distributed inference framework NVIDIA Dynamo on Kubernetes. AICR packages runtime requirements—such as GPU driver stacks, Dynamic Resource Allocation (DRA), gang scheduling, node health signals, and Prometheus metrics—into reproducible “recipes" with fixed component versions.

The most significant change in this release is that Dynamo 1.4.2 now standardizes on a configuration using TCP for the request plane and ZMQ for KV-cache events. This enables cache-aware routing for both the native Dynamo Frontend router and paths utilizing the Gateway API Inference Extension (GAIE) and Endpoint Picker Plugin (EPP).

Breaking Changes and Deprecations

Starting with Dynamo 1.4, frontends, workers, and optional GAIE/EPP components discover each other through Kubernetes resources. Consequently, the default AICR path no longer uses NATS, and NATS installation has also been removed from the default settings of the dynamo-platform chart.

Item Old Setting/Specification New Setting/Specification
NATS Usage Installed by default Available as an option (opt-in) only
API Version v1alpha1 v1beta1 (DynamoGraphDeployment CRD)

This change impacts platform teams planning to operate Dynamo with a Kubernetes-native configuration. Specific deployments that require NATS must enable it explicitly.

Key Changes

Introduction of Two Routing Paths

Two routing paths are now available for operating Dynamo on Kubernetes depending on the use case. One path uses the “Dynamo Frontend router," which is suitable for minimal configurations with model-specific endpoints. The other uses “Gateway API / EPP," which is suitable when integrating with shared Kubernetes ingresses managed through Gateway API resources. Both paths utilize KV-cache events published by workers to achieve accurate cache-aware routing.

Introduction of Validated Recipes via AICR

AICR reduces the effort required to manage infrastructure dependencies. It supports reproducible environment setup through a five-step process: capturing a cluster snapshot, generating a Dynamo inference recipe, rendering a deployment bundle, deploying via Helmfile or GitOps, and performing post-installation runtime verification. This allows platform teams to automatically verify that complex components—such as the GPU Operator, KAI Scheduler, and Prometheus stack—are correctly configured for a specific Dynamo version.

Changes to Communication Protocols for KV-cache Events

Dynamo 1.4.2 uses TCP for request forwarding and ZMQ for transmitting KV-cache events. For example, when using vLLM workers, you must configure KV-cache events to be emitted via the ZMQ publisher using settings such as the following:

--enable-prefix-caching --kv-transfer-config \
{"enable_kv_cache_events": true, "publisher": "zmq", "endpoint": "tcp://*:5557"}

With this configuration, the Dynamo worker runtime subscribes to local vLLM event sources and republishes normalized events to the Dynamo event plane, enabling the router to make decisions based on the latest cache state.

Supported Models and Hardware

AICR’s Dynamo inference recipes fix component versions to match specific hardware environments. For instance, the AICR v1.0.0 recipe for NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems in an Amazon EKS environment selects the following validated components:

  • NVIDIA GPU Operator v26.7.1
  • NVIDIA DRA GPU Driver 0.5.0
  • KAI Scheduler v0.16.9
  • Grove v0.1.0-alpha.13
  • Dynamo Platform 1.4.2

Additionally, Dynamo supports orchestration for running inference engines such as vLLM, SGLang, and NVIDIA TensorRT LLM within pods on Kubernetes.

How to Update

The following procedure is recommended for deploying runtimes using AICR. First, capture a cluster snapshot, and then generate the inference recipe:

# Capture cluster snapshot
aicr snapshot --namespace aicr-validation --output snapshot.yaml

# Generate Dynamo inference recipe
aicr recipe --snapshot snapshot.yaml --intent inference --platform dynamo --output recipe.yaml

Next, render the deployment bundle from the recipe. If deploying locally using Helmfile, use the following commands:

# Render bundle
aicr bundle --recipe recipe.yaml [other flags] --deployer helmfile --output bundle

# Execute deployment
helmfile -f bundle/helmfile.yaml diff
helmfile -f bundle/helmfile.yaml apply

This material also describes production deployment methods using GitOps (ArgoCD or Flux).

Related Articles

What to Read Next

Sources