NVIDIA Topograph: Open Source Cluster Topology Toolkit

NVIDIA Topograph: Open Source Cluster Topology Toolkit

At a Glance

Item Value
Publisher NVIDIA Developer
Published 2026-09-23
Source type Primary source (the publisher itself)

Values determined by this site’s code at collection time. Dates are JST.

Overview

NVIDIA has announced NVIDIA Topograph, an open-source toolkit that automatically discovers and normalizes cluster network topologies, enabling schedulers to perform topology-aware placement. This tool retrieves information from cloud APIs and on-premises fabric systems and outputs it as Kubernetes node labels or Slurm configuration files, making it possible to place AI workloads in the most efficient communication domains.

Announcement Details

NVIDIA Topograph consists of two main concepts: “Providers" and “Engines." Providers are responsible for detecting cluster topologies from cloud APIs or on-premises systems and normalizing them into a common model, while engines convert that model into a format interpretable by each workload manager.

Specifically, it supports the following environments and output formats:

  • Supported Cloud Providers: Google Cloud, Lambda, Nebius, Nscale, Oracle Cloud Infrastructure (OCI), and others are integrated, with further providers currently under development.
  • On-Premises Environments: Supports InfiniBand (using ibnetdiscover), Spectrum-X, or Multi-Node NVLink (MNNVL) domains.
  • Output Formats (Engines):
    • Node labels for Kubernetes, Node Feature Discovery (NFD) resources – Topology configurations for Slurm (tree, block, or partition-based YAML format for Slurm 25.05 and later) – ConfigMaps for Slinky – Instance-oriented topology JSON

Additionally, Topograph continuously monitors cluster changes and automatically regenerates the topology view, keeping schedulers up to date without manual maintenance. It can be deployed using Helm in Kubernetes environments and is provided as native Debian or RPM packages in Slurm environments. Furthermore, even without production hardware, simulation testing using utilities like kwok-nodes is possible.

Background

AI factories are systems with constrained power supplies, and advanced optimization is essential to maximize their value. Among these, the placement of GPU workloads is an extremely critical optimization item.

Inappropriate workload placement leads to fragmented topology domains and increased communication via shared links. As a result, this not only causes reduced throughput and higher job costs, but also leads to inefficient situations where GPUs stuck in a waiting state continue to consume provisioned power without advancing the workload.

Throughout training and inference processes, GPUs continuously exchange data, making communication locality important. NVIDIA NVLink and NVLink Switch provide high-bandwidth all-to-all connectivity at the rack scale, while NVIDIA Spectrum-X Ethernet provides predictable, low-latency networking across systems and racks. Whether schedulers have an accurate and up-to-date understanding of the relationship between these GPUs and the fabric is the key to achieving efficient placement.

Impact on Local LLM Users

For engineers operating open-weight models in large clusters, the introduction of Topograph directly translates to improvements in compute resource utilization efficiency, cost performance, and “tokens per watt."

Placement Control Using Specific Labels

Topograph exposes physical network configurations as Kubernetes node labels. This makes it possible to use existing schedulers to aggregate communication-heavy workloads onto physically close resources. Specifically, labels such as the following are available:

  • fabric.topograph.run/tier-<N>: A label indicating the hierarchy moving outward, with the switch closest to the node designated as tier-0
  • accelerator.topograph.run/domain: The accelerator domain
  • accelerator.topograph.run/sub-domain: An optional nested sub-domain

For example, in Kubernetes podAffinity, specifying these labels in topologyKey allows for control that prioritizes placing Pods within specific communication domains.

Advanced Scheduling via Ecosystem Integration

Topograph collaborates with existing orchestration tools to achieve more advanced placement.

  • Integration with KAI Scheduler: When using the CNCF Sandbox project KAI Scheduler, labels provided by Topograph can be organized and utilized in a hierarchical structure (e.g., zone -> tier-1 -> tier-0 -> hostname). This enables topology-aware “gang scheduling," improving efficiency in large-scale distributed training.
  • Use of Kueue and NFD: It can also integrate with Kubernetes Kueue and Node Feature Discovery (NFD). When using the NFD engine, topology information can be exposed as NodeFeatureGroups.
  • Integration with Slinky: In environments utilizing Slinky from SchedMD (acquired by NVIDIA in December 2025), Kubernetes nodes can be mapped to Slurm slurmd Pods through Topograph’s Slinky engine, and Slurm topology data can be written out as a ConfigMap.

Operational Flexibility and Verification Environments

This tool is provided as open-source, and provider interfaces are also made public. Therefore, it is possible to develop providers for custom environments and contribute them upstream. Moreover, because it continuously monitors cluster changes and automatically updates the topology view, it prevents configuration drift caused by manual maintenance.

Furthermore, without requiring production hardware, it is also possible to test topology-aware scheduling on virtual Kubernetes nodes through simulations using utilities like kwok-nodes.

Related Articles

Sources

Update History

  • 2026-09-23: Rebuilt the article (details are in the Japanese edition).
  • 2026-09-23: Rebuilt the article (details are in the Japanese edition).

Companies and FundingInfiniBand,Kubernetes,NVIDIA,NVIDIA Developer,Slurm,Topograph,Verified