<\/p>\n
| Item<\/th>\n | Value<\/th>\n<\/tr>\n<\/thead>\n |
|---|---|
| Publisher<\/td>\n | NVIDIA Developer<\/td>\n<\/tr>\n |
| Published<\/td>\n | 2026-09-22<\/td>\n<\/tr>\n |
| Source type<\/td>\n | Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n Values determined by this site’s code at collection time. Dates are JST.<\/em><\/p>\n <\/p>\n On September 21, 2026, NVIDIA announced the release of version 26.07 of the inference server “NVIDIA Dynamo-Triton” (formerly NVIDIA Triton Inference Server), introducing multi-device integration for NVIDIA TensorRT. This update enables a single model instance to directly manage multiple GPUs and efficiently execute distributed inference using NCCL (NVIDIA Collective Communications Library).<\/p>\n At the core of this announcement is the integration of the “multi-device inference” feature, fully supported starting from TensorRT 11.0, into Dynamo-Triton. This abstracts away the multi-GPU inference processes that developers previously had to control individually at the inference server level.<\/p>\n In Dynamo-Triton 26.07, a single instance defined as NVIDIA published benchmark results using the video generation model “NVIDIA Cosmos 3 Nano.” This model features a 36-layer denoising transformer and represents a heavy workload processing 44,160 video tokens per generation. The benchmarks were conducted under conditions generating a 189-frame video at 1280×720 resolution and 24 FPS with 35 denoising steps.<\/p>\n This integration adopts “Ulysses context parallelism,” distributing video tokens across up to 8 GPUs. The specific performance changes are as follows:<\/p>\n To enable the multi-device feature, set In terms of implementation, engines exported from PyTorch are compiled with Torch-TensorRT and converted into TensorRT’s public distributed collective layers (reduce-scatter, all-to-all, and all-gather). Through the Ulysses mechanism, changing the split axis before and after attention processing structures each rank to process a non-overlapping subset of the full video sequence.<\/p>\n Tests were also conducted on the impact of multi-GPU distributed processing on output quality. In tests using identical seed values comparing single-device results with multi-device (2, 4, and 8 GPUs) results, it is reported that all set quality thresholds (MAE \u2264 25, PSNR \u2265 18 dB) were cleared. The 8-GPU configuration (CP8) recorded an MAE of 16.316 and a PSNR of 19.400 dB, confirming visually consistent operation.<\/p>\n Generative AI computational and memory demands increasingly exceed the capabilities a single GPU can provide. To address this, NVIDIA introduced “NVIDIA TensorRT Multi-Device Inference” starting from TensorRT 11.0. This feature allows a single TensorRT network to run across multiple GPUs via distributed collective communication using NCCL (NVIDIA Collective Communications Library), while maintaining TensorRT’s inference optimizations.<\/p>\n This integration in Dynamo-Triton 26.07 aims to bridge the gap between this multi-GPU acceleration and practically usable inference services. Particularly for workloads handling long sequences like video generation, processing time on a single GPU directly impacts user wait times, driving the need for efficient distributed methods. By letting the inference server handle the complex lifecycle management of distributed processing, NVIDIA has established an environment where developers can more easily build high-performance inference setups.<\/p>\n For engineers operating open-weight models in local environments or on in-house servers, this update significantly lowers the difficulty of building and operating multi-GPU environments. Specifically, major changes are expected in three areas:<\/p>\n Conventionally, performing inference spanning multiple GPUs required developers to write communication, synchronization, and lifecycle management between individual GPUs (ranks) at the code level. In Dynamo-Triton 26.07, these are abstracted at the server level, allowing users to enable multi-GPU inference simply by writing a configuration file ( In the Cosmos 3 Nano example shown in the materials, transformer processing speed improved by approximately 6x, and end-to-end generation time was significantly reduced from 156 seconds to 34 seconds. This directly enhances the user experience in conversational applications using generative AI and media generation workloads where trial-and-error cycles are critical. In particular, the adoption of Ulysses context parallelism enables advanced distributed methods where each GPU handles a portion of the full sequence by dynamically changing the attention processing axis. Because this can be achieved via compile-time settings without heavily altering the model structure, application to new open-weight models may proceed relatively smoothly.<\/p>\n Users gain the flexibility to choose trade-offs between resources and latency. Whether to run the same model slowly on a single GPU or invest additional GPU resources to shorten response times can be easily switched according to infrastructure conditions and service level objectives (SLO). Regarding form factor, the materials focus on local execution and self-hosted serving through Dynamo-Triton 26.07 downloadable from NGC, with no mentions forcing a migration to API services. The emphasis is on benefiting from multi-GPUs on the backend while maintaining existing workflows.<\/p>\n Note that this announcement focuses on multi-device support in the TensorRT backend, and similar integrations for other backends are not described in the materials. Furthermore, metrics such as concurrent request throughput, cost per generated video, and total cost of ownership (TCO) are explicitly stated to be outside the scope of this benchmark, requiring users to evaluate these metrics in their own environments.<\/p>\n <\/p>\n |