AI Inference Performance with NVIDIA Blackwell Confidential Computing

September 23, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

NVIDIA has published the results of a validation regarding AI inference performance under Confidential Computing (CC) environments using NVIDIA Blackwell GPUs. It demonstrates that by using memory-encrypted confidential virtual machines (CVMs), confidential GPUs, and encrypted NVIDIA NVLink, high-performance AI inference can be executed while protecting sensitive data and model contexts.

Claims and Basis

According to NVIDIA’s announcement, when inference frameworks such as TensorRT LLM are optimized for confidential computing environments, the impact on inference performance can be kept to a minimum even with security features enabled. Measurements by NVIDIA’s engineering team showed that in DeepSeek-R1 model inference using NVIDIA Blackwell GPUs, enabling confidential computing (CC on) maintained 96.1% to 98.2% of the output token throughput compared to when it was disabled (CC off), and kept the average time per output token (TPOT) latency overhead within a range of 1.2% to 4.3%.

This result is based on a comparative experiment using the following workload configuration.

Workload Configuration

Measurement Metrics

To quantify the overhead caused by CC, the presenters use the following metrics:
– Output throughput retained: $100 \times \text{(Output tokens/s with CC on} \div \text{Output tokens/s with CC off)}$
– Latency overhead Time Per Output Token (TPOT): $100 \times (\text{TPOT with CC on} \div \text{TPOT with CC off} – 1)$

Hardware Requirements

The hardware and software configurations used for the validation are as follows:

• Hardware: 1 NVIDIA DGX B200 system (8 NVIDIA B200 GPUs)
• Platform: Intel TDX
• Host OS: Ubuntu 25.10 (Host kernel 6.17.0-20-generic)
• Guest OS: Ubuntu 24.04.4 LTS (Guest kernel 6.8.0-124-generic)
• Guest vCPUs: 256
• Guest NUMA: 2 nodes
• NVIDIA driver: 595.71.05
• VBIOS FW: 1.4.x [97.10.64.00.0C]
• GPU power limit: 1,000 W
• CUDA: 13.2
• TensorRT LLM: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22
• NCCL: v2.30
• OpenSSL: 3.6.0
• Orchestration: Docker Container + NVIDIA Container Toolkit

What Can Be Reproduced Locally

This document shows specific optimization methods introduced by TensorRT LLM to suppress performance degradation under confidential computing (CC) environments. These implementations can potentially be reviewed through the following pull requests (PRs) on GitHub:

• TensorRT LLM PR #11573: In the B200 CC configuration, GPUs cannot directly access protected CVM memory, so host-to-device transfers go through a software-encrypted bounce buffer. This can cause some copies to block the calling thread. To address this issue, TensorRT LLM optimizes by moving repeated token and sampling data readbacks to an asynchronous worker to prevent protected copies from blocking the main scheduler during decoding. It also avoids unconditionally using pinned memory for host-to-device transfers, instead making CC-aware memory selections and choosing pageable memory for affected paths.

• TensorRT LLM PR #11657: Kernel autotuners typically use CUDA events to compare candidate tactics, but in the tested CC configuration, CUDA event timestamps produced unstable signals, which could cause the autotuner to select slower tactics. As a countermeasure, TensorRT LLM uses the GPU’s %globaltimer for tactic measurements in CC environments (continuously using CUDA events outside of CC environments).

Additionally, users are guided to refer to the NVIDIA Trusted Computing documentation regarding deployment plans for confidential computing. However, it should be noted that the environment used for validation is based on specific datacenter hardware, the NVIDIA DGX B200 system.

What the Material Does Not Cover

This material does not mention the following points:

• Confidential computing performance and behavior on GPU architectures other than the Blackwell generation.
• Specific costs incurred when building and operating a confidential computing environment.
• Detailed benchmark results in other software and hardware environments, such as different operating systems, kernels, or different NUMA configurations.
• Detailed criteria on how to specifically select the optimal communication algorithm according to message size, topology, and workload characteristics in situations where NVLS (NVLink SHARP) multicast is unavailable.

Related Articles

• NVIDIA AIPerf: Benchmarking LLM Inference at Scale
• NVIDIA Topograph: Open Source Cluster Topology Toolkit
• NVIDIA Dynamo-Triton 26.07 Adds Multi-Device Inference
• NVIDIA Accelerates Dropless MoE Training in JAX

Sources

• https://developer.nvidia.com/blog/enabling-private-high-performance-production-ai-inference-with-nvidia-confidential-computing/

Update History

• 2026-09-23: Rebuilt the article (details are in the Japanese edition).
• 2026-09-23: Rebuilt the article (details are in the Japanese edition).