In September 2026, NVIDIA announced that optimizations for Nemotron 3 Ultra via NIM (NVIDIA Inference Microservice) have improved system throughput on 4xB200 systems by up to 2.5x compared to the baseline. Using the NIM 2.0.12 optimized serving stack, it achieves a throughput of 1,997 tokens per second at a target of 50 TPS per user.<\/p>\n
The newly released NIM 2.0.12 optimized serving stack demonstrates significant performance improvements compared to the open-source baseline configuration (NIM Off). In tests with a native 256K maximum context, the optimized NIM stack recorded 1,997 tokens per second compared to 718 tokens per second for the baseline.<\/p>\n
This performance boost is achieved by combining multiple optimization layers rather than a single feature. Key optimization techniques include auto-tuned Mixture of Experts (MoE) and Mamba kernels for the Blackwell architecture, tensor parallel execution distributing the model across 4 GPUs, prefix and model state reuse (including prefix caching and partial prefix matching), scheduler, batching, and memory tuning, and MTP (Multi-Token Prediction) speculative decoding.<\/p>\n
Additionally, NIM packages model- and GPU-aware serving settings along with verified configurations, standard APIs, and container lifecycles. Furthermore, commercial support through NVIDIA AI Enterprise, regular inference stack updates, and CVE (Common Vulnerabilities and Exposures) responses are also provided.<\/p>\n
For engineers operating open-weight models on their own NVIDIA GPU infrastructure, this announcement provides a concrete execution path to maximize inference throughput and effective performance.<\/p>\n
vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0<\/code>) are provided, along with configuration steps such as enabling speculative decoding.<\/li>\n- For performance measurement in their own traffic environments, verification commands using NVIDIA AIPerf (using Mooncake format JSONL traces, etc.) are guided, enabling the selection of Pareto points tailored to latency SLOs.<\/li>\n
- Regarding whether to focus on APIs or local execution, it is mentioned that in addition to local deployment via download from NGC, hosted APIs are also available for immediate testing.<\/li>\n<\/ul>\n
Sources<\/h2>\n