NVIDIA AIPerf: Benchmarking LLM Inference at Scale

September 19, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

NVIDIA Developer has announced “NVIDIA AIPerf", a tool aimed at benchmarking large-scale LLM inference. It adopts a multi-process architecture that bypasses single-process limitations and Python GIL bottlenecks, enabling performance measurement under workloads close to real-world operations.

Claims and Evidence

NVIDIA AIPerf is reported to be a completely rewritten successor to the traditional GenAI-Perf, designed to prevent client-side performance degradation under high-concurrency environments. According to the presenter’s explanation, it achieves accurate server benchmarking by employing a multi-process system where worker processes generate load, a separate record processor service handles results, and ZMQ coordinates them.

Supported features and datasets include the following:

• Over 15 endpoint types (chat, response, NIM ranking, image generation, etc.)
• Public datasets (such as ShareGPT) and trace replay formats (such as Mooncake, Baseten, and WEKA AgentX)
• Load control tailored to traffic characteristics (arrival patterns such as steady, Poisson, and gamma distributions, alongside burstiness adjustments)

Core metrics measured include TTFT (Time to First Token), ITL (Inter-Token Latency), request latency, and output token throughput, reported with percentile resolution (p25, p50, p75, p90, p95, p99) as well as minimum, maximum, average, and standard deviation. Additionally, in environments where DCGM and pynvml are available, GPU power consumption, utilization, and memory consumption telemetry are collected simultaneously.

Prerequisites

The conditions for the tutorial and verification in the material are as follows:

• Target model: Qwen/Qwen3-0.6B
• Server software: vLLM (vllm/vllm-openai:latest), with --reasoning-parser qwen3 enabled as an inference parser
• Hardware: Single GPU environment (--gpus all)
• Installation method: Installation using uv (uv tool install aiperf or uv pip install aiperf within a virtual environment)
• Note on aarch64 environments: Since the crick dependency is built from source, a C toolchain (build-essential for Debian/Ubuntu, Development Tools for RHEL) is required

Reproducible Scope

Procedures are published for starting the server and executing profiles using both static benchmarks and Poisson arrival patterns.

Server startup command:

docker pull vllm/vllm-openai:latest
docker run --gpus all -p 8000:8000 -e HF_TOKEN vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6B \
--reasoning-parser qwen3 \
--host 0.0.0.0 --port 8000

Static benchmark execution command:

aiperf profile \
--model Qwen/Qwen3-0.6B \
--endpoint-type chat \
--streaming \
--url localhost:8000 \
--synthetic-input-tokens-mean 128 \
--synthetic-input-tokens-stddev 0 \
--output-tokens-mean 128 \
--output-tokens-stddev 0 \
--extra-inputs min_tokens:128 \
--extra-inputs ignore_eos:true

Execution command with dynamic load patterns (Poisson distribution):

aiperf profile \
--model Qwen/Qwen3-0.6B \
--endpoint-type chat \
--streaming \
--url localhost:8000 \
--request-rate 10 \
--arrival-pattern poisson \
--synthetic-input-tokens-mean 512 \
--synthetic-input-tokens-stddev 128 \
--output-tokens-mean 128 \
--output-tokens-stddev 32 \
--random-seed 42 \
--request-count 200

Results are saved in CSV and JSON formats in addition to console output.

Sources

• NVIDIA Developer – Benchmarking LLM Inference at Scale with AIPerf