{"id":1815,"date":"2026-09-19T04:08:18","date_gmt":"2026-09-18T19:08:18","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/19\/nvidia-aiperf-benchmarking-llm-inference\/"},"modified":"2026-09-19T04:08:18","modified_gmt":"2026-09-18T19:08:18","slug":"nvidia-aiperf-benchmarking-llm-inference","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/19\/nvidia-aiperf-benchmarking-llm-inference\/","title":{"rendered":"NVIDIA AIPerf: Benchmarking LLM Inference at Scale"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Publisher<\/td>\n<td>NVIDIA Developer<\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-09-19<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code at collection time. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>NVIDIA Developer has announced &#8220;NVIDIA AIPerf&#8221;, a tool aimed at benchmarking large-scale LLM inference. It adopts a multi-process architecture that bypasses single-process limitations and Python GIL bottlenecks, enabling performance measurement under workloads close to real-world operations.<\/p>\n<h2>Claims and Evidence<\/h2>\n<p>NVIDIA AIPerf is reported to be a completely rewritten successor to the traditional GenAI-Perf, designed to prevent client-side performance degradation under high-concurrency environments. According to the presenter&#8217;s explanation, it achieves accurate server benchmarking by employing a multi-process system where worker processes generate load, a separate record processor service handles results, and ZMQ coordinates them.<\/p>\n<p>Supported features and datasets include the following:<\/p>\n<ul>\n<li>Over 15 endpoint types (chat, response, NIM ranking, image generation, etc.)<\/li>\n<li>Public datasets (such as ShareGPT) and trace replay formats (such as Mooncake, Baseten, and WEKA AgentX)<\/li>\n<li>Load control tailored to traffic characteristics (arrival patterns such as steady, Poisson, and gamma distributions, alongside burstiness adjustments)<\/li>\n<\/ul>\n<p>Core metrics measured include TTFT (Time to First Token), ITL (Inter-Token Latency), request latency, and output token throughput, reported with percentile resolution (p25, p50, p75, p90, p95, p99) as well as minimum, maximum, average, and standard deviation. Additionally, in environments where DCGM and pynvml are available, GPU power consumption, utilization, and memory consumption telemetry are collected simultaneously.<\/p>\n<h2>Prerequisites<\/h2>\n<p>The conditions for the tutorial and verification in the material are as follows:<\/p>\n<ul>\n<li>Target model: Qwen\/Qwen3-0.6B<\/li>\n<li>Server software: vLLM (<code>vllm\/vllm-openai:latest<\/code>), with <code>--reasoning-parser qwen3<\/code> enabled as an inference parser<\/li>\n<li>Hardware: Single GPU environment (<code>--gpus all<\/code>)<\/li>\n<li>Installation method: Installation using <code>uv<\/code> (<code>uv tool install aiperf<\/code> or <code>uv pip install aiperf<\/code> within a virtual environment)<\/li>\n<li>Note on aarch64 environments: Since the <code>crick<\/code> dependency is built from source, a C toolchain (<code>build-essential<\/code> for Debian\/Ubuntu, <code>Development Tools<\/code> for RHEL) is required<\/li>\n<\/ul>\n<h2>Reproducible Scope<\/h2>\n<p>Procedures are published for starting the server and executing profiles using both static benchmarks and Poisson arrival patterns.<\/p>\n<p>Server startup command:<\/p>\n<pre><code class=\"language-bash\">docker pull vllm\/vllm-openai:latest\ndocker run --gpus all -p 8000:8000 -e HF_TOKEN vllm\/vllm-openai:latest \\\n--model Qwen\/Qwen3-0.6B \\\n--reasoning-parser qwen3 \\\n--host 0.0.0.0 --port 8000\n<\/code><\/pre>\n<p>Static benchmark execution command:<\/p>\n<pre><code class=\"language-bash\">aiperf profile \\\n--model Qwen\/Qwen3-0.6B \\\n--endpoint-type chat \\\n--streaming \\\n--url localhost:8000 \\\n--synthetic-input-tokens-mean 128 \\\n--synthetic-input-tokens-stddev 0 \\\n--output-tokens-mean 128 \\\n--output-tokens-stddev 0 \\\n--extra-inputs min_tokens:128 \\\n--extra-inputs ignore_eos:true\n<\/code><\/pre>\n<p>Execution command with dynamic load patterns (Poisson distribution):<\/p>\n<pre><code class=\"language-bash\">aiperf profile \\\n--model Qwen\/Qwen3-0.6B \\\n--endpoint-type chat \\\n--streaming \\\n--url localhost:8000 \\\n--request-rate 10 \\\n--arrival-pattern poisson \\\n--synthetic-input-tokens-mean 512 \\\n--synthetic-input-tokens-stddev 128 \\\n--output-tokens-mean 128 \\\n--output-tokens-stddev 32 \\\n--random-seed 42 \\\n--request-count 200\n<\/code><\/pre>\n<p>Results are saved in CSV and JSON formats in addition to console output.<\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/developer.nvidia.com\/blog\/benchmarking-llm-inference-at-scale-with-aiperf\/\">NVIDIA Developer &#8211; Benchmarking LLM Inference at Scale with AIPerf<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Learn about NVIDIA AIPerf, a tool designed for benchmarking large-scale LLM inference with a multi-process architecture avoiding Python GIL bottlenecks.<\/p>\n","protected":false},"author":1,"featured_media":1814,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1528],"tags":[1582,695,1584,1547,592,524],"class_list":["post-1815","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-reports","tag-aiperf-en","tag-nvidia-en","tag-nvidia-developer-en","tag-verified","tag-vllm-en","tag--en"],"lang":"en","translations":{"en":1815,"ja":1813},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/1815","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=1815"}],"version-history":[{"count":0,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/1815\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/1814"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=1815"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=1815"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=1815"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}