TensorRT Edge-LLM Accelerates MLPerf Agentic Benchmark
Overview
On September 16, 2026, NVIDIA published measurement results using the edge AI inference platform “TensorRT Edge-LLM" on the standard edge AI agent benchmark “MLPerf Inference v6.1 Edge Agentic". Running the Qwen3.6-27B model on a single NVIDIA Jetson AGX Thor Developer Kit (128 GB unified memory, MAXN power mode), it recorded an output throughput of 52.33 tokens/s and completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds. This is 6.4 times faster compared to the reference execution using llama.cpp (2 hours and 37 minutes).
Announcement Details
Published measurement results by NVIDIA represent actual empirical data from local processing of agentic AI workloads handling large-capacity, multi-turn conversations on a single Jetson AGX Thor Developer Kit.
Benchmark Results and Comparison with Conventional Methods
Key measurement metrics for the combination of TensorRT Edge-LLM and Qwen3.6-27B on the MLPerf v6.1 Edge Agentic benchmark are as follows:
- Output throughput: 52.33 tokens/s
- Median Time to First Token (TTFT): 247.12 ms
- Median Time per Output Token (TPOT): 14.68 ms
- Overall accuracy via BFCL (Berkeley Function Calling Leaderboard) v4: 87.94%
As a comparison point, in the reference execution on Jetson AGX Thor using llama.cpp published by MLCommons, the total completion time using Qwen3.6-27B (Q4_K_M quantization) was 2 hours and 37 minutes. In contrast, execution using TensorRT Edge-LLM (using NVFP4 quantization, etc.) completed in 24 minutes and 36 seconds, achieving a 6.4x speedup based on completion time.
Three Key Technologies Accelerating Processing
This performance enhancement is achieved through the combination of three technical approaches that unlock the capabilities of the Jetson AGX Thor powered by the Blackwell GPU.
-
NVFP4 Quantization and FP8 KV Cache
Low-batch LLM decoding on edge platforms is primarily constrained by DRAM bandwidth. By applying the 4-bit floating-point format “NVFP4″—supported by the Blackwell GPU of Jetson AGX Thor—to weights and activations (including the language model head), and adopting FP8 for the KV cache, memory bandwidth load is reduced while minimizing the model’s memory footprint. This leaves a large area on the 128 GB unified memory for long contexts and speculative decoding states. -
Reuse of KV Cache and Recurrent States Between Agent Turns
Because AI agents alternate between tool execution results and user inputs, the total prompt length (shared history) increases in each turn. TensorRT Edge-LLM identifies prompt prefixes common to past conversations and reuses cached attention KV pages. It restores recurrent states and partial KV page states tailored to the Qwen3.6 hybrid architecture, prefilling only the newly added suffix. In this workload, approximately 96% of the total 13.6 million prompt tokens were served from the hot cache, with only about 500,000 tokens actually prefilled. -
Tree-based Multi-Token Prediction (Tree-based MTP)
In methods where a draft model predicts multiple future tokens and the target model verifies them in a single forward pass, candidate search using a tree structure is adopted instead of traditional linear MTP. By organizing high-probability token candidates into a tree structure for verification, multiple tokens are generated at once. This proves highly effective in function calling tasks where certain patterns, such as tool specifications and JSON grammatical structures, are easy to predict. Using a configuration of 8 draft steps, top-2 candidates at each depth, and a 16-node verification tree, decoding performance was further improved by approximately 40% compared to linear MTP with 3 draft steps.
Benchmark Configuration and Evaluation Method
Using OpenAI-compatible endpoints, the MLPerf Edge Agentic benchmark evaluates agent AI performance and accuracy across two phases:
- Performance Phase: Consists of 20 conversations mimicking the actual behavior of a software engineering agent, totaling 1,007 generation turns. As conversations progress, the context length grows, reaching up to approximately 23.5K tokens. Inline accuracy based on IoU (Intersection of Union) is measured simultaneously.
- Accuracy Phase: Uses prompts from the Berkeley Function Calling Leaderboard (BFCL) v4 to verify whether agents can accurately select functions and generate correct arguments while avoiding unnecessary tool calls (evaluated for edge environments with single-turn and reasoning functions off).
Background
According to the materials, the operational model of AI agents is shifting from traditional chatbots responding to a single query to multi-turn workflows that step through procedures, select tools, and continue thinking while evaluating execution results. Such agentic AI is increasingly being deployed not only in cloud data centers, but also in vehicles, robots, and various edge devices.
Because multi-turn tool utilization and long-context processing impose heavy loads on edge devices with limited power and memory space, high token generation speeds, efficient processing of shared history, and accurate function calling capabilities are strongly required.
Impact on Local LLM Users
For developers and engineers building open-weight model inference environments on local or edge environments, this announcement signifies the provision of concrete implementation models and optimized environments.
Edge Agent Operation and Advancement of Quantization
Since the NVFP4 quantized checkpoint for the model used, “Qwen3.6-27B," is published on Hugging Face, it can be tested in local environments provided the target hardware is available. Additionally, development teams can use the provided quantized checkpoints or perform Post-Training Quantization (PTQ) on any development system prior to deployment.
Compared to conventional Q4_K_M execution on CPU/GPU environments via llama.cpp, it demonstrates that combining optimization algorithms such as context reuse (KV cache reuse) and tree-based MTP can drastically reduce execution times in long-context multi-turn conversations.
Build and Reproduction Steps
The code and configurations used in this MLPerf submission are published in the TensorRT Edge-LLM repository under the release/0.9.1-mlpinf branch. Local reproduction steps and usage commands documented in the materials are as follows:
- Clone Repository and Initialize Submodules
git clone --branch release/0.9.1-mlpinf \
https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git submodule update --init --recursive
- Download Calibrated NVFP4 Checkpoint
huggingface-cli download \
centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf \
--local-dir "$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf"
- Export Checkpoint and Build TensorRT Engine
$VENV/bin/python -m tensorrt_edgellm.scripts.export \
"$WORK/Qwen3.6-27B-NVFP4-W4A4-mlpinf" \
"$WORK/onnx" \
--mtp-tree-base --skip-visual
- Start OpenAI-Compatible TensorRT Edge-LLM Server
export REPO="$PWD"
export VENV=/path/to/venv-edgellm-export
export WORK=/path/to/mlperf-artifacts
bash mlperf/serve_edgellm.sh
- Setup MLCommons Endpoint Harness and Run Benchmark
git clone https://github.com/mlcommons/endpoints.git
cd endpoints
python3.12 -m venv.venv
source.venv/bin/activate
pip install -e ".[dev,bfcl]"
inference-endpoint benchmark from-config \
--config "$REPO/mlperf/config.yaml"
Through the above steps, users can load the published open model checkpoint (centml/Qwen3.6-27B-NVFP4-W4A4-mlpinf) and execute a similar local inference environment and benchmark evaluation.