{"id":1418,"date":"2026-09-17T17:10:53","date_gmt":"2026-09-17T08:10:53","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/glossary-benchmarks-en\/"},"modified":"2026-09-18T11:57:22","modified_gmt":"2026-09-18T02:57:22","slug":"glossary-benchmarks-en","status":"publish","type":"page","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-benchmarks-en\/","title":{"rendered":"LLM Benchmark Glossary: How to Read the Scores"},"content":{"rendered":"<p>What the benchmark scores quoted in our articles actually measure, how to read the number, and whether higher or lower is better. This is the same glossary our pipeline hands to the article writer: a benchmark that is not listed here is never explained in an article, only compared. Definitions are maintained by hand in the site&#8217;s source code, not generated by an AI model.<\/p>\n<p>Two things to keep in mind. First, error rates (WER, CER) and perplexity are <strong>lower-is-better<\/strong> \u2014 reading them like accuracy flips the conclusion. Second, scores from different model cards are usually not directly comparable: prompt format, number of samples and judging method differ between reports.<\/p>\n<h2>Coding<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>HumanEval<\/td>\n<td>HumanEval+, EvalPlus<\/td>\n<td>Whether the model can implement short Python functions from a docstring specification; graded automatically with unit tests.<\/td>\n<td>Pass@1 = share of problems solved on the first attempt (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>MBPP<\/td>\n<td>MBPP+<\/td>\n<td>Whether the model can solve entry- to intermediate-level Python tasks.<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>BigCodeBench<\/td>\n<td>\u2014<\/td>\n<td>Practical Python tasks that require calling several libraries correctly.<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>LiveCodeBench<\/td>\n<td>\u2014<\/td>\n<td>Fresh competitive-programming problems collected after a cutoff date, designed to avoid training-data contamination.<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Coding (real repositories)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>SWE-bench<\/td>\n<td>SWE-bench Verified, SWE-bench Pro, SWE-bench Multimodal<\/td>\n<td>Whether the model can fix real GitHub issues by editing a repository; a hidden test suite decides.<\/td>\n<td>Resolved = share of issues fixed (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>Aider Polyglot<\/td>\n<td>Aider<\/td>\n<td>Whether the model can fix code in several languages by editing existing files.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Coding (competitive)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Codeforces<\/td>\n<td>CodeForces<\/td>\n<td>Competitive-programming skill.<\/td>\n<td>Rating (Elo) on the same scale as human contestants \u2014 <strong>not a percentage<\/strong><\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Agents (terminal)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Terminal-Bench<\/td>\n<td>TerminalBench, Terminal Bench<\/td>\n<td>Whether the model can complete a task end-to-end by running commands in a real terminal.<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Agents (tool use)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>\u03c4-bench<\/td>\n<td>tau-bench, TAU-bench, tau2-bench<\/td>\n<td>Whether the model can follow business policies while talking to a user and calling tools.<\/td>\n<td>Success rate (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>BFCL<\/td>\n<td>Berkeley Function Calling Leaderboard, Berkeley Function-Calling Leaderboard<\/td>\n<td>How accurately the model chooses functions (tools) and builds their arguments.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Agents (research)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>GAIA<\/td>\n<td>\u2014<\/td>\n<td>Real-world research questions that require tools such as search and file handling.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Agents (GUI)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>OSWorld<\/td>\n<td>\u2014<\/td>\n<td>Whether the model can complete tasks by operating a real operating-system desktop.<\/td>\n<td>Success rate (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Math<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>GSM8K<\/td>\n<td>\u2014<\/td>\n<td>Grade-school word problems solved step by step; close to saturated for top models, so differences are small.<\/td>\n<td>EM = exact match of the final answer (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>MATH<\/td>\n<td>MATH-500, MATH500<\/td>\n<td>High-school to competition-level mathematics.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Math (competition)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>AIME<\/td>\n<td>AIME 2024, AIME 2025<\/td>\n<td>Problems from the American Invitational Mathematics Examination, a US olympiad qualifier.<\/td>\n<td>Pass@1 (%). Each exam has only 15 problems, so one problem moves the score a lot<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Multilingual<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>MGSM<\/td>\n<td>\u2014<\/td>\n<td>GSM8K problems translated into many languages, to see whether math ability holds outside English.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>General knowledge<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>MMLU<\/td>\n<td>\u2014<\/td>\n<td>Multiple-choice questions across 57 subjects; the classic measure of broad, shallow knowledge. Top models are close together.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>General knowledge and reasoning<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>MMLU-Pro<\/td>\n<td>MMLU Pro<\/td>\n<td>A harder MMLU with ten answer choices and more reasoning-heavy questions.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>AGIEval<\/td>\n<td>\u2014<\/td>\n<td>Exams written for humans, such as university entrance and professional qualification tests.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Hard science questions<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>GPQA Diamond<\/td>\n<td>GPQA-Diamond, GPQA<\/td>\n<td>PhD-level physics, chemistry and biology questions written to resist web search (Diamond is the hardest subset).<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Extremely hard questions<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>HLE<\/td>\n<td>Humanity&#8217;s Last Exam, Humanitys Last Exam<\/td>\n<td>Expert-written questions across many fields; every model still scores low, so a few dozen percent is already high.<\/td>\n<td>Pass@1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Reasoning<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>BBH<\/td>\n<td>BIG-Bench Hard<\/td>\n<td>The BIG-Bench tasks on which models of the time trailed humans.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>BBEH<\/td>\n<td>BIG-Bench Extra Hard<\/td>\n<td>A harder successor to BBH, created after BBH saturated.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Expert knowledge<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>SuperGPQA<\/td>\n<td>\u2014<\/td>\n<td>Graduate-level questions across a much wider range of disciplines.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Multilingual (Chinese)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>C-Eval<\/td>\n<td>CEval, CMMLU<\/td>\n<td>Multiple-choice academic and professional questions in Chinese.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Reading comprehension<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DROP<\/td>\n<td>\u2014<\/td>\n<td>Reading questions that require picking numbers out of a passage and adding, subtracting or comparing them.<\/td>\n<td>F1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Commonsense reasoning<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>HellaSwag<\/td>\n<td>\u2014<\/td>\n<td>Choosing the natural continuation of a sentence. An old benchmark, saturated for top models.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>Winogrande<\/td>\n<td>WinoGrande<\/td>\n<td>Resolving which noun a pronoun refers to from context.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>ARC-Challenge<\/td>\n<td>ARC-c, AI2 Reasoning Challenge<\/td>\n<td>Grade-school science questions that are hard to answer by retrieval alone.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Factuality<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>SimpleQA<\/td>\n<td>SimpleQA-Verified<\/td>\n<td>Short factual questions with a single correct answer; tests whether the model makes things up instead of admitting ignorance.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>TruthfulQA<\/td>\n<td>\u2014<\/td>\n<td>Whether the model avoids repeating common misconceptions.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Instruction following<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>IFEval<\/td>\n<td>\u2014<\/td>\n<td>Whether formal instructions such as \u201cunder 300 words\u201d or \u201canswer in JSON\u201d are followed, independent of content.<\/td>\n<td>Compliance rate (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Chat quality<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Arena-Hard<\/td>\n<td>ArenaHard, MT-Bench, MTBench<\/td>\n<td>Responses to difficult chat prompts, judged relatively by another LLM.<\/td>\n<td>Win rate or score. The judge is also an LLM, so absolute values are not comparable across reports<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Long context<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>LongBench<\/td>\n<td>LongBench-v2, LongBench-V2<\/td>\n<td>Summarising, answering questions about and understanding code in long documents.<\/td>\n<td>EM\/F1 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>RULER<\/td>\n<td>\u2014<\/td>\n<td>How much of the advertised context length is actually usable.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<tr>\n<td>Needle in a Haystack<\/td>\n<td>NIAH, Needle-in-a-Haystack<\/td>\n<td>Whether one sentence hidden somewhere in a long context can be retrieved.<\/td>\n<td>Retrieval rate (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>MMMU<\/td>\n<td>MMMU-Pro<\/td>\n<td>College-level questions that include charts and photos, answered from the image.<\/td>\n<td>EM (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding (documents)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>DocVQA<\/td>\n<td>\u2014<\/td>\n<td>Answering questions about document images such as invoices and forms.<\/td>\n<td>ANLS \/ accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding (charts)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>ChartQA<\/td>\n<td>\u2014<\/td>\n<td>Reading values and trends from chart images.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding (math)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>MathVista<\/td>\n<td>\u2014<\/td>\n<td>Math problems that involve figures and graphs, solved from the image.<\/td>\n<td>Accuracy (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding (grounding)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>RefCOCO<\/td>\n<td>RefCOCO+, RefCOCOg<\/td>\n<td>Locating an object in an image from a description such as \u201cthe second red car from the left\u201d.<\/td>\n<td>Acc@0.5 = correct if overlap (IoU) with the reference box is at least 0.5 (%)<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Image understanding (text)<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>OCRBench<\/td>\n<td>\u2014<\/td>\n<td>Reading text embedded in images.<\/td>\n<td>Score<\/td>\n<td>Higher<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Speech recognition<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>WER<\/td>\n<td>Word Error Rate<\/td>\n<td>Word-level error rate of transcriptions.<\/td>\n<td><strong>Lower is better<\/strong> (%)<\/td>\n<td>Lower<\/td>\n<\/tr>\n<tr>\n<td>CER<\/td>\n<td>Character Error Rate<\/td>\n<td>Character-level error rate of transcriptions.<\/td>\n<td><strong>Lower is better<\/strong> (%)<\/td>\n<td>Lower<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Raw language-model accuracy<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Benchmark<\/th>\n<th>Also written as<\/th>\n<th>What it measures<\/th>\n<th>How to read the number<\/th>\n<th>Better<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Perplexity<\/td>\n<td>PPL<\/td>\n<td>How well the model predicts the next token; commonly used to compare quality loss from quantization.<\/td>\n<td><strong>Lower is better<\/strong><\/td>\n<td>Lower<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>45 benchmarks. Last updated 2026-09-18 (JST). See also our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">quantization and model-format glossary<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>What the benchmark scores quoted in our articles actually measure, how to read the number, and whether higher or lower is better. This is the same glossary our pipeline hands to the article writer: a benchmark that is not listed here is never explained in an article, only compared. Definitions are maintained by hand in [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-1418","page","type-page","status-publish","hentry"],"lang":"en","translations":{"en":1418,"ja":1416},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/pages\/1418","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=1418"}],"version-history":[{"count":1,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/pages\/1418\/revisions"}],"predecessor-version":[{"id":1558,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/pages\/1418\/revisions\/1558"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=1418"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}