LLM Benchmark Glossary: How to Read the Scores

September 18, 2026

What the benchmark scores quoted in our articles actually measure, how to read the number, and whether higher or lower is better. This is the same glossary our pipeline hands to the article writer: a benchmark that is not listed here is never explained in an article, only compared. Definitions are maintained by hand in the site’s source code, not generated by an AI model.

Two things to keep in mind. First, error rates (WER, CER) and perplexity are lower-is-better — reading them like accuracy flips the conclusion. Second, scores from different model cards are usually not directly comparable: prompt format, number of samples and judging method differ between reports.

Coding

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
HumanEval HumanEval+, EvalPlus Whether the model can implement short Python functions from a docstring specification; graded automatically with unit tests. Pass@1 = share of problems solved on the first attempt (%) Higher
MBPP MBPP+ Whether the model can solve entry- to intermediate-level Python tasks. Pass@1 (%) Higher
BigCodeBench — Practical Python tasks that require calling several libraries correctly. Pass@1 (%) Higher
LiveCodeBench — Fresh competitive-programming problems collected after a cutoff date, designed to avoid training-data contamination. Pass@1 (%) Higher

Coding (real repositories)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
SWE-bench SWE-bench Verified, SWE-bench Pro, SWE-bench Multimodal Whether the model can fix real GitHub issues by editing a repository; a hidden test suite decides. Resolved = share of issues fixed (%) Higher
Aider Polyglot Aider Whether the model can fix code in several languages by editing existing files. Accuracy (%) Higher

Coding (competitive)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
Codeforces CodeForces Competitive-programming skill. Rating (Elo) on the same scale as human contestants — not a percentage Higher

Agents (terminal)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
Terminal-Bench TerminalBench, Terminal Bench Whether the model can complete a task end-to-end by running commands in a real terminal. Pass@1 (%) Higher

Agents (tool use)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
τ-bench tau-bench, TAU-bench, tau2-bench Whether the model can follow business policies while talking to a user and calling tools. Success rate (%) Higher
BFCL Berkeley Function Calling Leaderboard, Berkeley Function-Calling Leaderboard How accurately the model chooses functions (tools) and builds their arguments. Accuracy (%) Higher

Agents (research)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
GAIA — Real-world research questions that require tools such as search and file handling. Accuracy (%) Higher

Agents (GUI)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
OSWorld — Whether the model can complete tasks by operating a real operating-system desktop. Success rate (%) Higher

Math

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
GSM8K — Grade-school word problems solved step by step; close to saturated for top models, so differences are small. EM = exact match of the final answer (%) Higher
MATH MATH-500, MATH500 High-school to competition-level mathematics. EM (%) Higher

Math (competition)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
AIME AIME 2024, AIME 2025 Problems from the American Invitational Mathematics Examination, a US olympiad qualifier. Pass@1 (%). Each exam has only 15 problems, so one problem moves the score a lot Higher

Multilingual

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
MGSM — GSM8K problems translated into many languages, to see whether math ability holds outside English. EM (%) Higher

General knowledge

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
MMLU — Multiple-choice questions across 57 subjects; the classic measure of broad, shallow knowledge. Top models are close together. Accuracy (%) Higher

General knowledge and reasoning

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
MMLU-Pro MMLU Pro A harder MMLU with ten answer choices and more reasoning-heavy questions. EM (%) Higher
AGIEval — Exams written for humans, such as university entrance and professional qualification tests. EM (%) Higher

Hard science questions

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
GPQA Diamond GPQA-Diamond, GPQA PhD-level physics, chemistry and biology questions written to resist web search (Diamond is the hardest subset). Pass@1 (%) Higher

Extremely hard questions

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
HLE Humanity’s Last Exam, Humanitys Last Exam Expert-written questions across many fields; every model still scores low, so a few dozen percent is already high. Pass@1 (%) Higher

Reasoning

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
BBH BIG-Bench Hard The BIG-Bench tasks on which models of the time trailed humans. EM (%) Higher
BBEH BIG-Bench Extra Hard A harder successor to BBH, created after BBH saturated. EM (%) Higher

Expert knowledge

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
SuperGPQA — Graduate-level questions across a much wider range of disciplines. EM (%) Higher

Multilingual (Chinese)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
C-Eval CEval, CMMLU Multiple-choice academic and professional questions in Chinese. EM (%) Higher

Reading comprehension

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
DROP — Reading questions that require picking numbers out of a passage and adding, subtracting or comparing them. F1 (%) Higher

Commonsense reasoning

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
HellaSwag — Choosing the natural continuation of a sentence. An old benchmark, saturated for top models. Accuracy (%) Higher
Winogrande WinoGrande Resolving which noun a pronoun refers to from context. Accuracy (%) Higher
ARC-Challenge ARC-c, AI2 Reasoning Challenge Grade-school science questions that are hard to answer by retrieval alone. Accuracy (%) Higher

Factuality

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
SimpleQA SimpleQA-Verified Short factual questions with a single correct answer; tests whether the model makes things up instead of admitting ignorance. Accuracy (%) Higher
TruthfulQA — Whether the model avoids repeating common misconceptions. Accuracy (%) Higher

Instruction following

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
IFEval — Whether formal instructions such as “under 300 words” or “answer in JSON” are followed, independent of content. Compliance rate (%) Higher

Chat quality

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
Arena-Hard ArenaHard, MT-Bench, MTBench Responses to difficult chat prompts, judged relatively by another LLM. Win rate or score. The judge is also an LLM, so absolute values are not comparable across reports Higher

Long context

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
LongBench LongBench-v2, LongBench-V2 Summarising, answering questions about and understanding code in long documents. EM/F1 (%) Higher
RULER — How much of the advertised context length is actually usable. Accuracy (%) Higher
Needle in a Haystack NIAH, Needle-in-a-Haystack Whether one sentence hidden somewhere in a long context can be retrieved. Retrieval rate (%) Higher

Image understanding

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
MMMU MMMU-Pro College-level questions that include charts and photos, answered from the image. EM (%) Higher

Image understanding (documents)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
DocVQA — Answering questions about document images such as invoices and forms. ANLS / accuracy (%) Higher

Image understanding (charts)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
ChartQA — Reading values and trends from chart images. Accuracy (%) Higher

Image understanding (math)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
MathVista — Math problems that involve figures and graphs, solved from the image. Accuracy (%) Higher

Image understanding (grounding)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
RefCOCO RefCOCO+, RefCOCOg Locating an object in an image from a description such as “the second red car from the left”. Acc@0.5 = correct if overlap (IoU) with the reference box is at least 0.5 (%) Higher

Image understanding (text)

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
OCRBench — Reading text embedded in images. Score Higher

Speech recognition

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
WER Word Error Rate Word-level error rate of transcriptions. Lower is better (%) Lower
CER Character Error Rate Character-level error rate of transcriptions. Lower is better (%) Lower

Raw language-model accuracy

→ Scroll horizontally to see all columns

Benchmark Also written as What it measures How to read the number Better
Perplexity PPL How well the model predicts the next token; commonly used to compare quality loss from quantization. Lower is better Lower

45 benchmarks. Last updated 2026-09-18 (JST). See also our quantization and model-format glossary.