Quantization and Model Formats: What Q4_K_M Actually Means

A glossary of the quantization and model-format terms you meet when running models locally, in the depth needed to read our articles and the VRAM quick reference. The definitions are written by the site’s editors, not generated by an AI model. For benchmark scores, see the benchmark glossary.

What quantization is

Model weights (parameters) are stored at 16 bits each (BF16 / FP16) or 32 bits during training. Quantization re-encodes them in fewer bits — typically 4 or 8 — to shrink the file and the memory needed to run it. Fewer bits means smaller and faster, at a gradual cost in output quality.

A rough rule for memory: parameters × bits per weight ÷ 8. A 7-billion-parameter (7B) model is about 14GB in BF16, about 7GB at 8 bits and about 3.5–4GB at 4 bits. The “Hardware Requirements” tables in our articles and the quick reference take the actual size of the distributed files and add 20% as runtime overhead.

Bits (approx.) Size of a 7B model Quality When to use
16-bit (BF16 / FP16) ~14GB Original When quality matters most; the baseline before quantization
8-bit (Q8_0 / FP8 / INT8) ~7GB Nearly lossless When memory is not a constraint
5–6-bit (Q5_K_M / Q6_K) ~5–6GB Slight loss Balance of quality and size
4-bit (Q4_K_M / IQ4_XS / AWQ / GPTQ) ~4GB Fine for most uses The most common choice
2–3-bit (Q3_K / IQ2 / IQ3) ~2.5–3.5GB Noticeable loss Squeezing a large model into small memory

GGUF and llama.cpp quantization names

GGUF is the model file format read by llama.cpp and the tools built on it (Ollama, LM Studio, KoboldCpp and others). One file holds the weights, tokenizer and configuration, and it runs on CPU alone or with a GPU. A Hugging Face repository named “something-GGUF” contains the same model quantized at several bit widths.

How to read a suffix such as Q4_K_M:

  • Leading number (Q4, Q5, Q8): approximate bits per weight.
  • K: the “K-quant” scheme, which handles weights in blocks and gives important parts
    more bits. Better quality than the older Q4_0 style at the same size.
  • S / M / L: size-versus-quality within the same bit width (Small / Medium / Large).
    Q4_K_M — 4-bit, K-quant, medium — is the usual “if in doubt, start here” pick.
  • IQ (IQ4_XS, IQ3_M, IQ2_XXS …): a newer family that uses an “importance matrix”
    (imatrix). Smaller than the Q family at the same bit width, though CPU inference can be a little slower. XXS, XS, S, M are size steps.
  • Q8_0: 8-bit, nearly lossless, and a common reference point.
  • imatrix: calibration data used while quantizing to decide which weights matter.
    Sometimes shipped as imatrix.dat; not needed to run the model.

Our memory estimates use measured-ish bits-per-weight figures per quantization name (for example Q4_K_M ≈ 4.8 bits, Q8_0 ≈ 8.5 bits) rather than the nominal 4 or 8, because per-block scale values add a little on top.

Formats other than GGUF

Format Typical runtime Notes
safetensors (BF16 / FP16) Transformers, vLLM, SGLang Hugging Face’s standard format: the unquantized weights, and the largest files
AWQ vLLM, SGLang, Transformers 4-bit, GPU-oriented; protects the weights that matter most based on activation statistics
GPTQ vLLM, ExLlama, Transformers 4-bit (up to 8-bit), GPU-oriented; older and very widely supported
EXL2 / EXL3 ExLlamaV2 / ExLlamaV3 NVIDIA-only; bit widths can be fractional, which helps fit large models into limited VRAM
MLX MLX on Apple Silicon Macs Built for unified memory on Macs; “mlx-community” publishes many conversions
FP8 vLLM, SGLang (Hopper / Ada and newer GPUs) 8-bit floating point; fast on recent GPUs and increasingly shipped officially
NVFP4 / MXFP4 TensorRT-LLM, vLLM (Blackwell-generation GPUs) 4-bit floating point with hardware support on the newest GPUs
bitsandbytes (bnb 4-bit / 8-bit) Transformers Quantizes on load, so no converted file is needed; usually slower
INT4 / INT8 Various Generic names for integer quantization; many of the schemes above are instances

Words for model size

  • Parameter count (7B, 27B, 552B): B means billion. Our articles convert these figures
    in code rather than letting an AI model do the arithmetic.
  • MoE (Mixture of Experts) and “A4B”-style names: 36B-A4B means 36B total parameters,
    of which 4B are active for each token. Memory is governed by the total (36B); speed behaves closer to the active count (4B). A small active count does not mean the model fits a small GPU.
  • Context length: how many input plus output tokens the model can handle at once.
    Longer contexts need more KV cache (below).

KV cache and context length — why you need more than the table says

While generating, the runtime keeps a KV cache (intermediate state for every token so far) in memory alongside the model. It grows with context length and with the number of simultaneous requests, and at very long contexts (tens to hundreds of thousands of tokens) it can rival the size of the model itself.

Our “estimated memory” figures budget a flat 20% for this. Leave more headroom if you use long contexts or serve several requests at once. Most inference engines can also quantize the KV cache itself to 8 or 4 bits.

Reading our tables

  • The memory figures in “Hardware Requirements” and the quick reference are not quoted
    from model cards    ; this site computes them from the actual file sizes (method in the editorial policy).
  • The “Your VRAM” tiers (8GB / 12GB / 16GB / 24GB …) give the smallest tier a given
    quantization fits in. With a 24GB GPU, every tier from 8GB to 24GB applies to you.
  • On CPU-only machines and Apple Silicon, system (unified) memory plays the role of VRAM.
    The numbers are read the same way.