GPTQ Model Format Explained: Supported Engines and Models
What Is GPTQ?
GPTQ is a post-training method, published in 2022, for quantizing an already-trained LLM to about 3–4 bits. It feeds a small amount of calibration text through the model and rounds the weights one layer at a time so that the error introduced by quantization is compensated by the weights not yet quantized. On Hugging Face, GPTQ models come as repositories with GPTQ in the name (containing .safetensors files).
Why It Matters
- One of the first widely used 4-bit formats for GPUs. It became a standard way to fit a large model on a single GPU, and quantized builds exist for many models, including older ones.
- Broad engine support. vLLM, SGLang and Transformers load it. In vLLM’s compatibility table, GPTQ covers a wide range of NVIDIA GPU generations.
- Choice of bit width and group size. Repositories may offer 4-bit and 3-bit builds with different group sizes (such as
128g); smaller means a smaller file and more quality loss.
Tips for Running It Locally
- It assumes an NVIDIA GPU. On CPU or a Mac, choose GGUF or MLX. Ollama and LM Studio (llama.cpp-based) cannot load it.
- It overlaps with AWQ. Both quantize weights only to 4 bits for GPUs. If a model has both, choose by your engine’s support and by the evaluations the quantizer publishes.
- The tooling moved to GPTQModel. AutoGPTQ is no longer developed, and vLLM’s documentation points to GPTQModel.
Sources: the GPTQ paper (arXiv:2210.17323), vLLM’s quantization documentation and the ModelCloud/GPTQModel README (all as of 2026-09-27).
Our Coverage and Data
Local Model Watch has published 1 article(s) on models available in GPTQ: 0 where the repository itself is in GPTQ, and 1 where we found a GPTQ build of the model. The lists below only include builds we have checked (the publisher’s organization and well-known quantizers); a model missing here may still have a GPTQ build elsewhere. Part of our model format index.
Main Engines That Load This Format
| Engine | Overview |
|---|---|
| vLLM | High-throughput LLM serving engine built around PagedAttention. Handles concurrent requests and quantized weights (AWQ / GPTQ / FP8) and is widely used on server-class GPUs. |
| SGLang | LLM serving framework known for RadixAttention prefix caching and structured output. A common alternative to vLLM for serving. |
Models Available in GPTQ
→ Scroll horizontally to see all columns
| Published | Model | Where to get it | Quantizations | Smallest VRAM tier | Article |
|---|---|---|---|---|---|
| 2026-09-08 | openbmb/MiniCPM5-2B | openbmb/MiniCPM5-2B-GPTQ | GPTQ | 4GB | MiniCPM5-2B On-Device Model Strong in Code and Math: 4GB+ VRAM |
“Smallest VRAM tier" is the smallest tier in the requirements table of each article (for other builds, of those builds; for image, video and audio models, always the article’s own table, which counts every component). Leave headroom for context length.
Last updated 2026-09-08 (JST). The explanation at the top of this page was written with the help of AI from the primary sources it cites. The tables under “Our Coverage and Data" are assembled by code from our article log.