GGUF Model Format Explained: Supported Engines and Models
What Is GGUF?
GGUF is the model file format built for the C/C++ inference engine llama.cpp and the ggml library underneath it. It stores the weights (tensors) together with the metadata a runtime needs — architecture, tokenizer, chat template — in a single file. Download one file and it runs, which is why GGUF is the most common format in local LLM use.
Why It Matters
- A fine-grained choice of quantizations. The same model usually comes as Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, IQ2_XXS and more, so you can pick the one that fits your GPU memory. See our quantization and model-format glossary for how to read the names.
- Runs without a GPU. It works on CPU alone and on NVIDIA (CUDA), AMD (ROCm / Vulkan) and Apple Silicon (Metal) GPUs, and can offload only part of the layers to the GPU when the model does not fit.
- Broad app support. Ollama, LM Studio, KoboldCpp, Jan and many others use llama.cpp internally and load GGUF files directly.
Tips for Running It Locally
- Q4_K_M is the usual starting point. It balances quality loss and file size well and is the default recommendation of many quantizers. Move up to Q5_K_M or Q6_K if you have memory to spare.
- You need the file size plus room for context. The KV cache grows with conversation length, so a GPU that barely fits the file is not enough. The requirements tables in our articles are computed from the actual file sizes.
- Vision models need a separate
mmprojfile. The vision encoder is usually shipped as its own GGUF; download it alongside the main file. - Many models have no official GGUF. In that case, use a build from a quantizer such as bartowski or unsloth. The list below separates articles whose repository is itself GGUF from those where we found a GGUF build.
Sources: the GGUF specification (docs/gguf.md in ggml-org/ggml), Hugging Face Hub documentation on GGUF and the llama.cpp README (all as of 2026-09-27).
Our Coverage and Data
Local Model Watch has published 22 article(s) on models available in GGUF: 12 where the repository itself is in GGUF, and 10 where we found a GGUF build of the model. The lists below only include builds we have checked (the publisher’s organization and well-known quantizers); a model missing here may still have a GGUF build elsewhere. Part of our model format index.
Main Engines That Load This Format
| Engine | Overview |
|---|---|
| llama.cpp | LLM inference engine written in C/C++. Runs GGUF models on CPU and GPU (CUDA / Metal / Vulkan / ROCm) and underpins much of the local-LLM ecosystem, including Ollama, LM Studio and KoboldCpp. |
| Ollama | Local LLM runtime that pulls and runs models with a single command. Ships an OpenAI-compatible API server for macOS, Linux and Windows. |
| KoboldCpp | Single-file runtime that bundles llama.cpp with a web UI and API. Download a GGUF model and launch. |
| llamafile | Packages model weights and llama.cpp into one executable that runs on any OS without installation. |
| LocalAI | Self-hosted, OpenAI-API-compatible inference server that fronts multiple backends for text, image and audio. |
| Jan | Offline desktop chat app with llama.cpp built in; doubles as a local API server. |
Models Available in GGUF
→ Scroll horizontally to see all columns
“Smallest VRAM tier" is the smallest tier in the requirements table of each article (for other builds, of those builds; for image, video and audio models, always the article’s own table, which counts every component). Leave headroom for context length.
Last updated 2026-09-26 (JST). The explanation at the top of this page was written with the help of AI from the primary sources it cites. The tables under “Our Coverage and Data" are assembled by code from our article log.