llama.cpp v0.9.0 Released: FP8 Quantization and More

At a Glance
| Item | Value |
|---|---|
| Repository | ggml-org/llama.cpp |
| Version | v0.9.0 |
| Published | 2026-11-04 |
| License | MIT |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
The latest version of llama.cpp, v0.9.0, an LLM (Large Language Model) inference engine written in C/C++, has been released. The most impactful change for users in this update is the support for OCP FP8 E4M3 format quantization. This enables the execution of models utilizing this new data format specialized for inference.
Breaking Changes and Deprecations
Along with the update to the ggml library, changes have been made to the API related to tensor copying.
| Old API | New API |
|---|---|
_2d API |
ggml_backend_tensor_copy_n/set_n/get_n (batched tensor copy API) |
This change affects users who develop using the ggml backend API directly. Code dependent on the existing _2d API will need to be migrated to the new batched tensor copy API.
Key Changes
FP8 Quantization Support
The OCP FP8 E4M3 tensor type has been newly added. E4M3 is the format most commonly used in inference. With this support, inference using more efficient quantization becomes possible. To utilize the new data format, specifying a new ftype may be required when using models created with the corresponding quantization format.
KV Cache Efficiency
The new flash_attn_ext_rows operator has been introduced. This improves the efficiency of unified KV cache in multi-sequence inference, recovering prefill performance equivalent to when ununified KV was used. Performance improvements can be expected for users handling multiple sequences simultaneously or building complex inference workflows.
Gemma 4 Optimization
Significant prefill performance improvements through graph optimization have been achieved for Gemma 4 E2B and E4B models. Users utilizing Gemma 4 series models can benefit from faster processing speeds at the start of inference without needing special configuration changes.
Multimodal Feature Expansion
Voice generation and transcription capabilities have been significantly enhanced. In addition to added support for Chatterbox TTS, offline transcription with Nemotron 3.5 Transcribe supporting 40 languages (including speaker attribution) is now possible. Pocket TTS also allows the use of predefined voices without requiring voice cloning. Furthermore, accurate support for the Qwen3-VL video prompt format has been added. This is a very powerful update for users running multimodal models locally.
Server API Improvements
In the /v1/responses endpoint of the llama.cpp server, extra_params (such as seed and min_p) are now merged into the chat completion response. This allows developers controlling models via the API to handle parameters more flexibly.
Supported Models and Hardware
In this release, support has been added for the following new model architectures and models:
- Clover: Support for
Clover-35B-A3B-Previewdiffusion model (AR mode only). - Limite: Support for
Limite 1B - Violetto. - Metis-1.6: Support for
Praxis (~2.9B)andLogos (~10.2B). These are hybrid Mamba-2 / GQA attention models featuring fixed-depth recursion and LatentMoE. - Mistral Large 4: Newly supported.
- LFM3: NVFP4 GGUF format and temporal split have been enabled.
- Gemma 4: Significant prefill performance improvements via graph optimization have been realized for E2B and E4B models.
Additionally, on the hardware side, performance improvements and support for new operations are progressing across CPU, CUDA, SYCL, Metal, Vulkan, OpenCL, and Hexagon backends through the ggml v0.29.0 update.
How to Get It
To update llama.cpp, update your repository to the latest state. The general procedure is as follows:
git pull
For detailed build instructions, please check the official repository.
Speed on Our CPU
Measured by Local Model Watch on our CPU-only server (Neoverse-N1, 3 threads) with llama-bench (512-token prompt, 128-token generation), always on the same model: Qwen3-0.6B-Q4_K_M.gguf from unsloth/Qwen3-0.6B-GGUF (revision 50968a4468). Versions like v0.5.0 ship no binaries, so we used the first official ubuntu-arm64 build that contains the version; “+N” is how many commits that build is ahead of it. These are CPU figures, not a guide to GPU speed, and small differences can be run-to-run noise (± is the standard deviation).
| Version | Build measured | Prompt | Generation |
|---|---|---|---|
| v0.9.0 | b12110 | 128.1 ± 0.9 tok/s | 42.9 ± 0.6 tok/s |
| v0.8.0 | b11960 | 127.9 ± 0.1 tok/s | 42.4 ± 0.4 tok/s |
Related Articles
- llama.cpp v1.0.0 Released: Major Updates and New Features
- llama.cpp v0.8.0: Multimodal Enhancements and LFM3 Support
- llama.cpp v0.7.0 Released with MoE GPU Cache and New Models
- llama.cpp v0.6.0 Released with llama_batch_ext and Metal Boosts
What to Read Next
- Follow this tool → llama.cpp overview and release history (9 releases tracked)
- Other inference engines and runtimes → Ollama / vLLM / SGLang
Sources
- ggml-org/llama.cpp v0.9.0
- Quant: OCP FP8 E4M3 support
- llama : follow ctx_other devices and flash attention in Gemma4 assistant
- llama : remove simple batch splitting for DSV4
- model, mtmd: add chatterbox TTS support
- mtmd: accurate support for qwen3-vl video prompt format
- server : fix /v1/responses endpoint to accept extra_params
- feat: add Clover support
- [Model] Support Limite 1B – Violetto

