llama.cpp v0.9.0 Released: FP8 Quantization and More

llama.cpp v0.9.0 Released: FP8 Quantization and More

At a Glance

Item Value
Repository ggml-org/llama.cpp
Version v0.9.0
Published 2026-11-04
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

The latest version of llama.cpp, v0.9.0, an LLM (Large Language Model) inference engine written in C/C++, has been released. The most impactful change for users in this update is the support for OCP FP8 E4M3 format quantization. This enables the execution of models utilizing this new data format specialized for inference.

Breaking Changes and Deprecations

Along with the update to the ggml library, changes have been made to the API related to tensor copying.

Old API New API
_2d API ggml_backend_tensor_copy_n/set_n/get_n (batched tensor copy API)

This change affects users who develop using the ggml backend API directly. Code dependent on the existing _2d API will need to be migrated to the new batched tensor copy API.

Key Changes

FP8 Quantization Support

The OCP FP8 E4M3 tensor type has been newly added. E4M3 is the format most commonly used in inference. With this support, inference using more efficient quantization becomes possible. To utilize the new data format, specifying a new ftype may be required when using models created with the corresponding quantization format.

KV Cache Efficiency

The new flash_attn_ext_rows operator has been introduced. This improves the efficiency of unified KV cache in multi-sequence inference, recovering prefill performance equivalent to when ununified KV was used. Performance improvements can be expected for users handling multiple sequences simultaneously or building complex inference workflows.

Gemma 4 Optimization

Significant prefill performance improvements through graph optimization have been achieved for Gemma 4 E2B and E4B models. Users utilizing Gemma 4 series models can benefit from faster processing speeds at the start of inference without needing special configuration changes.

Multimodal Feature Expansion

Voice generation and transcription capabilities have been significantly enhanced. In addition to added support for Chatterbox TTS, offline transcription with Nemotron 3.5 Transcribe supporting 40 languages (including speaker attribution) is now possible. Pocket TTS also allows the use of predefined voices without requiring voice cloning. Furthermore, accurate support for the Qwen3-VL video prompt format has been added. This is a very powerful update for users running multimodal models locally.

Server API Improvements

In the /v1/responses endpoint of the llama.cpp server, extra_params (such as seed and min_p) are now merged into the chat completion response. This allows developers controlling models via the API to handle parameters more flexibly.

Supported Models and Hardware

In this release, support has been added for the following new model architectures and models:

  • Clover: Support for Clover-35B-A3B-Preview diffusion model (AR mode only).
  • Limite: Support for Limite 1B - Violetto.
  • Metis-1.6: Support for Praxis (~2.9B) and Logos (~10.2B). These are hybrid Mamba-2 / GQA attention models featuring fixed-depth recursion and LatentMoE.
  • Mistral Large 4: Newly supported.
  • LFM3: NVFP4 GGUF format and temporal split have been enabled.
  • Gemma 4: Significant prefill performance improvements via graph optimization have been realized for E2B and E4B models.

Additionally, on the hardware side, performance improvements and support for new operations are progressing across CPU, CUDA, SYCL, Metal, Vulkan, OpenCL, and Hexagon backends through the ggml v0.29.0 update.

How to Get It

To update llama.cpp, update your repository to the latest state. The general procedure is as follows:

git pull

For detailed build instructions, please check the official repository.

Speed on Our CPU

Measured by Local Model Watch on our CPU-only server (Neoverse-N1, 3 threads) with llama-bench (512-token prompt, 128-token generation), always on the same model: Qwen3-0.6B-Q4_K_M.gguf from unsloth/Qwen3-0.6B-GGUF (revision 50968a4468). Versions like v0.5.0 ship no binaries, so we used the first official ubuntu-arm64 build that contains the version; “+N” is how many commits that build is ahead of it. These are CPU figures, not a guide to GPU speed, and small differences can be run-to-run noise (± is the standard deviation).

Version Build measured Prompt Generation
v0.9.0 b12110 128.1 ± 0.9 tok/s 42.9 ± 0.6 tok/s
v0.8.0 b11960 127.9 ± 0.1 tok/s 42.4 ± 0.4 tok/s

Related Articles

What to Read Next

Sources