{"id":15091,"date":"2026-11-17T11:03:20","date_gmt":"2026-11-17T02:03:20","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=15091"},"modified":"2026-11-17T11:12:56","modified_gmt":"2026-11-17T02:12:56","slug":"llama-cpp-v0-9-0-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/llama-cpp-v0-9-0-released\/","title":{"rendered":"llama.cpp v0.9.0 Released: FP8 Quantization and More"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\">ggml-org\/llama.cpp<\/a><\/td>\n<\/tr>\n<tr>\n<td>Version<\/td>\n<td><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/releases\/tag\/v0.9.0\">v0.9.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-04<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>MIT<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>The latest version of <code>llama.cpp<\/code>, v0.9.0, an LLM (Large Language Model) inference engine written in C\/C++, has been released. The most impactful change for users in this update is the support for OCP FP8 E4M3 format quantization. This enables the execution of models utilizing this new data format specialized for inference.<\/p>\n<h2>Breaking Changes and Deprecations<\/h2>\n<p>Along with the update to the <code>ggml<\/code> library, changes have been made to the API related to tensor copying.<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Old API<\/th>\n<th>New API<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>_2d<\/code> API<\/td>\n<td><code>ggml_backend_tensor_copy_n<\/code>\/<code>set_n<\/code>\/<code>get_n<\/code> (batched tensor copy API)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>This change affects users who develop using the <code>ggml<\/code> backend API directly. Code dependent on the existing <code>_2d<\/code> API will need to be migrated to the new batched tensor copy API.<\/p>\n<h2>Key Changes<\/h2>\n<h3>FP8 Quantization Support<\/h3>\n<p>The OCP FP8 E4M3 tensor type has been newly added. E4M3 is the format most commonly used in inference. With this support, inference using more efficient quantization becomes possible. To utilize the new data format, specifying a new <code>ftype<\/code> may be required when using models created with the corresponding quantization format.<\/p>\n<h3>KV Cache Efficiency<\/h3>\n<p>The new <code>flash_attn_ext_rows<\/code> operator has been introduced. This improves the efficiency of unified KV cache in multi-sequence inference, recovering prefill performance equivalent to when ununified KV was used. Performance improvements can be expected for users handling multiple sequences simultaneously or building complex inference workflows.<\/p>\n<h3>Gemma 4 Optimization<\/h3>\n<p>Significant prefill performance improvements through graph optimization have been achieved for Gemma 4 E2B and E4B models. Users utilizing Gemma 4 series models can benefit from faster processing speeds at the start of inference without needing special configuration changes.<\/p>\n<h3>Multimodal Feature Expansion<\/h3>\n<p>Voice generation and transcription capabilities have been significantly enhanced. In addition to added support for Chatterbox TTS, offline transcription with Nemotron 3.5 Transcribe supporting 40 languages (including speaker attribution) is now possible. Pocket TTS also allows the use of predefined voices without requiring voice cloning. Furthermore, accurate support for the Qwen3-VL video prompt format has been added. This is a very powerful update for users running multimodal models locally.<\/p>\n<h3>Server API Improvements<\/h3>\n<p>In the <code>\/v1\/responses<\/code> endpoint of the <code>llama.cpp<\/code> server, <code>extra_params<\/code> (such as <code>seed<\/code> and <code>min_p<\/code>) are now merged into the chat completion response. This allows developers controlling models via the API to handle parameters more flexibly.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>In this release, support has been added for the following new model architectures and models:<\/p>\n<ul>\n<li><strong>Clover<\/strong>: Support for <code>Clover-35B-A3B-Preview<\/code> diffusion model (AR mode only).<\/li>\n<li><strong>Limite<\/strong>: Support for <code>Limite 1B - Violetto<\/code>.<\/li>\n<li><strong>Metis-1.6<\/strong>: Support for <code>Praxis (~2.9B)<\/code> and <code>Logos (~10.2B)<\/code>. These are hybrid Mamba-2 \/ GQA attention models featuring fixed-depth recursion and LatentMoE.<\/li>\n<li><strong>Mistral Large 4<\/strong>: Newly supported.<\/li>\n<li><strong>LFM3<\/strong>: NVFP4 GGUF format and temporal split have been enabled.<\/li>\n<li><strong>Gemma 4<\/strong>: Significant prefill performance improvements via graph optimization have been realized for E2B and E4B models.<\/li>\n<\/ul>\n<p>Additionally, on the hardware side, performance improvements and support for new operations are progressing across CPU, CUDA, SYCL, Metal, Vulkan, OpenCL, and Hexagon backends through the <code>ggml<\/code> v0.29.0 update.<\/p>\n<h2>How to Get It<\/h2>\n<p>To update <code>llama.cpp<\/code>, update your repository to the latest state. The general procedure is as follows:<\/p>\n<pre><code class=\"language-bash\">git pull\n<\/code><\/pre>\n<p>For detailed build instructions, please check the official repository.<\/p>\n<p><!-- lmw:version-speed --><\/p>\n<h2>Speed on Our CPU<\/h2>\n<p><em>Measured by Local Model Watch on our CPU-only server (Neoverse-N1, 3 threads) with llama-bench (512-token prompt, 128-token generation), always on the same model: <code>Qwen3-0.6B-Q4_K_M.gguf<\/code> from <a href=\"https:\/\/huggingface.co\/unsloth\/Qwen3-0.6B-GGUF\/tree\/50968a4468ef4233ed78cd7c3de230dd1d61a56b\">unsloth\/Qwen3-0.6B-GGUF<\/a> (revision <code>50968a4468<\/code>). Versions like v0.5.0 ship no binaries, so we used the first official ubuntu-arm64 build that contains the version; \u201c+N\u201d is how many commits that build is ahead of it. These are CPU figures, not a guide to GPU speed, and small differences can be run-to-run noise (\u00b1 is the standard deviation).<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Version<\/th>\n<th>Build measured<\/th>\n<th>Prompt<\/th>\n<th>Generation<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>v0.9.0<\/td>\n<td>b12110<\/td>\n<td>128.1 \u00b1 0.9 tok\/s<\/td>\n<td>42.9 \u00b1 0.6 tok\/s<\/td>\n<\/tr>\n<tr>\n<td>v0.8.0<\/td>\n<td>b11960<\/td>\n<td>127.9 \u00b1 0.1 tok\/s<\/td>\n<td>42.4 \u00b1 0.4 tok\/s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:version-speed --><\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/llamacpp-v100-released\/\">llama.cpp v1.0.0 Released: Major Updates and New Features<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/llama-cpp-v0-8-0-multimodal-lfm3-support\/\">llama.cpp v0.8.0: Multimodal Enhancements and LFM3 Support<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/llamacpp-v0-7-0-released\/\">llama.cpp v0.7.0 Released with MoE GPU Cache and New Models<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/06\/llamacpp-v0-6-0-released\/\">llama.cpp v0.6.0 Released with llama_batch_ext and Metal Boosts<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Follow this tool<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-llama-cpp-en\/\">llama.cpp overview and release history (9 releases tracked)<\/a><\/li>\n<li><strong>Other inference engines and runtimes<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-ollama-en\/\">Ollama<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-sglang-en\/\">SGLang<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/releases\/tag\/v0.9.0\">ggml-org\/llama.cpp v0.9.0<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/27512\">Quant: OCP FP8 E4M3 support<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31635\">llama : follow ctx_other devices and flash attention in Gemma4 assistant<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31300\">llama : remove simple batch splitting for DSV4<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30993\">model, mtmd: add chatterbox TTS support<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31619\">mtmd: accurate support for qwen3-vl video prompt format<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30959\">server : fix \/v1\/responses endpoint to accept extra_params<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31306\">feat: add Clover support<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29433\">[Model] Support Limite 1B &#8211; Violetto<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>llama.cpp v0.9.0 is released, featuring OCP FP8 E4M3 quantization support, KV cache improvements, multimodal updates, and support for new models.<\/p>\n","protected":false},"author":1,"featured_media":15090,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[1133,4559,163,1073,1547],"class_list":["post-15091","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-fp8-en","tag-gemma-4-en","tag-gguf-en","tag-llama-cpp-en","tag-verified"],"lang":"en","translations":{"en":15091,"ja":15089},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=15091"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15091\/revisions"}],"predecessor-version":[{"id":18044,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/15091\/revisions\/18044"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/15090"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=15091"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=15091"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=15091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}