ggml v0.25.0 Released: FlashAttention and MoE Optimizations

ggml v0.25.0 Released: FlashAttention and MoE Optimizations

At a Glance

Item Value
Repository ggml-org/ggml
Version v0.25.0
Published 2026-09-23
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code at collection time. Dates are JST.

Overview

The latest version “v0.25.0" of ggml-org/ggml has been released. ggml is an open-source machine learning tensor library written in C++ designed to achieve efficient inference on local hardware (MIT license, 15,398 GitHub stars).

The most impactful change for users in this version is the expanded application and fused kernel optimization of FlashAttention, MoE (Mixture of Experts), SSM (State Space Model), and Hyper-connection processing across CPU, GPU, and various accelerator backends. Performance and compatibility when running the latest LLM architectures have been improved.

Breaking Changes and Deprecations

This release includes changes regarding the RPC protocol for distributing processing across a network and GGUF format alignment calculations.

Target Before (Old) After (New)
RPC Protocol Old major version Changed to Major version 7
GGUF Data Alignment Relative position from the start of the file Relative position from the GGUF start position (llama/28993)

Due to the protocol version update, it is not possible to communicate by mixing old version RPC servers and clients. Users utilizing the network distributed inference function (RPC backend) need to update both the server and client binaries to the same version. Additionally, those who develop and operate custom parsers that directly parse GGUF files need to note that the alignment calculation basis has changed.

Key Changes

FlashAttention Feature Additions and Backend Optimization

FlashAttention (FA) kernels have been expanded and accelerated across major backends such as CUDA/HIP, Metal, Vulkan, OpenCL, SYCL, and Hexagon. In CUDA, shape tuning has been applied for NVIDIA Ampere and later architectures, and optimizations for Sparse Flash Attention and Gemma 4 have been enabled. In Metal, kernels supporting head sizes (HSK=96, HSV=64) used in MiniCPM3 and others have been added, and Vulkan has optimization kernels implemented for Intel Xe architectures (Xe-LPG Plus/Xe2/Xe3). Memory efficiency and calculation speed when processing long contexts on supported hardware are improved. Model reconversion is not required; changes are reflected simply by updating the library.

Enhanced Fused MoE and SSM Kernels

Kernel fusion optimizations have been promoted to improve calculation efficiency for MoE (Mixture of Experts) and SSM (State Space Model) models. In the Metal backend, fusion optimizations for MoE and SSM_CONV have been added, while OpenVINO has expanded optimizations for MoE inference and stateful decoding on GPUs, along with a compressed MoE path. Additionally, in Vulkan, the row ID limit for mul_mat_id has been raised from 256 to 512 experts, supporting dispatch for larger MoE models. Fusion of RMS Norm+Scale and SSM Conv+SiLU has also been implemented in SYCL. Inference latency reduction is expected in environments operating MoE configuration models or SSM-based models such as Mamba.

Expanded Support for Hyper-connection Structures

Support for Hyper-connection operations adopted in models like Qwen4Exp has been expanded across various backends. In addition to adding ggml_dsv4_hc_pre_gated() for performing gated processing in the C++ core part, dedicated hyper-connection operators (such as dsv4_hc_pre and hc_post) have been added and optimized in the Metal, Vulkan, and SYCL backends. This enables models with the latest architectures using hyper-connection technology to be executed directly and at high speed on various GPUs.

Processing Speedups in Quantization Formats such as IQ1_M

Improvements have also been made to the calculation processing of quantization formats. In the ultra-low quantization format IQ1_M, a change (llama/28706) to construct prefix sums once per block was made, improving calculation efficiency. Furthermore, the IQ3_S MMQ matrix multiplication kernel was implemented in the Vulkan backend, and low-bit quantization kernels for IQ2 and IQ3 using SWAR technology have been accelerated in the HIP (ROCm) environment. Support for Q4_K and Q6_K quantization formats has also been added for the Hexagon NPU environment. Environments operating applicable quantized models can achieve improved inference execution speeds without re-quantizing.

Supported Models and Hardware

This version brings functional expansions and optimizations for diverse model architectures, various accelerators, and quantization formats.

Added and Optimized Model Architectures

  • Gemma 4: Performance tuning (llama/28450) for FlashAttention shapes such as gemma4-26b-a4b was applied in NVIDIA Ampere and later GPU environments.
  • Qwen4 / qwen4exp: In addition to supporting Sparse Flash Attention for Qwen4 (llama/28770), hyper-connection operators used in qwen4exp (such as dsv4_hc_pre and hc_post) were expanded across various backends like Metal, Vulkan, and SYCL.
  • MiniCPM3: A dedicated FlashAttention kernel (llama/28599) for head sizes required by MiniCPM3 (HSK=96, HSV=64) was added in the Metal backend.
  • Gated Delta Net: A Gated Delta Net kernel (llama/29199) optimized with HMX (Hexagon Matrix Extensions) was implemented for the Qualcomm Hexagon NPU, and fused Gated Delta Net + Copy processing (llama/28976) is now supported in WebGPU environments as well.

Hardware and Backend Expansions

  • Intel GPU (Vulkan / SYCL / OpenVINO): FlashAttention optimization kernels (llama/24406) for Intel Xe architectures (Xe-LPG Plus / Xe2 / Xe3) were added in the Vulkan backend. Furthermore, the OpenVINO backend was updated to 2026.4 (llama/29009), optimizing MoE inference and stateful decoding processing in GPU environments.
  • Qualcomm Hexagon NPU: A comprehensive overhaul of buffers and DMA processing supporting 64-bit mapping (llama/29197) was carried out, alongside acceleration of FlashAttention mask processing via direct-map DMA cache (llama/29282) and support for continuous memory copies using DMA.
  • OpenCL Supported Devices: Implemented binary kernels including dp4a binary kernels for Q4_0 non-MoE combining A8 precision (llama/29055), binary GEMM kernels for q4_K/q8_1 (llama/29056) and q6_K (llama/28678), plus FlashAttention (llama/29046) and general-purpose ssm_scan (llama/28881).
  • NVIDIA / AMD GPU (CUDA / HIP): CUDA received conv2d acceleration using Implicit-GEMM (llama/29135) and crossover optimization from MMVQ to MMQ on SM70 (Volta) (llama/28912). HIP (ROCm) enabled AllReduce (llama/27825) and optimized MoE tile heuristics on RDNA3.5 (llama/28935).
  • CPU / Other Architectures: A Repack kernel for Q1_0 quantization was added for ARM environments (llama/23492), and int16 transposition fixes were made for SpacemiT environments (llama/25161).

Expanded Quantization Format Support

  • K-Quants Support for Hexagon: Support for K-Quants formats Q4_K and Q6_K (llama/28994) was added in the Qualcomm Hexagon NPU environment.
  • IQ3_S Support for Vulkan: MMQ matrix multiplication kernels for the IQ3_S quantization format (llama/28822) were introduced to the Vulkan backend.
  • A8 Quantization Kernels for OpenCL: Binary kernels for high-speed processing of various quantization formats such as A8 Q4_0, A8 q4_K/q8_1, and A8 q6_K were added.

How to Update

When building from source code, retrieve the source from the GitHub repository and build using CMake. The basic steps described in the README are as follows:

git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake..
cmake --build. --config Release -j 8

When enabling specific backends such as CUDA, Metal, Vulkan, or SYCL, specify the options for each backend during cmake execution as necessary.

Sources

Engines and ToolsFlashAttention,ggml,GGUF,MoE,Verified