ggml v0.28.0 Released: Sparse Attention and MoE Optimizations

ggml v0.28.0 Released: Sparse Attention and MoE Optimizations

At a Glance

Item Value
Repository ggml-org/ggml
Version v0.28.0
Published 2026-10-28
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

The latest version of ggml, a C++ tensor library for machine learning, “ggml v0.28.0" has been released. ggml is an MIT-licensed open-source library widely used as a foundation for Large Language Model (LLM) inference in local environments.

This version adds the fused mask operator GGML_OP_TOP_K_MASK for Sparse Attention processing and organizes dispatch behavior in Mixture of Experts (MoE) processing. Significant throughput improvements have been made across various backends, including CUDA, SYCL, and Metal. This release offers major benefits for developers looking to run MoE models and sparse attention models at high speeds in local environments.

Breaking Changes / Deprecations

In this version, the index specification rules for MoE-related operations and the RPC protocol version have been updated.

Item Previous Behavior Behavior from v0.28.0 onwards
-1 index for ggml_mul_mat_id / ggml_add_id Depended on patch processing as an out-of-bounds index Explicitly skips calculation for the corresponding slot and zero-fills the output row
RPC Protocol Version 8.0.0 8.0.1 (GGML_OP_COUNT changed to 102)

These changes affect developers implementing routing logic such as MoE using ggml_mul_mat_id or ggml_add_id directly, as well as environments performing distributed inference using RPC features. If you are using RPC features, you must update both the client and server sides to this version.

Key Changes

Addition of New Operator GGML_OP_TOP_K_MASK for Sparse Attention

A new operator GGML_OP_TOP_K_MASK and API function ggml_top_k_mask() that fuses top-k sparse attention mask processing have been added. Implementations are provided for CPU, CUDA, and Metal backends. This reduces attention calculation overhead in architectures utilizing sparse attention.

Acceleration of MoE and CUDA Backend

Optimizations for MoE processing in CUDA environments have progressed, adding fused top-k processing for grouped experts and introducing a dedicated fused path for large-scale configurations with up to 384 experts. Furthermore, fusion of FFN gate/up matrix multiplication and GLU using interleaved gate/up rows in MMQ, as well as fusion of MRoPE and SET_ROWS, has been implemented. Additionally, support for chunked launch has been added to smooth out CUDA graph submission overhead.

Graph Recording and XMX Optimization in SYCL Backend

In the SYCL backend targeting Intel GPUs and others, a graph record/replay feature has been added to reduce per-operator invocation overhead. Additionally, multi-column matrix multiplication kernels for Intel XMX matrix engines have been revamped, and dequantization processes have been accelerated, including a fast path for Q8 and wide stores for Q4_K/Q5_K.

Relaxation of Memory Layout Constraints and Core Feature Improvements

In core features, constraints on ggml_scale have been relaxed to allow it to accept non-contiguous row views in addition to contiguous memory. Furthermore, non-contiguous inputs are now supported for the SUM operator in CPU, CUDA, Vulkan, and WebGPU. Additionally, ggml_prec_get_acc() and ggml_prec_get_src() accessors have been added to retrieve accumulator precision and source-specific precision for computation nodes.

Expansion of Diverse Hardware Backend Features

In the OpenCL backend, initial support for Q2_K and Q3_K quantization formats has been added, along with TOP_K for sparse attention and a lightning indexer. In the Vulkan backend, RMS_NORM has been rewritten using subgroup reduction, and WebGPU has added support for the Q2_0 quantization format. Additionally, the repacking process for the Hexagon (HTP) backend has been revamped, and Flash Attention (FLASH_ATTN_EXT) is now available in the ZDNN backend.

Improved Stability and Bug Fixes

A problem where the timer counter could overflow with ggml_time_ms and ggml_time_us in Windows environments has been fixed. Improvements have also been made to ensure that processes do not terminate abnormally when excessive RDMA transfers or std::length_error occur during RPC connections, allowing for proper error handling.

Supported Models and Hardware

This version includes expansions of quantization formats and optimizations of hardware features for various edge devices and accelerators.

Expanded Quantization Format Support

  • Q2_K and Q3_K support in OpenCL environments: Initial support for Q2_K and Q3_K quantization formats has been added to various matrix operation kernels such as gemv, gemm, mul-mat, and mul-id in the OpenCL backend. This includes operation paths for MoE, enabling inference of low-bit quantized MoE models in environments using OpenCL.
  • Q2_0 support in WebGPU: Support for the Q2_0 format has been added to the WebGPU backend. This allows models to be run with an even smaller memory footprint in browser environments or portable GPU inference environments.
  • Low-bit quantization on Qualcomm Hexagon (HTP): Q1_0 and Q2_0 format weights are now supported in the Hexagon backend. This includes support for MUL_MAT operations using Hadamard rotation and GET_ROWS targeting both flat and tiled layouts.
  • KleidiAI Q8_0 optimization for Arm CPU: For the Arm architecture, KleidiAI Q8_0 SVE dotprod GEMV kernels and SVE i8mm GEMM kernels have been added. It also includes functionality to detect the presence of the SVE-i8MM instruction set at runtime.

Enhanced Accelerator and GPU Support

  • Qualcomm Hexagon DSP / HMX: Updated to Snapdragon Toolchain v0.8, enabling the Hexagon v85 architecture. For Flash Attention execution in multi-core HMX, KV head team splitting and multi-stream execution have been introduced, along with fixes for long context and sliding window processing. Additionally, customization of VTCM size via the GGML_HEXAGON_VTCM_SIZE environment variable is supported.
  • Qualcomm Adreno X1: In OpenCL environments, a pre-compiled MxFP4 format MoE DP4A prefill GEMM kernel for Adreno X1 has been introduced.
  • IBM ZDNN: Flash Attention operations (FLASH_ATTN_EXT) are now available in the ZDNN backend.
  • Intel XMX: In the SYCL environment for Intel GPUs, multi-column matrix multiplication kernels utilizing the Intel XMX (Matrix Extensions) matrix engine have been rebuilt, improving processing efficiency.
  • AMD HIP (ROCm): Fixed an issue with bf16 precision WMMA operand types in RDNA3 and RDNA4 generations (gfx11 and gfx12). A portable fallback process has also been added for environments where byte substitution instructions are unavailable.

Model Architecture Related Updates

In the commit history, the inclusion of operations related to DeepSeek-V4.1 (DSV4.1 flash) and review fixes have been reflected. Additionally, it includes the fusion of MRoPE and SET_ROWS in the Qwen architecture and improvements to selection logic related to sparse attention.

How to Update

If you are using a build from source, please obtain the latest code from the GitHub repository and rebuild. The standard build procedure described in the official README is as follows:

git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake..
cmake --build. --config Release -j 8

If an existing build directory exists, please run git pull in the repository root and then run cmake --build again to update the binaries.

For environments using distributed inference via RPC, the communication protocol patch version has been raised to 8.0.1, so you must update both the client and server executables to the same version.

Related Articles

What to Read Next

Sources