ggml v0.28.0 Released: Sparse Attention and MoE Optimizations

- 1. At a Glance
- 2. Overview
- 3. Breaking Changes / Deprecations
- 4. Key Changes
- 4.1. Addition of New Operator GGML_OP_TOP_K_MASK for Sparse Attention
- 4.2. Acceleration of MoE and CUDA Backend
- 4.3. Graph Recording and XMX Optimization in SYCL Backend
- 4.4. Relaxation of Memory Layout Constraints and Core Feature Improvements
- 4.5. Expansion of Diverse Hardware Backend Features
- 4.6. Improved Stability and Bug Fixes
- 5. Supported Models and Hardware
- 6. How to Update
- 7. Related Articles
- 8. What to Read Next
- 9. Sources
At a Glance
| Item | Value |
|---|---|
| Repository | ggml-org/ggml |
| Version | v0.28.0 |
| Published | 2026-10-28 |
| License | MIT |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
The latest version of ggml, a C++ tensor library for machine learning, “ggml v0.28.0" has been released. ggml is an MIT-licensed open-source library widely used as a foundation for Large Language Model (LLM) inference in local environments.
This version adds the fused mask operator GGML_OP_TOP_K_MASK for Sparse Attention processing and organizes dispatch behavior in Mixture of Experts (MoE) processing. Significant throughput improvements have been made across various backends, including CUDA, SYCL, and Metal. This release offers major benefits for developers looking to run MoE models and sparse attention models at high speeds in local environments.
Breaking Changes / Deprecations
In this version, the index specification rules for MoE-related operations and the RPC protocol version have been updated.
| Item | Previous Behavior | Behavior from v0.28.0 onwards |
|---|---|---|
-1 index for ggml_mul_mat_id / ggml_add_id |
Depended on patch processing as an out-of-bounds index | Explicitly skips calculation for the corresponding slot and zero-fills the output row |
| RPC Protocol Version | 8.0.0 | 8.0.1 (GGML_OP_COUNT changed to 102) |
These changes affect developers implementing routing logic such as MoE using ggml_mul_mat_id or ggml_add_id directly, as well as environments performing distributed inference using RPC features. If you are using RPC features, you must update both the client and server sides to this version.
Key Changes
Addition of New Operator GGML_OP_TOP_K_MASK for Sparse Attention
A new operator GGML_OP_TOP_K_MASK and API function ggml_top_k_mask() that fuses top-k sparse attention mask processing have been added. Implementations are provided for CPU, CUDA, and Metal backends. This reduces attention calculation overhead in architectures utilizing sparse attention.
Acceleration of MoE and CUDA Backend
Optimizations for MoE processing in CUDA environments have progressed, adding fused top-k processing for grouped experts and introducing a dedicated fused path for large-scale configurations with up to 384 experts. Furthermore, fusion of FFN gate/up matrix multiplication and GLU using interleaved gate/up rows in MMQ, as well as fusion of MRoPE and SET_ROWS, has been implemented. Additionally, support for chunked launch has been added to smooth out CUDA graph submission overhead.
Graph Recording and XMX Optimization in SYCL Backend
In the SYCL backend targeting Intel GPUs and others, a graph record/replay feature has been added to reduce per-operator invocation overhead. Additionally, multi-column matrix multiplication kernels for Intel XMX matrix engines have been revamped, and dequantization processes have been accelerated, including a fast path for Q8 and wide stores for Q4_K/Q5_K.
Relaxation of Memory Layout Constraints and Core Feature Improvements
In core features, constraints on ggml_scale have been relaxed to allow it to accept non-contiguous row views in addition to contiguous memory. Furthermore, non-contiguous inputs are now supported for the SUM operator in CPU, CUDA, Vulkan, and WebGPU. Additionally, ggml_prec_get_acc() and ggml_prec_get_src() accessors have been added to retrieve accumulator precision and source-specific precision for computation nodes.
Expansion of Diverse Hardware Backend Features
In the OpenCL backend, initial support for Q2_K and Q3_K quantization formats has been added, along with TOP_K for sparse attention and a lightning indexer. In the Vulkan backend, RMS_NORM has been rewritten using subgroup reduction, and WebGPU has added support for the Q2_0 quantization format. Additionally, the repacking process for the Hexagon (HTP) backend has been revamped, and Flash Attention (FLASH_ATTN_EXT) is now available in the ZDNN backend.
Improved Stability and Bug Fixes
A problem where the timer counter could overflow with ggml_time_ms and ggml_time_us in Windows environments has been fixed. Improvements have also been made to ensure that processes do not terminate abnormally when excessive RDMA transfers or std::length_error occur during RPC connections, allowing for proper error handling.
Supported Models and Hardware
This version includes expansions of quantization formats and optimizations of hardware features for various edge devices and accelerators.
Expanded Quantization Format Support
- Q2_K and Q3_K support in OpenCL environments: Initial support for Q2_K and Q3_K quantization formats has been added to various matrix operation kernels such as gemv, gemm, mul-mat, and mul-id in the OpenCL backend. This includes operation paths for MoE, enabling inference of low-bit quantized MoE models in environments using OpenCL.
- Q2_0 support in WebGPU: Support for the Q2_0 format has been added to the WebGPU backend. This allows models to be run with an even smaller memory footprint in browser environments or portable GPU inference environments.
- Low-bit quantization on Qualcomm Hexagon (HTP): Q1_0 and Q2_0 format weights are now supported in the Hexagon backend. This includes support for
MUL_MAToperations using Hadamard rotation andGET_ROWStargeting both flat and tiled layouts. - KleidiAI Q8_0 optimization for Arm CPU: For the Arm architecture, KleidiAI Q8_0 SVE dotprod GEMV kernels and SVE i8mm GEMM kernels have been added. It also includes functionality to detect the presence of the SVE-i8MM instruction set at runtime.
Enhanced Accelerator and GPU Support
- Qualcomm Hexagon DSP / HMX: Updated to Snapdragon Toolchain v0.8, enabling the Hexagon v85 architecture. For Flash Attention execution in multi-core HMX, KV head team splitting and multi-stream execution have been introduced, along with fixes for long context and sliding window processing. Additionally, customization of VTCM size via the
GGML_HEXAGON_VTCM_SIZEenvironment variable is supported. - Qualcomm Adreno X1: In OpenCL environments, a pre-compiled MxFP4 format MoE DP4A prefill GEMM kernel for Adreno X1 has been introduced.
- IBM ZDNN: Flash Attention operations (
FLASH_ATTN_EXT) are now available in the ZDNN backend. - Intel XMX: In the SYCL environment for Intel GPUs, multi-column matrix multiplication kernels utilizing the Intel XMX (Matrix Extensions) matrix engine have been rebuilt, improving processing efficiency.
- AMD HIP (ROCm): Fixed an issue with bf16 precision WMMA operand types in RDNA3 and RDNA4 generations (gfx11 and gfx12). A portable fallback process has also been added for environments where byte substitution instructions are unavailable.
Model Architecture Related Updates
In the commit history, the inclusion of operations related to DeepSeek-V4.1 (DSV4.1 flash) and review fixes have been reflected. Additionally, it includes the fusion of MRoPE and SET_ROWS in the Qwen architecture and improvements to selection logic related to sparse attention.
How to Update
If you are using a build from source, please obtain the latest code from the GitHub repository and rebuild. The standard build procedure described in the official README is as follows:
git clone https://github.com/ggml-org/ggml
cd ggml
mkdir build && cd build
cmake..
cmake --build. --config Release -j 8
If an existing build directory exists, please run git pull in the repository root and then run cmake --build again to update the binaries.
For environments using distributed inference via RPC, the communication protocol patch version has been raised to 8.0.1, so you must update both the client and server executables to the same version.
Related Articles
- ggml v1.0.0 Released: Major Version Update and Improvements
- GGML v0.29.0 Released: Multi-Node Tensor Parallelism and More
- ggml v0.27.0 Released with Tensor-Parallel RPC and Major Speedups
- ggml v0.25.2 Released with Hardware Backend Optimizations
What to Read Next
- Follow this tool → ggml overview and release history (45 releases tracked)
- Other quantization, model formats and fine-tuning → ExLlamaV3 / Unsloth
Sources
- ggml-org/ggml v0.28.0 Release Notes: https://github.com/ggml-org/ggml/releases/tag/v0.28.0
- llama.cpp PR #30640 (TOP_K_MASK): https://github.com/ggml-org/llama.cpp/pull/30640
- llama.cpp PR #30656 (Precision accessors): https://github.com/ggml-org/llama.cpp/pull/30656
- llama.cpp PR #26631 (MUL_MAT_ID -1 index): https://github.com/ggml-org/llama.cpp/pull/26631
- llama.cpp PR #31146 (SCALE non-contiguous): https://github.com/ggml-org/llama.cpp/pull/31146
- llama.cpp PR #28851 (SUM non-contiguous): https://github.com/ggml-org/llama.cpp/pull/28851
- llama.cpp PR #29181 (CUDA MoE top-k fusion): https://github.com/ggml-org/llama.cpp/pull/29181
- llama.cpp PR #31288 (CUDA top-k for 384 experts): https://github.com/ggml-org/llama.cpp/pull/31288
- llama.cpp PR #29948 (CUDA MMQ GLU fusion): https://github.com/ggml-org/llama.cpp/pull/29948
- llama.cpp PR #31025 (CUDA MRoPE + SET_ROWS): https://github.com/ggml-org/llama.cpp/pull/31025
- llama.cpp PR #30804 (Chunked CUDA graph launch): https://github.com/ggml-org/llama.cpp/pull/30804
- llama.cpp PR #28725 (SYCL graph record/replay): https://github.com/ggml-org/llama.cpp/pull/28725
- llama.cpp PR #29864 (Intel XMX matmul): https://github.com/ggml-org/llama.cpp/pull/29864
- llama.cpp PR #30999 (OpenCL Q2_K): https://github.com/ggml-org/llama.cpp/pull/30999
- llama.cpp PR #31172 (OpenCL Q3_K): https://github.com/ggml-org/llama.cpp/pull/31172
- llama.cpp PR #31026 (OpenCL topk and lightning indexer): https://github.com/ggml-org/llama.cpp/pull/31026
- llama.cpp PR #30691 (OpenCL MxFP4 MoE DP4A): https://github.com/ggml-org/llama.cpp/pull/30691
- llama.cpp PR #30914 (WebGPU Q2_0): https://github.com/ggml-org/llama.cpp/pull/30914
- llama.cpp PR #30122 (Hexagon Q1_0 / Q2_0): https://github.com/ggml-org/llama.cpp/pull/30122
- llama.cpp PR #30808 (KleidiAI Q8_0 SVE): https://github.com/ggml-org/llama.cpp/pull/30808
- llama.cpp PR #30854 (Hexagon v85 / toolchain v0.8): https://github.com/ggml-org/llama.cpp/pull/30854
- llama.cpp PR #31221 (Hexagon VTCM size): https://github.com/ggml-org/llama.cpp/pull/31221
- llama.cpp PR #30512 (ZDNN Flash Attention): https://github.com/ggml-org/llama.cpp/pull/30512

