{"id":13961,"date":"2026-11-17T11:02:49","date_gmt":"2026-11-17T02:02:49","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=13961"},"modified":"2026-11-17T11:12:04","modified_gmt":"2026-11-17T02:12:04","slug":"ggml-v0-28-0-release-sparse-attention-moe-optimizations","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ggml-v0-28-0-release-sparse-attention-moe-optimizations\/","title":{"rendered":"ggml v0.28.0 Released: Sparse Attention and MoE Optimizations"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/github.com\/ggml-org\/ggml\">ggml-org\/ggml<\/a><\/td>\n<\/tr>\n<tr>\n<td>Version<\/td>\n<td><a href=\"https:\/\/github.com\/ggml-org\/ggml\/releases\/tag\/v0.28.0\">v0.28.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-10-28<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>MIT<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>The latest version of ggml, a C++ tensor library for machine learning, &#8220;ggml v0.28.0&#8221; has been released. ggml is an MIT-licensed open-source library widely used as a foundation for Large Language Model (LLM) inference in local environments.<\/p>\n<p>This version adds the fused mask operator <code>GGML_OP_TOP_K_MASK<\/code> for Sparse Attention processing and organizes dispatch behavior in Mixture of Experts (MoE) processing. Significant throughput improvements have been made across various backends, including CUDA, SYCL, and Metal. This release offers major benefits for developers looking to run MoE models and sparse attention models at high speeds in local environments.<\/p>\n<h2>Breaking Changes \/ Deprecations<\/h2>\n<p>In this version, the index specification rules for MoE-related operations and the RPC protocol version have been updated.<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th style=\"text-align: left;\">Item<\/th>\n<th style=\"text-align: left;\">Previous Behavior<\/th>\n<th style=\"text-align: left;\">Behavior from v0.28.0 onwards<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td style=\"text-align: left;\"><code>-1<\/code> index for <code>ggml_mul_mat_id<\/code> \/ <code>ggml_add_id<\/code><\/td>\n<td style=\"text-align: left;\">Depended on patch processing as an out-of-bounds index<\/td>\n<td style=\"text-align: left;\">Explicitly skips calculation for the corresponding slot and zero-fills the output row<\/td>\n<\/tr>\n<tr>\n<td style=\"text-align: left;\">RPC Protocol Version<\/td>\n<td style=\"text-align: left;\">8.0.0<\/td>\n<td style=\"text-align: left;\">8.0.1 (<code>GGML_OP_COUNT<\/code> changed to 102)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>These changes affect developers implementing routing logic such as MoE using <code>ggml_mul_mat_id<\/code> or <code>ggml_add_id<\/code> directly, as well as environments performing distributed inference using RPC features. If you are using RPC features, you must update both the client and server sides to this version.<\/p>\n<h2>Key Changes<\/h2>\n<h3>Addition of New Operator GGML_OP_TOP_K_MASK for Sparse Attention<\/h3>\n<p>A new operator <code>GGML_OP_TOP_K_MASK<\/code> and API function <code>ggml_top_k_mask()<\/code> that fuses top-k sparse attention mask processing have been added. Implementations are provided for CPU, CUDA, and Metal backends. This reduces attention calculation overhead in architectures utilizing sparse attention.<\/p>\n<h3>Acceleration of MoE and CUDA Backend<\/h3>\n<p>Optimizations for MoE processing in CUDA environments have progressed, adding fused top-k processing for grouped experts and introducing a dedicated fused path for large-scale configurations with up to 384 experts. Furthermore, fusion of FFN gate\/up matrix multiplication and GLU using interleaved gate\/up rows in MMQ, as well as fusion of MRoPE and <code>SET_ROWS<\/code>, has been implemented. Additionally, support for chunked launch has been added to smooth out CUDA graph submission overhead.<\/p>\n<h3>Graph Recording and XMX Optimization in SYCL Backend<\/h3>\n<p>In the SYCL backend targeting Intel GPUs and others, a graph record\/replay feature has been added to reduce per-operator invocation overhead. Additionally, multi-column matrix multiplication kernels for Intel XMX matrix engines have been revamped, and dequantization processes have been accelerated, including a fast path for Q8 and wide stores for Q4_K\/Q5_K.<\/p>\n<h3>Relaxation of Memory Layout Constraints and Core Feature Improvements<\/h3>\n<p>In core features, constraints on <code>ggml_scale<\/code> have been relaxed to allow it to accept non-contiguous row views in addition to contiguous memory. Furthermore, non-contiguous inputs are now supported for the <code>SUM<\/code> operator in CPU, CUDA, Vulkan, and WebGPU. Additionally, <code>ggml_prec_get_acc()<\/code> and <code>ggml_prec_get_src()<\/code> accessors have been added to retrieve accumulator precision and source-specific precision for computation nodes.<\/p>\n<h3>Expansion of Diverse Hardware Backend Features<\/h3>\n<p>In the OpenCL backend, initial support for Q2_K and Q3_K quantization formats has been added, along with <code>TOP_K<\/code> for sparse attention and a lightning indexer. In the Vulkan backend, <code>RMS_NORM<\/code> has been rewritten using subgroup reduction, and WebGPU has added support for the Q2_0 quantization format. Additionally, the repacking process for the Hexagon (HTP) backend has been revamped, and Flash Attention (<code>FLASH_ATTN_EXT<\/code>) is now available in the ZDNN backend.<\/p>\n<h3>Improved Stability and Bug Fixes<\/h3>\n<p>A problem where the timer counter could overflow with <code>ggml_time_ms<\/code> and <code>ggml_time_us<\/code> in Windows environments has been fixed. Improvements have also been made to ensure that processes do not terminate abnormally when excessive RDMA transfers or <code>std::length_error<\/code> occur during RPC connections, allowing for proper error handling.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>This version includes expansions of quantization formats and optimizations of hardware features for various edge devices and accelerators.<\/p>\n<h3>Expanded Quantization Format Support<\/h3>\n<ul>\n<li><strong>Q2_K and Q3_K support in OpenCL environments<\/strong>: Initial support for Q2_K and Q3_K quantization formats has been added to various matrix operation kernels such as gemv, gemm, mul-mat, and mul-id in the OpenCL backend. This includes operation paths for MoE, enabling inference of low-bit quantized MoE models in environments using OpenCL.<\/li>\n<li><strong>Q2_0 support in WebGPU<\/strong>: Support for the Q2_0 format has been added to the WebGPU backend. This allows models to be run with an even smaller memory footprint in browser environments or portable GPU inference environments.<\/li>\n<li><strong>Low-bit quantization on Qualcomm Hexagon (HTP)<\/strong>: Q1_0 and Q2_0 format weights are now supported in the Hexagon backend. This includes support for <code>MUL_MAT<\/code> operations using Hadamard rotation and <code>GET_ROWS<\/code> targeting both flat and tiled layouts.<\/li>\n<li><strong>KleidiAI Q8_0 optimization for Arm CPU<\/strong>: For the Arm architecture, KleidiAI Q8_0 SVE dotprod GEMV kernels and SVE i8mm GEMM kernels have been added. It also includes functionality to detect the presence of the SVE-i8MM instruction set at runtime.<\/li>\n<\/ul>\n<h3>Enhanced Accelerator and GPU Support<\/h3>\n<ul>\n<li><strong>Qualcomm Hexagon DSP \/ HMX<\/strong>: Updated to Snapdragon Toolchain v0.8, enabling the Hexagon v85 architecture. For Flash Attention execution in multi-core HMX, KV head team splitting and multi-stream execution have been introduced, along with fixes for long context and sliding window processing. Additionally, customization of VTCM size via the <code>GGML_HEXAGON_VTCM_SIZE<\/code> environment variable is supported.<\/li>\n<li><strong>Qualcomm Adreno X1<\/strong>: In OpenCL environments, a pre-compiled MxFP4 format MoE DP4A prefill GEMM kernel for Adreno X1 has been introduced.<\/li>\n<li><strong>IBM ZDNN<\/strong>: Flash Attention operations (<code>FLASH_ATTN_EXT<\/code>) are now available in the ZDNN backend.<\/li>\n<li><strong>Intel XMX<\/strong>: In the SYCL environment for Intel GPUs, multi-column matrix multiplication kernels utilizing the Intel XMX (Matrix Extensions) matrix engine have been rebuilt, improving processing efficiency.<\/li>\n<li><strong>AMD HIP (ROCm)<\/strong>: Fixed an issue with bf16 precision WMMA operand types in RDNA3 and RDNA4 generations (gfx11 and gfx12). A portable fallback process has also been added for environments where byte substitution instructions are unavailable.<\/li>\n<\/ul>\n<h3>Model Architecture Related Updates<\/h3>\n<p>In the commit history, the inclusion of operations related to DeepSeek-V4.1 (DSV4.1 flash) and review fixes have been reflected. Additionally, it includes the fusion of MRoPE and <code>SET_ROWS<\/code> in the Qwen architecture and improvements to selection logic related to sparse attention.<\/p>\n<h2>How to Update<\/h2>\n<p>If you are using a build from source, please obtain the latest code from the GitHub repository and rebuild. The standard build procedure described in the official README is as follows:<\/p>\n<pre><code class=\"language-bash\">git clone https:\/\/github.com\/ggml-org\/ggml\ncd ggml\n\nmkdir build &amp;&amp; cd build\ncmake..\ncmake --build. --config Release -j 8\n<\/code><\/pre>\n<p>If an existing build directory exists, please run <code>git pull<\/code> in the repository root and then run <code>cmake --build<\/code> again to update the binaries.<\/p>\n<p>For environments using distributed inference via RPC, the communication protocol patch version has been raised to 8.0.1, so you must update both the client and server executables to the same version.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ggml-v1-0-0-released\/\">ggml v1.0.0 Released: Major Version Update and Improvements<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ggml-v0290-released\/\">GGML v0.29.0 Released: Multi-Node Tensor Parallelism and More<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/ggml-v0-27-0-released-2\/\">ggml v0.27.0 Released with Tensor-Parallel RPC and Major Speedups<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/24\/ggml-v0-25-2-released\/\">ggml v0.25.2 Released with Hardware Backend Optimizations<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Follow this tool<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-ggml-en\/\">ggml overview and release history (45 releases tracked)<\/a><\/li>\n<li><strong>Other quantization, model formats and fine-tuning<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-exllamav3-en\/\">ExLlamaV3<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-unsloth-en\/\">Unsloth<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li>ggml-org\/ggml v0.28.0 Release Notes: <a href=\"https:\/\/github.com\/ggml-org\/ggml\/releases\/tag\/v0.28.0\">https:\/\/github.com\/ggml-org\/ggml\/releases\/tag\/v0.28.0<\/a><\/li>\n<li>llama.cpp PR #30640 (TOP_K_MASK): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30640\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30640<\/a><\/li>\n<li>llama.cpp PR #30656 (Precision accessors): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30656\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30656<\/a><\/li>\n<li>llama.cpp PR #26631 (MUL_MAT_ID -1 index): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/26631\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/26631<\/a><\/li>\n<li>llama.cpp PR #31146 (SCALE non-contiguous): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31146\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31146<\/a><\/li>\n<li>llama.cpp PR #28851 (SUM non-contiguous): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/28851\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/28851<\/a><\/li>\n<li>llama.cpp PR #29181 (CUDA MoE top-k fusion): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29181\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29181<\/a><\/li>\n<li>llama.cpp PR #31288 (CUDA top-k for 384 experts): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31288\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31288<\/a><\/li>\n<li>llama.cpp PR #29948 (CUDA MMQ GLU fusion): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29948\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29948<\/a><\/li>\n<li>llama.cpp PR #31025 (CUDA MRoPE + SET_ROWS): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31025\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31025<\/a><\/li>\n<li>llama.cpp PR #30804 (Chunked CUDA graph launch): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30804\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30804<\/a><\/li>\n<li>llama.cpp PR #28725 (SYCL graph record\/replay): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/28725\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/28725<\/a><\/li>\n<li>llama.cpp PR #29864 (Intel XMX matmul): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29864\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/29864<\/a><\/li>\n<li>llama.cpp PR #30999 (OpenCL Q2_K): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30999\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30999<\/a><\/li>\n<li>llama.cpp PR #31172 (OpenCL Q3_K): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31172\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31172<\/a><\/li>\n<li>llama.cpp PR #31026 (OpenCL topk and lightning indexer): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31026\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31026<\/a><\/li>\n<li>llama.cpp PR #30691 (OpenCL MxFP4 MoE DP4A): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30691\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30691<\/a><\/li>\n<li>llama.cpp PR #30914 (WebGPU Q2_0): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30914\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30914<\/a><\/li>\n<li>llama.cpp PR #30122 (Hexagon Q1_0 \/ Q2_0): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30122\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30122<\/a><\/li>\n<li>llama.cpp PR #30808 (KleidiAI Q8_0 SVE): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30808\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30808<\/a><\/li>\n<li>llama.cpp PR #30854 (Hexagon v85 \/ toolchain v0.8): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30854\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30854<\/a><\/li>\n<li>llama.cpp PR #31221 (Hexagon VTCM size): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31221\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/31221<\/a><\/li>\n<li>llama.cpp PR #30512 (ZDNN Flash Attention): <a href=\"https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30512\">https:\/\/github.com\/ggml-org\/llama.cpp\/pull\/30512<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The latest ggml release introduces GGML_OP_TOP_K_MASK for sparse attention, MoE optimizations for CUDA, and expanded support for various hardware backends.<\/p>\n","protected":false},"author":1,"featured_media":13960,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[658,1187,161,165,1170,1547],"class_list":["post-13961","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-c-en","tag-cuda-en","tag-ggml-en","tag-moe-en","tag-sparse-attention-en","tag-verified"],"lang":"en","translations":{"en":13961,"ja":13959},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13961","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=13961"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13961\/revisions"}],"predecessor-version":[{"id":17978,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13961\/revisions\/17978"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/13960"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=13961"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=13961"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=13961"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}