ggml v0.27.0 Released with Tensor-Parallel RPC and Major Speedups

ggml v0.27.0 Released with Tensor-Parallel RPC and Major Speedups

At a Glance

Item Value
Repository ggml-org/ggml
Version v0.27.0
Published 2026-10-15
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

Version 0.27.0 of ggml, the C++ tensor library for machine learning, has been released. ggml serves as the low-level core library for local LLM execution runtimes such as llama.cpp.

This release updates the RPC protocol to major version 8, introduces a tensor-parallel RPC server mode, and brings substantial performance-boosting kernels across major backends including CUDA, CPU, Vulkan, SYCL, and Metal. This is an important update for users performing distributed inference over a network or those looking to squeeze maximum inference speed out of the latest quantization methods and architectures on various hardware.

Breaking Changes and Deprecations

The following changes have been made regarding the RPC communication protocol and library symbol visibility.

Item Old (v0.26.0 and earlier) New (v0.27.0)
RPC Protocol RPC_PROTO_MAJOR_VERSION 7 RPC_PROTO_MAJOR_VERSION 8
Default Symbol Visibility Public (exports unintended internal symbols as well) Hidden (unnecessary internal symbols are no longer exported)
Duplicate expert specification for ggml_mul_mat_id Undefined (implementation-dependent looping behavior, etc.) Explicitly specified as “undefined" in the spec

In environments performing distributed execution across multiple machines using RPC features, both clients and servers must be updated to v0.27.0 simultaneously. Because the protocol major version has been incremented, mixed communication with older versions will not work. Additionally, developers directly linking ggml as a shared library should verify that they do not depend on internal symbols that have now been hidden.

Key Changes

RPC Tensor-Parallel Server Support

A tensor-parallel server mode has been introduced to the RPC (Remote Procedure Call) backend. It supports asynchronous graph computations, custom all-reduce operations, and 2D tensor transfers. This improves efficiency in environments where tensors are distributed across multiple compute nodes via a network to cooperatively run large-scale models. Furthermore, the environment variable GGML_RPC_DEBUG is now treated as a verbosity level, improving the readability of transport logs and tensor transfer logs.

Scheduler API Extensions

Two callback functions have been added to the backend scheduler: a copy callback (ggml_backend_sched_set_copy_callback) allowing users to control split weight copies on the application side, and a refactoring callback (ggml_backend_sched_set_realloc_callback) to notify about compute buffer reallocations during inference. This enables applications to exercise fine-grained control over multi-device memory layouts and dynamic buffer management.

Expansion of Optimization Paths in Various Backends

Extensive performance-boosting kernels have been added for various hardware accelerators.
– CUDA / HIP: Chunked kernels for gated-delta-net were introduced, increasing the number of state columns per warp. Support for 64-row MMQ tiles for RDNA3.5, BF16 support for XIELU, and improvements to top-k algorithm selection were also implemented.
– CPU: An AVX2 dot product kernel for Q2_0 was added. Furthermore, matrix multiplication for Q4_K is now supported in Arm SME2 environments via KleidiAI, and Q6_K is optimized to route through the Q8_0 path. FP16-to-FP32 lookup tables when F16C is enabled on x86 environments have also been restored.
– Vulkan: L2 cache-aware workgroup scheduling was implemented for coopmat2 matrix multiplication, along with support for bf16 activations in mul_mat. On AMD environments, an optimization was added to copy strided FP16 KV caches once and distribute reads across memory channels.
– SYCL: Added weight reordering and arithmetic decoding for MXFP4, IQ3, and Q5_K to accelerate MoE, grouped MoE XMX GEMM, and faster MLA prefill using MKL flash attention.
– Metal: Tensor API flash attention kernels for BF16 were added, making head-dimension 128/96 kernels available across data types.
– OpenCL: dp4a-based binary GEMM kernels for MoE were introduced, alongside expanded head dimension support for flash attention.

Correctness and Memory Safety Fixes

A bug in ggml_acc and ggml_set where negative offset values bypassed boundary checks during sign extension, causing out-of-bounds writes, has been fixed. Additionally, the output layout of ggml_conv_1d during batch inputs was fixed to correctly be [N, OC, OL], and CLAMP operations on non-contiguous views were revised to work properly in the CPU and CUDA backends.

Supported Models and Hardware

ggml v0.27.0 brings optimizations and feature additions for numerous processor architectures and quantization formats. The specific support details that users can benefit from on their hardware environments are as follows:

Expansion of Quantization Formats

  • MXFP4 / IQ3 / Q5_K: Weight reordering and arithmetic decoding for Mixture of Experts (MoE) calculations were introduced in the SYCL backend.
  • Q2_0: An AVX2 dot product kernel was added to the CPU backend, improving processing efficiency for low-bit quantized models.
  • Q4_K / Q6_K: Support for Q4_K matrix multiplication in Arm SME2 environments (via KleidiAI) and dp4a-based Q4_K / Q6_K (vs Q8_1) MoE binary GEMM kernels in OpenCL environments were added. GET_ROWS tiling processing for Q4_K / Q6_K was also implemented for Hexagon DSP.
  • Q8_0: Support for Q8_0 matrix multiplication was newly added in the zDNN backend (for IBM zSystems).

Enhanced Support for Model Architectures

  • MoE (Mixture of Experts) Models: MoE inference processing has been enhanced across backends, including grouped XMX GEMM in SYCL and the addition of dp4a kernels in OpenCL (such as Qualcomm Adreno).
  • MLA (Multi-head Latent Attention) Models: Faster MLA prefill using MKL flash attention was added to the SYCL backend, streamlining context processing for models adopting the MLA architecture.
  • Gated-Delta-Net Models: Chunked kernels were introduced for CUDA and HIP environments, and arithmetic fusion of alpha gates (add + unary + mul) was performed in SYCL.
  • Flash Attention Extensions: The supported range of head dimensions was expanded in the OpenCL backend to handle configurations such as dk512. Head dimension 128 and 96 kernels were also established in the Metal backend, along with the addition of BF16 Tensor API kernels.

Hardware Accelerator Support

  • CPU (x86 / Arm): KleidiAI acceleration for Arm SME2, and F16C detection fixes and FP16-to-FP32 lookup table restoration for AVX2 environments on x86.
  • CUDA / AMD HIP: 64-row MMQ tiles are supported for select quantization types on the AMD RDNA3.5 architecture. Adjustments were made to prioritize MMVQ when executing MUL_MAT_ID on older NVIDIA architectures (sm_60).
  • Vulkan: Shader counts were defined for Intel Arc Pro B65 and B70, and fixes were applied to address memory read imbalances and slowdowns on AMD GPUs and integrated GPUs (iGPUs). Sparse flash attention via coopmat2 and L2 cache-optimized scheduling are also available.
  • Qualcomm Hexagon DSP: A feature to configure HTP DCVS/HMX power modes via the GGML_HEXAGON_POWER_MODE environment variable was added. Major revisions to copy processing utilizing DMA and HVX, as well as the addition of IM2COL_3D and pooling processing, were performed.
  • Others: WebGPU backend now supports 2D workgroup dispatch, and OpenVINO backend received fixes for GPU stateful execution.

How to Get It

ggml is a C++ library built from source. To build and use the latest version in your local environment, run the following steps:

git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake..
cmake --build. --config Release -j 8

If you are using an existing repository, run git fetch and git checkout v0.27.0 inside your working directory before rebuilding.

Related Articles

What to Read Next

Sources