GGML v0.29.0 Released: Multi-Node Tensor Parallelism and More

GGML v0.29.0 Released: Multi-Node Tensor Parallelism and More

At a Glance

Item Value
Repository ggml-org/ggml
Version v0.29.0
Published 2026-11-04
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

Version v0.29.0 of ggml-org/ggml (MIT License), a C++ tensor library for machine learning, has been released.

This version adds initial support for a multi-process communicator interface aimed at multi-node tensor parallelism, along with the introduction of OCP FP8 E4M3 tensor types and new Flash Attention operators. Furthermore, extensive optimizations and bug fixes have been applied across a wide range ofbackends including CPU, CUDA, SYCL, Metal, Vulkan, OpenCL, and Hexagon, improving the performance and stability of local inference and computation.

Key Changes

Multi-Node Tensor Parallelism and Communication API Addition A new multi-process communicator interface (ggml_backend_comm_init_rank_t) has been introduced for collective communication between different ranks. For the CUDA backend, NCCL support—available via an RPC server—has been added through this interface, enabling tensor parallelism across multiple nodes. Additionally, features to enhance execution efficiency in multi-GPU and distributed environments have been assembled, such as the addition of ggml_sycl_comm_exchange() for data exchange in SYCL and tensor parallel allreduce functionality from host shared buffers in Vulkan.

OCP FP8 E4M3 Support and New Flash Attention Operators GGML_TYPE_F8_E4M3 (OCP FP8 E4M3) and float conversion helpers have been added as new data types. Along with this, the ggml_flash_attn_ext_rows() operator has been introduced, where each query slice uses an index tensor to reference its own Key/Value row list for attention calculations. This operator is implemented in the CPU, CUDA, and Metal backends, enabling efficient processing of multiple sequences.

Direct GGUF Writing and Improved Memory Allocation on Apple Platforms As a core feature improvement, the GGUF writer now supports zero-copy processing, allowing tensor data on host buffers to be written directly to the output stream without intermediate copies. This improves writing efficiency during export. Also, for large-scale memory allocations on Apple platforms, the method has been switched from the conventional posix_memalign to using vm_allocate. In addition, the IQ4 quantization processing aims to improve generation speed by using pre-calculated midpoints.

Performance Improvements for CPU and GPU Backends Kernel optimizations for various hardware environments have been implemented. In the CPU backend, MXFP4 GEMV, Q6_K GEMM, and q8_0 4×8 GEMV kernels utilizing 256-bit SVE have been added. In the CUDA backend, fused processing for MoE gate/up projections and GLU has been implemented in MMQ. In SYCL, DPAS Flash Attention kernels supporting quantized KV caches (q4_0/q8_0) for Intel XMX have been added, and in Metal, matrix-vector product operations for MXFP4 have been accelerated. The OpenCL backend has also expanded GEMV/GEMM support for quantization formats such as Q2_0, IQ1, IQ2, IQ3, and IQ4_XS.

Bug Fixes Across Various Backends Fixes have been implemented to prevent calculation accuracy errors and unexpected behavior. In the CUDA backend, issues such as calculation errors in Q8_1 correction sums in MMQ and the erroneous pre-releasing of CUDA graphs have been fixed. In the Metal backend, a bug in the cpy kernel for destinations that are non-contiguous in the 0th dimension has been resolved. In the Vulkan backend, out-of-bounds access when referencing NVFP4 UE4M3 scales was fixed, alongside fixes for rounding processing from f32 to q2_0/q4_0/q5_0 and operation bugs in FLASH_ATTN_EXT in SYCL.

Supported Models and Hardware

With the introduction of the new data type GGML_TYPE_F8_E4M3 (OCP FP8 E4M3), calculations using the FP8 format become possible. Additionally, in the OpenCL backend, GEMV/GEMM support has been expanded for quantization formats such as Q2_0, IQ1, IQ2, IQ3, and IQ4_XS.

On the hardware front, the following optimizations are included:
– CPU: Addition of MXFP4 GEMV, Q6_K GEMM, and q8_0 4×8 GEMV kernels utilizing 256-bit SVE.
– CUDA: Enabling sparse attention in CDNA.
– SYCL: Addition of DPAS flash attention kernels supporting quantized KV caches (q4_0/q8_0) for Intel XMX.
– Metal: Acceleration of MXFP4 mat-vec.
– Vulkan: Improved small-n split-K selection in RDNA3 and RDNA4.
– Hexagon: Support for new operators such as LIGHTNING_INDEXER, TOP_K_MASK, UPSCALE, CONV_2D, and DSV4_HC_PRE/POST.

How to Get It

If you are building from source, you can do so with the following steps:

git clone https://github.com/ggml-org/ggml
cd ggml

mkdir build && cd build
cmake..
cmake --build. --config Release -j 8

Related Articles

What to Read Next

Sources