llama.cpp v0.7.0 Released with MoE GPU Cache and New Models

llama.cpp v0.7.0 Released with MoE GPU Cache and New Models

At a Glance

Item Value
Repository ggml-org/llama.cpp
Version v0.7.0
Published 2026-10-16
License MIT
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

llama.cpp v0.7.0 has been released. llama.cpp is a large language model (LLM) inference engine implemented in C/C++.

The most important change in this update is the addition of a new GPU caching feature for MoE (Mixture of Experts) models. Experts kept in host memory can now be executed via a GPU LRU cache, improving decode speeds by up to approximately 2.2x in environments where experts are offloaded.

Breaking Changes / Deprecations

The following changes may affect existing environments and applications.

Item Old New
Server default port 8080 9931
Chat output data format Single string Array of parts
PLaMo-3 tokenization Previous method plamo3 (with pre-segmentation)

Because the server port number has changed, existing connection settings or scripts must be updated to port 9931 (though Docker images will continue to use 8080). Additionally, applications using the chat API must handle the output changing from a string to an array. When using the PLaMo-3 model, update tokenizer.ggml.model to “plamo3" or re-convert the GGFU file.

Key Changes

GPU Cache for MoE Experts

A feature has been added to execute MoE experts held in host memory using an LRU cache on the GPU. The cache size can be specified with the --moe-cache-mib option. In experiments using an RTX 5090, it is reported that the decode speed improved from 30.8 t/s to 67.8 t/s (2.20x). This represents a massive performance improvement for users operating with experts offloaded from the GPU.

Enhanced Speculative Decoding

LiliCorr speculative decoding and MTP (Multi-Token Prediction) for GLM5-Next have been added. This is expected to speed up inference in specific model architectures.

Reduced VRAM Usage for Embedding Models

When the pooling type is set to anything other than NONE, embedding model generation processes now skip the LM head and logits buffer. This reduces VRAM usage by up to 10x. Memory efficiency is significantly improved in workloads that frequently use embedding models.

Web UI Model Management Features

A new “Models Manager" has been added to the Web UI. All provided models can be managed in a single table, allowing for quantization format expansion, searching, filtering, sorting, and favoriting. Operations such as loading/unloading models, deleting them from disk, and resumable downloads with progress indicators can also be performed directly from the UI.

Decision-Making Model Support

Decision-making models with new architectures have been supported, including LiquidAI d1/d1-omni, vLLM decision 2/d3, and pplx-decider.

Supported Models and Hardware

This release supports a vast number of new model architectures. Specifically, GGFU conversion and inference are now possible for the following models:

  • K2 Horizon: Supports both dense and MoVA variants, with splitting via -sm tensor also available.
  • Prism Bonsai 2 27B: Added support for ternary models.
  • EmbeddingGemma-2: Supports text, vision, and audio embeddings.
  • Multimodal models: Added support for MiniCPM-V 4.7, Nemotron Super 3.5 VL, Nemotron Bolt 74B A9B, and the cohere2 vision encoder.
  • Decision-making models: Supported models include LiquidAI d1-3B, d1-omni-600M, vLLM decision 2, vLLM d3, and pplx-decider.
  • Others: Includes maion-coder, Mellum, and Rerankers (including exact GELU support for ModernBERT encoders).

On the hardware front, accompanied by the update to ggml v0.27.0, optimizations are progressing for various accelerators, such as chunked gated-delta-net kernels in CUDA/HIP, KleidiAI SME2 k-quant matmuls and AVX2 Q2_0 dot products on the CPU, L2-aware coopmat2 scheduling and bf16 activations in Vulkan, and tensor-API and wider-head-dim flash attention in Metal.

How to Get It

To update llama.cpp, fetch the latest source code from the repository and rebuild it.

git pull
make

(* Please adjust specific build commands as appropriate for your environment and build system)

Speed on Our CPU

Measured by Local Model Watch on our CPU-only server (Neoverse-N1, 3 threads) with llama-bench (512-token prompt, 128-token generation), always on the same model: Qwen3-0.6B-Q4_K_M.gguf from unsloth/Qwen3-0.6B-GGUF (revision 50968a4468). Versions like v0.5.0 ship no binaries, so we used the first official ubuntu-arm64 build that contains the version; “+N” is how many commits that build is ahead of it. These are CPU figures, not a guide to GPU speed, and small differences can be run-to-run noise (± is the standard deviation).

Version Build measured Prompt Generation
v0.7.0 b11669 128.4 ± 0.1 tok/s 43.1 ± 0.2 tok/s
v0.6.0 b11429 128.4 ± 0.6 tok/s 43.0 ± 1.4 tok/s

Related Articles

What to Read Next

Sources