llama.cpp v0.7.0 Released with MoE GPU Cache and New Models

At a Glance
| Item | Value |
|---|---|
| Repository | ggml-org/llama.cpp |
| Version | v0.7.0 |
| Published | 2026-10-16 |
| License | MIT |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
llama.cpp v0.7.0 has been released. llama.cpp is a large language model (LLM) inference engine implemented in C/C++.
The most important change in this update is the addition of a new GPU caching feature for MoE (Mixture of Experts) models. Experts kept in host memory can now be executed via a GPU LRU cache, improving decode speeds by up to approximately 2.2x in environments where experts are offloaded.
Breaking Changes / Deprecations
The following changes may affect existing environments and applications.
| Item | Old | New |
|---|---|---|
| Server default port | 8080 | 9931 |
| Chat output data format | Single string | Array of parts |
| PLaMo-3 tokenization | Previous method | plamo3 (with pre-segmentation) |
Because the server port number has changed, existing connection settings or scripts must be updated to port 9931 (though Docker images will continue to use 8080). Additionally, applications using the chat API must handle the output changing from a string to an array. When using the PLaMo-3 model, update tokenizer.ggml.model to “plamo3" or re-convert the GGFU file.
Key Changes
GPU Cache for MoE Experts
A feature has been added to execute MoE experts held in host memory using an LRU cache on the GPU. The cache size can be specified with the --moe-cache-mib option. In experiments using an RTX 5090, it is reported that the decode speed improved from 30.8 t/s to 67.8 t/s (2.20x). This represents a massive performance improvement for users operating with experts offloaded from the GPU.
Enhanced Speculative Decoding
LiliCorr speculative decoding and MTP (Multi-Token Prediction) for GLM5-Next have been added. This is expected to speed up inference in specific model architectures.
Reduced VRAM Usage for Embedding Models
When the pooling type is set to anything other than NONE, embedding model generation processes now skip the LM head and logits buffer. This reduces VRAM usage by up to 10x. Memory efficiency is significantly improved in workloads that frequently use embedding models.
Web UI Model Management Features
A new “Models Manager" has been added to the Web UI. All provided models can be managed in a single table, allowing for quantization format expansion, searching, filtering, sorting, and favoriting. Operations such as loading/unloading models, deleting them from disk, and resumable downloads with progress indicators can also be performed directly from the UI.
Decision-Making Model Support
Decision-making models with new architectures have been supported, including LiquidAI d1/d1-omni, vLLM decision 2/d3, and pplx-decider.
Supported Models and Hardware
This release supports a vast number of new model architectures. Specifically, GGFU conversion and inference are now possible for the following models:
- K2 Horizon: Supports both dense and MoVA variants, with splitting via
-sm tensoralso available. - Prism Bonsai 2 27B: Added support for ternary models.
- EmbeddingGemma-2: Supports text, vision, and audio embeddings.
- Multimodal models: Added support for MiniCPM-V 4.7, Nemotron Super 3.5 VL, Nemotron Bolt 74B A9B, and the cohere2 vision encoder.
- Decision-making models: Supported models include LiquidAI d1-3B, d1-omni-600M, vLLM decision 2, vLLM d3, and pplx-decider.
- Others: Includes maion-coder, Mellum, and Rerankers (including exact GELU support for ModernBERT encoders).
On the hardware front, accompanied by the update to ggml v0.27.0, optimizations are progressing for various accelerators, such as chunked gated-delta-net kernels in CUDA/HIP, KleidiAI SME2 k-quant matmuls and AVX2 Q2_0 dot products on the CPU, L2-aware coopmat2 scheduling and bf16 activations in Vulkan, and tensor-API and wider-head-dim flash attention in Metal.
How to Get It
To update llama.cpp, fetch the latest source code from the repository and rebuild it.
git pull
make
(* Please adjust specific build commands as appropriate for your environment and build system)
Speed on Our CPU
Measured by Local Model Watch on our CPU-only server (Neoverse-N1, 3 threads) with llama-bench (512-token prompt, 128-token generation), always on the same model: Qwen3-0.6B-Q4_K_M.gguf from unsloth/Qwen3-0.6B-GGUF (revision 50968a4468). Versions like v0.5.0 ship no binaries, so we used the first official ubuntu-arm64 build that contains the version; “+N” is how many commits that build is ahead of it. These are CPU figures, not a guide to GPU speed, and small differences can be run-to-run noise (± is the standard deviation).
| Version | Build measured | Prompt | Generation |
|---|---|---|---|
| v0.7.0 | b11669 | 128.4 ± 0.1 tok/s | 43.1 ± 0.2 tok/s |
| v0.6.0 | b11429 | 128.4 ± 0.6 tok/s | 43.0 ± 1.4 tok/s |
Related Articles
- llama.cpp v1.0.0 Released: Major Updates and New Features
- llama.cpp v0.9.0 Released: FP8 Quantization and More
- llama.cpp v0.8.0: Multimodal Enhancements and LFM3 Support
- llama.cpp v0.6.0 Released with llama_batch_ext and Metal Boosts
What to Read Next
- Follow this tool → llama.cpp overview and release history (9 releases tracked)
- Other inference engines and runtimes → Ollama / vLLM / SGLang
Sources
- ggml-org/llama.cpp v0.7.0 Release Notes
- PR #29887: add a GPU cache for MoE experts kept in host memory
- PR #30045: vocab : implement PLaMo-3 tokenizer pre-segmentation
- PR #30498: server: save only restorable slot states to the prompt cache
- PR #30160: graph : reorder get_rows for embeddings
- PR #30461: models: drop the t_embd expand of the embedding graphs
- PR #30198: models: remove redundant ggml_cont calls now supported by CUDA unary kernels
- PR #30042: llama: remove the gather path of the glm5-next sparse attention

