llama.cpp Speculative Decoding Test GGUF Models Released

llama.cpp Speculative Decoding Test GGUF Models Released

At a Glance

Item Value
Repository ggml-org/spec-for-testing-GGUF
Published 2026-10-30
License other
Formats GGUF
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

ggml-org has released “ggml-org/spec-for-testing-GGUF", a repository containing target and draft GGUF models for speculative-decoding integration testing in llama.cpp. This repository includes GGUF models supporting various speculative decoding methods such as MTP, DFlash, DSpark, EAGLE-3, DFlash2, and LiLiCorr.

Claims and Evidence

Regarding “Qwen/Qwen3.5-4B", the base model of this repository, the presenter’s measurements show the following results across various benchmarks:

  • It records 79.1 on MMLU-Pro, a general knowledge and reasoning benchmark.
  • It records 52.9 on SuperGPQA, which tests specialized knowledge across a broad range of fields.
  • It records 76.2 on GPQA Diamond, a difficult scientific benchmark.
  • It records 89.8 on IFEval, which measures instruction-following capability.
  • It records 55.8 on LiveCodeBench v6, which measures coding capability.
  • It records 79.9 on \u03c4-bench, which measures agent tool use.
  • It records 77.6 on MMMU, which measures specialized image understanding knowledge.
  • It records 85.1 on Mathvista(mini), a math problem benchmark in image understanding.

Prerequisites

The prerequisites and support status for this repository and testing models are as follows:

  • Target Repository: ggml-org/spec-for-testing-GGUF
  • Use Case: Provision of target and draft GGUF models for speculative-decoding integration tests in llama.cpp
  • Supported Methods: MTP, DFlash, DSpark, EAGLE-3, DFlash2, LiLiCorr
  • Source Models:
  • Qwen/Qwen3.5-4B – Qwen/Qwen3-4B – z-lab/Qwen3-4B-DFlash-b16 – deepseek-ai/dspark_qwen3_4b_block7 – AngelSlim/Qwen3-4B_eagle3 – Qwen/Qwen3.8-27B – incoai/Qwen3.8-27B-DFlash2 – nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF – nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF
  • Basic Specifications of Source Model (Qwen3.5-4B): The parameter count is 4B. The native context length is 262,144 tokens, and it can be extended up to 1,010,000 tokens using YaRN. Compatible frameworks include Hugging Face Transformers, vLLM, SGLang, and KTransformers.

What Can Be Reproduced Locally

Readers can download GGUF files directly from Hugging Face and use them in their local environments.

  • Acquiring Model Files: Target GGUF and draft GGUF for speculative decoding validation can be obtained from the ggml-org/spec-for-testing-GGUF repository.
  • Execution Procedures in Serving Environments: For the base model Qwen/Qwen3.5-4B, example commands to launch a server with speculative decoding and various features enabled are publicly available.

An example command to launch a server with MTP (Multi-Token Prediction) enabled in SGLang is as follows:

python -m sglang.launch_server --model-path Qwen/Qwen3.5-4B --port 8000 --tp-size 1 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

An example command to launch with MTP enabled in vLLM is as follows:

vllm serve Qwen/Qwen3.5-4B --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

An example command to launch a server in Hugging Face Transformers is as follows:

transformers serve --force-model Qwen/Qwen3.5-4B --port 8000 --continuous-batching

What the Documentation Does Not Cover

  • The documentation does not include comparative data on specific inference speed improvements (tokens/sec) or changes in memory consumption when applying each speculative decoding method (MTP, DFlash, DSpark, EAGLE-3, DFlash2, LiLiCorr).
  • Specific command-line procedures for integrating and running tests with these test GGUF models on llama.cpp are not directly described within the model card.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 4.7B parameters (taken from the base model Qwen/Qwen3.5-4B)

Your VRAM Quantization File size Est. memory needed
4GB (laptop iGPU / phone class) Q4_K_M 0.5GB 0.6GB

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Related Articles

What to Read Next

Sources