llama.cpp Speculative Decoding Test GGUF Models Released

At a Glance
| Item | Value |
|---|---|
| Repository | ggml-org/spec-for-testing-GGUF |
| Published | 2026-10-30 |
| License | other |
| Formats | GGUF |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
ggml-org has released “ggml-org/spec-for-testing-GGUF", a repository containing target and draft GGUF models for speculative-decoding integration testing in llama.cpp. This repository includes GGUF models supporting various speculative decoding methods such as MTP, DFlash, DSpark, EAGLE-3, DFlash2, and LiLiCorr.
Claims and Evidence
Regarding “Qwen/Qwen3.5-4B", the base model of this repository, the presenter’s measurements show the following results across various benchmarks:
- It records 79.1 on MMLU-Pro, a general knowledge and reasoning benchmark.
- It records 52.9 on SuperGPQA, which tests specialized knowledge across a broad range of fields.
- It records 76.2 on GPQA Diamond, a difficult scientific benchmark.
- It records 89.8 on IFEval, which measures instruction-following capability.
- It records 55.8 on LiveCodeBench v6, which measures coding capability.
- It records 79.9 on \u03c4-bench, which measures agent tool use.
- It records 77.6 on MMMU, which measures specialized image understanding knowledge.
- It records 85.1 on Mathvista(mini), a math problem benchmark in image understanding.
Prerequisites
The prerequisites and support status for this repository and testing models are as follows:
- Target Repository:
ggml-org/spec-for-testing-GGUF - Use Case: Provision of target and draft GGUF models for speculative-decoding integration tests in llama.cpp
- Supported Methods: MTP, DFlash, DSpark, EAGLE-3, DFlash2, LiLiCorr
- Source Models:
Qwen/Qwen3.5-4B–Qwen/Qwen3-4B–z-lab/Qwen3-4B-DFlash-b16–deepseek-ai/dspark_qwen3_4b_block7–AngelSlim/Qwen3-4B_eagle3–Qwen/Qwen3.8-27B–incoai/Qwen3.8-27B-DFlash2–nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF–nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF- Basic Specifications of Source Model (Qwen3.5-4B): The parameter count is 4B. The native context length is 262,144 tokens, and it can be extended up to 1,010,000 tokens using YaRN. Compatible frameworks include Hugging Face Transformers, vLLM, SGLang, and KTransformers.
What Can Be Reproduced Locally
Readers can download GGUF files directly from Hugging Face and use them in their local environments.
- Acquiring Model Files: Target GGUF and draft GGUF for speculative decoding validation can be obtained from the
ggml-org/spec-for-testing-GGUFrepository. - Execution Procedures in Serving Environments: For the base model
Qwen/Qwen3.5-4B, example commands to launch a server with speculative decoding and various features enabled are publicly available.
An example command to launch a server with MTP (Multi-Token Prediction) enabled in SGLang is as follows:
python -m sglang.launch_server --model-path Qwen/Qwen3.5-4B --port 8000 --tp-size 1 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
An example command to launch with MTP enabled in vLLM is as follows:
vllm serve Qwen/Qwen3.5-4B --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
An example command to launch a server in Hugging Face Transformers is as follows:
transformers serve --force-model Qwen/Qwen3.5-4B --port 8000 --continuous-batching
What the Documentation Does Not Cover
- The documentation does not include comparative data on specific inference speed improvements (tokens/sec) or changes in memory consumption when applying each speculative decoding method (MTP, DFlash, DSpark, EAGLE-3, DFlash2, LiLiCorr).
- Specific command-line procedures for integrating and running tests with these test GGUF models on llama.cpp are not directly described within the model card.
Hardware Requirements
Estimated requirements (calculated by Local Model Watch) — 4.7B parameters (taken from the base model Qwen/Qwen3.5-4B)
| Your VRAM | Quantization | File size | Est. memory needed |
|---|---|---|---|
| 4GB (laptop iGPU / phone class) | Q4_K_M | 0.5GB | 0.6GB |
Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.
Related Articles
- Qwen3.5-4B-GGUF Vision-Language Model: 4GB+ VRAM
- ZwZ2-4B-GGUF Vision-Language Model: 4GB+ VRAM
- UserRepGen-4B Text Generation Model: 4GB+ VRAM, GGUF Builds
- Tev1-4B-experimental Text Generation Model: 4GB+ VRAM, GGUF Builds
What to Read Next
- Find models by VRAM (This model runs from the 4GB tier) → Other models that run on a 8GB GPU
- Engines mentioned in this article → ggml / llama.cpp
- What Q4_K_M mean → Quantization and model-format glossary
- How to read MMLU-Pro, SuperGPQA, GPQA Diamond → Benchmark glossary

