Ling-3.0-tiny-UBTree Draft Model for Speculative Decoding: 4GB+ VRAM

At a Glance
| Item | Value |
|---|---|
| Repository | inclusionAI/Ling-3.0-tiny-UBTree |
| Publisher guide | inclusionAI (Ant Group): models and licenses |
| Published | 2026-10-21 |
| License | mit |
| Formats | safetensors |
| Paper | arXiv:2609.39972 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
“inclusionAI/Ling-3.0-tiny-UBTree" is a draft model for speculative decoding used in combination with the lightweight inference model “inclusionAI/Ling-3.0-tiny“. This model does not perform text generation on its own; instead, it operates in parallel to accelerate the inference process of the target model, Ling-3.0-tiny.
According to the paper referenced by the model card, “UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding“, traditional parallel drafting methods face the issue that as the diversity (entropy) of the target model’s output probabilities increases, candidate diversity drops, reducing the acceleration effect. UBTree is designed to efficiently construct tree-structured draft candidates without sacrificing parallelism by combining a Unigram proposal part (a 5-layer DFlash backbone) that independently generates candidate tokens at each position with a Bigram selection part that predicts transition scores between adjacent candidates.
Specifications
The specifications of this model are published in the model card as follows. The parameter count included in the distributed checkpoints does not include the weights of the token embeddings and language model head shared with the target model.
- Architecture: Ling3UBTreeModel
- Parameters: 183,127,326 (183.1M)
- Draft backbone: 5 full-attention Transformer layers
- Weight data type: BF16
- Hidden size: 1,536
- Attention heads / KV heads: 16 / 16
- Target auxiliary feature layers (0-indexed): 4, 8, 12, 16, 22
- Draft block size: 16 (runtime convention including anchor)
- Candidate Top-k: 64 per draft position
- Default verification tree budget: 64 non-root nodes + 1 anchor
- Training method: DFlash warm start, joint training of backbone and UBTree selector
Meanwhile, the specifications of the target base model, “inclusionAI/Ling-3.0-tiny", feature a sparse MoE (Mixture of Experts) hybrid linear architecture with a total parameter count of 7.9B and 1.3B active parameters per token (alternating 3 layers of Kimi Delta Attention and 1 layer of Multi-Head Latent Attention).
Performance
Below is the “Benchmark results" table published in the public repository’s model card. At this stage, specific measured figures have not yet been reported by the publishers, and all items are listed as “TBD".
| Dataset | Acceptance length | Speedup |
|---|---|---|
| GSM8K | TBD | TBD |
| MATH-500 | TBD | TBD |
| AIME 2025 | TBD | TBD |
| HumanEval | TBD | TBD |
| MBPP | TBD | TBD |
| LiveCodeBench | TBD | TBD |
Regarding the evaluation metrics listed in this table, the model card defines “Acceptance length" as the average number of tokens determined per verification step including target tokens, and “Speedup" as the speedup ratio relative to autoregressive decoding with the target model alone under identical hardware, input, and decoding settings. Target benchmarks include GSM8K, which measures accuracy on grade-school word problems; MATH-500, covering high school to competition-level mathematics; AIME 2025, handling American Invitational Mathematics Examination problems; HumanEval and MBPP, measuring Python function implementation accuracy; and LiveCodeBench, which collects recent programming tasks.
While measurement results for this specific checkpoint are not yet filled in, the paper (arXiv:2609.39972) reports that the UBTree architecture itself achieved an average speedup of 5.84x to 6.94x over autoregressive decoding across seven standard benchmarks targeting Qwen3-4B and Qwen3-8B. Additionally, regarding the performance of the base model Ling-3.0-tiny itself, the model card notes scores of 25 on the Artificial Analysis Intelligence Index v4.1.1 and 16 on the Artificial Analysis Agentic Index, alongside reported output speeds exceeding 160 tokens/s in Artificial Analysis tests.
Strengths and Use Cases
This model, “inclusionAI/Ling-3.0-tiny-UBTree", is specialized for speculative decoding to parallelize inference and increase text generation speed in combination with its target model, “inclusionAI/Ling-3.0-tiny". It does not function as a standalone dialogue or text generation model and is designed specifically as a auxiliary draft model to assist the inference process of the target model. The assigned language tags are Chinese (zh) and English (en).
The strengths and intended use cases of the base model “Ling-3.0-tiny" to which this model is applied are described in the model card as follows:
- Hybrid Reasoning and Agent Capabilities: Combines rapid responses for routine tasks with multi-step reasoning for complex problems within a single model. Users can toggle the thinking mode on or off per request by setting the
enable_thinkingoption. It is designed to handle general agent tasks, coding, mathematical and scientific reasoning, and instruction-following in a well-balanced manner. - Efficient Long-Context Processing: Features a 3:1 alternating block layout combining 3 layers of Kimi Delta Attention (KDA) and 1 layer of Multi-Head Latent Attention (MLA), enabling efficient long-context processing.
- Deployment to Edge and Local Environments: Optimized to allow advanced reasoning capabilities in resource-constrained environments without requiring datacenter-class GPUs. Operation has been verified on environments such as NVIDIA DGX Spark, Apple Silicon MacBooks, and Mac minis.
As a speculative decoding technique, UBTree uses the Unigram proposal and Bigram selection parts to parallelize diverse and promising candidate branches even in scenarios where the target model’s output probability entropy is high. Therefore, it is well-suited for production environments involving long outputs or complex steps—such as the reasoning thinking modes and agent workflows that the base model excels at—to reduce latency and increase throughput while maintaining output quality.
Hardware Requirements
Estimated requirements (calculated by Local Model Watch) — 183M parameters
| Your VRAM | Quantization | File size | Est. memory needed |
|---|---|---|---|
| 4GB (laptop iGPU / phone class) | BF16 | 0.3GB | 0.4GB |
Inference engine support (architecture name matched against each project’s own model registry in its source code, checked 2026-11-17): llama.cpp: not registered, vLLM: not registered, MLX (mlx-lm): registered. “Not registered" means the name is absent from that registry today, not that the model cannot run.
Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.
Can You Run It Locally?
Not usable in Ollama, LM Studio and llama.cpp yet.
The publisher ships safetensors only, and llama.cpp’s registry does not list this architecture. llama.cpp would need to add support before these tools can run it. Today it can be run with transformers, using the memory figures in the table above.
License — mit (Commercial use allowed): Permits commercial use, modification and redistribution, provided the copyright notice and license text are retained.
Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists. License summaries are not legal advice — check the publisher’s original terms before relying on them.
Recent Models in the Same Size Class
Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.
→ Scroll horizontally to see all columns
| Model | Parameters | Smallest VRAM tier | License | Our article |
|---|---|---|---|---|
| openbmb/MiniCPM-V-4.7-35B-A3B-DSpark-gguf | 389M | 4GB | apache-2.0 | MiniCPM-V-4.7-35B-A3B-DSpark-gguf Draft Model for Speedup: 4GB+ VRAM (2026-11-17) |
| openbmb/MiniCPM-V-4.7-1B-DSpark-gguf | 214M | 4GB | apache-2.0 | MiniCPM-V-4.7-1B-DSpark-gguf Text Generation Model: 4GB+ VRAM (2026-11-17) |
| IFM/Jais-3-0.9B-GGUF | 1.1B | 4GB | apache-2.0 | Jais-3-0.9B-GGUF Bilingual Dense Model: 4GB+ VRAM (2026-11-17) |
| nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 | 4.0B | 4GB | openmdw-1.1 | NVIDIA Releases Dory1.0-4B: A Recurrent 4B LLM with Dynamic Loops (2026-11-17) |
| LiquidAI/LFM2.5-24B-A2B-DSpark | 328M | 4GB | lfm1.0 | LFM2.5-24B-A2B-DSpark Speculative Decoding Draft Model: 4GB+ VRAM (2026-11-17) |
How to Get It
Weight files for this model are distributed in safetensors format on the Hugging Face repository. It is released under the MIT License and is not a gated model requiring prior agreement to terms of use, allowing direct downloading from the repository.
Regarding inference engine support, the model card previews integration with SGLang. However, the Ling-3-specific UBTree adaptation code has not yet been publicly released to the open-source SGLang community, with the model card explicitly noting “SGLang integration — stay tuned!". Installation steps and serving recipes are scheduled to be added to the model card once official integration with SGLang is complete.
Meanwhile, inference instructions for the target base model “inclusionAI/Ling-3.0-tiny" are already available for SGLang, vLLM, and Ollama using MLX on Apple Silicon (via source builds at the pull request stage). However, running tree-style speculative decoding incorporating this checkpoint, “inclusionAI/Ling-3.0-tiny-UBTree", requires waiting for upcoming support updates from the SGLang side.
Related Articles
- SingVL_S1_4B_GGUF 4B Multimodal Decision-Making Model: 4GB+ VRAM
- ZoomGain-4B: 4GB+ VRAM, GGUF Builds
- SingVL_S1_4B Multimodal Decision Model with Image Support: 4GB+ VRAM
- ZwZ2-4B-GGUF Vision-Language Model: 4GB+ VRAM
What to Read Next
- Find models by VRAM (This model runs from the 4GB tier) → Other models that run on a 8GB GPU
- Formats this model is available in → Safetensors format guide and models
- Learn about the publisher → inclusionAI (Ant Group): models, licenses and articles
- Other models for the same task → Other text generation models
- How to read GSM8K, MATH, AIME → Benchmark glossary
Sources
- inclusionAI/Ling-3.0-tiny-UBTree (Hugging Face): https://huggingface.co/inclusionAI/Ling-3.0-tiny-UBTree
- inclusionAI/Ling-3.0-tiny (Hugging Face): https://huggingface.co/inclusionAI/Ling-3.0-tiny
- UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding (arXiv): https://arxiv.org/abs/2609.39972

