Tencent Releases Simple Attention Sparsification for Qwen3
At a Glance
| Item | Value |
|---|---|
| Repository | tencent/Simple-Attention-Sparsification |
| Published | 2026-09-14 |
| Formats | PyTorch (pth) |
| Paper | arXiv:2609.13141 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code at collection time. Dates are JST.
Overview
Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models. SAS is a sparse attention method that uses continuous gates to select and rank KV blocks, allowing end-to-end optimization via language modeling loss. The released checkpoints contain only the router portion, and running them requires the corresponding Qwen3 base model and a dedicated backend.
Specifications
- Base model: Qwen/Qwen3-4B
- Tags: qwen3, sparse-attention, long-context, sglang, sas, text-generation
- Checkpoint type: Router-only checkpoint (the Qwen3 backbone is frozen and not bundled)
Specifications and Configuration Table
| Directory | Base model | Gate parameters | Gate checkpoint size |
|---|---|---|---|
Qwen3-4B-AttnGates |
Qwen3-4B | 33.0M | 64 MiB |
Qwen3-8B-AttnGates |
Qwen3-8B | 33.0M | 64 MiB |
Qwen3-14B-AttnGates |
Qwen3-14B | 42.0M | 81 MiB |
| Setting | Value |
|---|---|
| KV block size | 64 tokens |
| Training Top-K | 31 historical blocks |
| Gate hidden size | 128 |
| Query projection | Qproj |
| Key block pooling | max + min + average |
| Q/K normalization | enabled |
| Gate RoPE | enabled |
| Training sequence length | 32,768 tokens |
| Training data | OpenR1-Math-220k |
Strengths and Use Cases
The underlying base model (Qwen/Qwen3-4B) features the ability to switch within a single model between a “thinking mode" for complex logical reasoning, mathematics, and coding, and a “non-thinking mode" for efficient general-purpose dialogue. It also features high agent capabilities in tool calling, support for over 100 languages and dialects, and native support for a 32,768-token context length (with extended context support also possible using YaRN).
The introduction of SAS checkpoints enables efficient context processing and inference in an SGLang-based sparse attention environment.
Hardware Requirements
Estimated requirements (calculated by Local Model Watch) — 4.0B parameters (taken from the base model Qwen/Qwen3-4B)
| Your VRAM | Quantization | File size | Est. memory needed |
|---|---|---|---|
| 12GB (RTX 4070 / 3060 12GB, etc.) | BF16 | 7.5GB | 9.0GB |
Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.
Recent Models in the Same Size Class
Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.
→ Scroll horizontally to see all columns
| Model | Parameters | Smallest VRAM tier | License | Our article |
|---|---|---|---|---|
| harshatheg/Qwen-2.5-1B-RLCD | 1.5B | 8GB | apache-2.0 | Fast Structured Generation on Apple Silicon with MLX and Qwen (2026-09-16) |
| openbmb/MiniCPM5-2B | 2.5B | 4GB | apache-2.0 | OpenBMB Releases MiniCPM5-2B: A SOTA 2B On-Device Model (2026-09-07) |
How to Get It
Use after cloning the repository and setting up the dedicated environment.
git clone https://github.com/Tencent-Hunyuan/Simple-Attention-Sparsification.git
cd Simple-Attention-Sparsification
git submodule update --init --recursive
curl -fsSL https://pixi.sh/install.sh | bash
pixi install
cd third_party/sglang-blocksparse
pixi install
cd../..
For inference, use the seer_attn backend within the sglang-blocksparse fork.