Simple-Attention-Sparsification Text Generation Model: 12GB+ VRAM

September 14, 2026October 4, 2026

At a Glance

Values determined by this site’s code when the information was collected. Dates are JST.

What we checked ourselves

• The same Japanese text takes about as many tokens as with the Qwen3 tokenizer.

Details and conditions are in “Our Own Measurements” below.

Overview

Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models. SAS is a sparse attention method that uses continuous gates to select and rank KV blocks, allowing end-to-end optimization via language modeling loss. The released checkpoints contain only the router portion, and running them requires the corresponding Qwen3 base model and a dedicated backend.

Specifications

• Base model: Qwen/Qwen3-4B
• Tags: qwen3, sparse-attention, long-context, sglang, sas, text-generation
• Checkpoint type: Router-only checkpoint (the Qwen3 backbone is frozen and not bundled)

Specifications and Configuration Table

Strengths and Use Cases

The underlying base model (Qwen/Qwen3-4B) features the ability to switch within a single model between a “thinking mode" for complex logical reasoning, mathematics, and coding, and a “non-thinking mode" for efficient general-purpose dialogue. It also features high agent capabilities in tool calling, support for over 100 languages and dialects, and native support for a 32,768-token context length (with extended context support also possible using YaRN).

The introduction of SAS checkpoints enables efficient context processing and inference in an SGLang-based sparse attention environment.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 4.0B parameters (taken from the base model Qwen/Qwen3-4B)

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

Not usable in Ollama, LM Studio and llama.cpp yet — we have found no GGUF build.

The publisher ships PyTorch (pth) only.

Compiled by this site’s code from the published formats, converted builds we have found, and each engine’s own model registry. “Not found" means we have not seen such a build, not that none exists.

Our Own Measurements

Values we checked ourselves on our server (no GPU) by actually reading this model’s files, without running the model — not figures copied from the model card. How we measure, and the results for every model: Our Measurements.

Measurements We Did Not Take

This model does not state a license, so we cannot confirm permission to use it. Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation). Only values we checked without running the model, such as token counts and the GGUF header, are shown.

Japanese Token Efficiency

The same Japanese text takes about as many tokens as with the Qwen3 tokenizer.

Counted with the tokenizer.json of Qwen/Qwen3-4B on a fixed text we wrote ourselves (876 Japanese characters across news, conversation, technical docs, a formal email, travel writing and a recipe) and its English translation. Fewer tokens mean more Japanese fits in the context window.

Recent Models in the Same Size Class

Models with up to 4B parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site’s estimates; licenses are as stated on the model cards.

How to Get It

Use after cloning the repository and setting up the dedicated environment.

git clone https://github.com/Tencent-Hunyuan/Simple-Attention-Sparsification.git
cd Simple-Attention-Sparsification

git submodule update --init --recursive

curl -fsSL https://pixi.sh/install.sh | bash
pixi install

cd third_party/sglang-blocksparse
pixi install
cd../..

For inference, use the seer_attn backend within the sglang-blocksparse fork.

Related Articles

• Tencent Releases WeVisDoc-2B and WeVisDoc-4B Document Parsing Models

What to Read Next

• Find models by VRAM (This model runs from the 12GB tier) → Other models that run on a 12GB GPU
• Learn about the publisher → Tencent: models, licenses and articles
• Other models for the same task → Other text generation models

Sources

• https://huggingface.co/tencent/Simple-Attention-Sparsification
• https://huggingface.co/Qwen/Qwen3-4B

Update History

• 2026-09-24: Rebuilt the article: Added a “can you run it locally?" verdict from the distribution formats, converted builds, engine registries, license terms and measured compression
• 2026-09-25: Changed the title to show what the article covers (VRAM requirements, file list, etc.).
• 2026-09-28: Added our own measurements: Japanese token efficiency.