{"id":623,"date":"2026-09-14T20:34:12","date_gmt":"2026-09-14T11:34:12","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/14\/tencent-simple-attention-sparsification-qwen3\/"},"modified":"2026-09-18T21:42:01","modified_gmt":"2026-09-18T12:42:01","slug":"tencent-simple-attention-sparsification-qwen3","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/14\/tencent-simple-attention-sparsification-qwen3\/","title":{"rendered":"Tencent Releases Simple Attention Sparsification for Qwen3"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification\">tencent\/Simple-Attention-Sparsification<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-09-14<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td>PyTorch (pth)<\/td>\n<\/tr>\n<tr>\n<td>Paper<\/td>\n<td><a href=\"https:\/\/arxiv.org\/abs\/2609.13141\">arXiv:2609.13141<\/a><\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code at collection time. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models. SAS is a sparse attention method that uses continuous gates to select and rank KV blocks, allowing end-to-end optimization via language modeling loss. The released checkpoints contain only the router portion, and running them requires the corresponding Qwen3 base model and a dedicated backend.<\/p>\n<h2>Specifications<\/h2>\n<ul>\n<li>Base model: Qwen\/Qwen3-4B<\/li>\n<li>Tags: qwen3, sparse-attention, long-context, sglang, sas, text-generation<\/li>\n<li>Checkpoint type: Router-only checkpoint (the Qwen3 backbone is frozen and not bundled)<\/li>\n<\/ul>\n<h2>Specifications and Configuration Table<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Directory<\/th>\n<th>Base model<\/th>\n<th style=\"text-align: right;\">Gate parameters<\/th>\n<th style=\"text-align: right;\">Gate checkpoint size<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>Qwen3-4B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-4B\">Qwen3-4B<\/a><\/td>\n<td style=\"text-align: right;\">33.0M<\/td>\n<td style=\"text-align: right;\">64 MiB<\/td>\n<\/tr>\n<tr>\n<td><code>Qwen3-8B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-8B\">Qwen3-8B<\/a><\/td>\n<td style=\"text-align: right;\">33.0M<\/td>\n<td style=\"text-align: right;\">64 MiB<\/td>\n<\/tr>\n<tr>\n<td><code>Qwen3-14B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-14B\">Qwen3-14B<\/a><\/td>\n<td style=\"text-align: right;\">42.0M<\/td>\n<td style=\"text-align: right;\">81 MiB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Setting<\/th>\n<th style=\"text-align: right;\">Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>KV block size<\/td>\n<td style=\"text-align: right;\">64 tokens<\/td>\n<\/tr>\n<tr>\n<td>Training Top-K<\/td>\n<td style=\"text-align: right;\">31 historical blocks<\/td>\n<\/tr>\n<tr>\n<td>Gate hidden size<\/td>\n<td style=\"text-align: right;\">128<\/td>\n<\/tr>\n<tr>\n<td>Query projection<\/td>\n<td style=\"text-align: right;\"><code>Qproj<\/code><\/td>\n<\/tr>\n<tr>\n<td>Key block pooling<\/td>\n<td style=\"text-align: right;\">max + min + average<\/td>\n<\/tr>\n<tr>\n<td>Q\/K normalization<\/td>\n<td style=\"text-align: right;\">enabled<\/td>\n<\/tr>\n<tr>\n<td>Gate RoPE<\/td>\n<td style=\"text-align: right;\">enabled<\/td>\n<\/tr>\n<tr>\n<td>Training sequence length<\/td>\n<td style=\"text-align: right;\">32,768 tokens<\/td>\n<\/tr>\n<tr>\n<td>Training data<\/td>\n<td style=\"text-align: right;\">OpenR1-Math-220k<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Strengths and Use Cases<\/h2>\n<p>The underlying base model (Qwen\/Qwen3-4B) features the ability to switch within a single model between a &#8220;thinking mode&#8221; for complex logical reasoning, mathematics, and coding, and a &#8220;non-thinking mode&#8221; for efficient general-purpose dialogue. It also features high agent capabilities in tool calling, support for over 100 languages and dialects, and native support for a 32,768-token context length (with extended context support also possible using YaRN).<\/p>\n<p>The introduction of SAS checkpoints enables efficient context processing and inference in an SGLang-based sparse attention environment.<\/p>\n<p><!-- lmw:hardware --><\/p>\n<h2>Hardware Requirements<\/h2>\n<p><strong>Estimated requirements (calculated by Local Model Watch)<\/strong> \u2014 4.0B parameters (taken from the base model Qwen\/Qwen3-4B)<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Your VRAM<\/th>\n<th>Quantization<\/th>\n<th>File size<\/th>\n<th>Est. memory needed<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>12GB (RTX 4070 \/ 3060 12GB, etc.)<\/td>\n<td>BF16<\/td>\n<td>7.5GB<\/td>\n<td>9.0GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model&#8217;s authors. Compare with other models in our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a>. What the quantization names mean: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">glossary<\/a>.<\/em><\/p>\n<p><!-- \/lmw:hardware --><\/p>\n<p><!-- lmw:peers --><\/p>\n<h2>Recent Models in the Same Size Class<\/h2>\n<p><em>Models with <\/em><em>up to 4B<\/em><em> parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site&#8217;s estimates; licenses are as stated on the model cards.<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Parameters<\/th>\n<th>Smallest VRAM tier<\/th>\n<th>License<\/th>\n<th>Our article<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>harshatheg\/Qwen-2.5-1B-RLCD<\/td>\n<td>1.5B<\/td>\n<td>8GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/17\/qwen-25-1b-rlcd-mlx-constrained-decoding\/\">Fast Structured Generation on Apple Silicon with MLX and Qwen<\/a> (2026-09-16)<\/td>\n<\/tr>\n<tr>\n<td>openbmb\/MiniCPM5-2B<\/td>\n<td>2.5B<\/td>\n<td>4GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/08\/minicpm5-2b-released\/\">OpenBMB Releases MiniCPM5-2B: A SOTA 2B On-Device Model<\/a> (2026-09-07)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:peers --><\/p>\n<h2>How to Get It<\/h2>\n<p>Use after cloning the repository and setting up the dedicated environment.<\/p>\n<pre><code class=\"language-bash\">git clone https:\/\/github.com\/Tencent-Hunyuan\/Simple-Attention-Sparsification.git\ncd Simple-Attention-Sparsification\n\ngit submodule update --init --recursive\n\ncurl -fsSL https:\/\/pixi.sh\/install.sh | bash\npixi install\n\ncd third_party\/sglang-blocksparse\npixi install\ncd..\/..\n<\/code><\/pre>\n<p>For inference, use the <code>seer_attn<\/code> backend within the <code>sglang-blocksparse<\/code> fork.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/18\/tencent-releases-wevisdoc-document-parsing-models\/\">Tencent Releases WeVisDoc-2B and WeVisDoc-4B Document Parsing Models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification\">https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-4B\">https:\/\/huggingface.co\/Qwen\/Qwen3-4B<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models, enabling efficient context processing and inference.<\/p>\n","protected":false},"author":1,"featured_media":622,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[310],"tags":[136,375,169,1170,1172,117],"class_list":["post-623","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-new-models","tag-llm-en","tag-qwen3-en","tag-sglang-en","tag-sparse-attention-en","tag-tencent-en","tag--en"],"lang":"en","translations":{"en":623,"ja":621},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=623"}],"version-history":[{"count":7,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623\/revisions"}],"predecessor-version":[{"id":1721,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623\/revisions\/1721"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/622"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=623"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=623"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=623"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}