{"id":623,"date":"2026-09-14T20:34:12","date_gmt":"2026-09-14T11:34:12","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/14\/tencent-simple-attention-sparsification-qwen3\/"},"modified":"2026-10-04T05:13:46","modified_gmt":"2026-10-03T20:13:46","slug":"tencent-simple-attention-sparsification-qwen3","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/14\/tencent-simple-attention-sparsification-qwen3\/","title":{"rendered":"Simple-Attention-Sparsification Text Generation Model: 12GB+ VRAM"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification\">tencent\/Simple-Attention-Sparsification<\/a><\/td>\n<\/tr>\n<tr>\n<td>Publisher guide<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-tencent-en\/\">Tencent: models and licenses<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-09-14<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td>PyTorch (pth)<\/td>\n<\/tr>\n<tr>\n<td>Paper<\/td>\n<td><a href=\"https:\/\/arxiv.org\/abs\/2609.13141\">arXiv:2609.13141<\/a><\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<p><!-- lmw:lab-summary --><\/p>\n<p><strong>What we checked ourselves<\/strong><\/p>\n<ul>\n<li>The same Japanese text takes about as many tokens as with the Qwen3 tokenizer.<\/li>\n<\/ul>\n<p>Details and conditions are in \u201cOur Own Measurements\u201d below.<\/p>\n<p><!-- \/lmw:lab-summary --><\/p>\n<h2>Overview<\/h2>\n<p>Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models. SAS is a sparse attention method that uses continuous gates to select and rank KV blocks, allowing end-to-end optimization via language modeling loss. The released checkpoints contain only the router portion, and running them requires the corresponding Qwen3 base model and a dedicated backend.<\/p>\n<h2>Specifications<\/h2>\n<ul>\n<li>Base model: Qwen\/Qwen3-4B<\/li>\n<li>Tags: qwen3, sparse-attention, long-context, sglang, sas, text-generation<\/li>\n<li>Checkpoint type: Router-only checkpoint (the Qwen3 backbone is frozen and not bundled)<\/li>\n<\/ul>\n<h2>Specifications and Configuration Table<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Directory<\/th>\n<th>Base model<\/th>\n<th style=\"text-align: right;\">Gate parameters<\/th>\n<th style=\"text-align: right;\">Gate checkpoint size<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>Qwen3-4B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-4B\">Qwen3-4B<\/a><\/td>\n<td style=\"text-align: right;\">33.0M<\/td>\n<td style=\"text-align: right;\">64 MiB<\/td>\n<\/tr>\n<tr>\n<td><code>Qwen3-8B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-8B\">Qwen3-8B<\/a><\/td>\n<td style=\"text-align: right;\">33.0M<\/td>\n<td style=\"text-align: right;\">64 MiB<\/td>\n<\/tr>\n<tr>\n<td><code>Qwen3-14B-AttnGates<\/code><\/td>\n<td><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-14B\">Qwen3-14B<\/a><\/td>\n<td style=\"text-align: right;\">42.0M<\/td>\n<td style=\"text-align: right;\">81 MiB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Setting<\/th>\n<th style=\"text-align: right;\">Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>KV block size<\/td>\n<td style=\"text-align: right;\">64 tokens<\/td>\n<\/tr>\n<tr>\n<td>Training Top-K<\/td>\n<td style=\"text-align: right;\">31 historical blocks<\/td>\n<\/tr>\n<tr>\n<td>Gate hidden size<\/td>\n<td style=\"text-align: right;\">128<\/td>\n<\/tr>\n<tr>\n<td>Query projection<\/td>\n<td style=\"text-align: right;\"><code>Qproj<\/code><\/td>\n<\/tr>\n<tr>\n<td>Key block pooling<\/td>\n<td style=\"text-align: right;\">max + min + average<\/td>\n<\/tr>\n<tr>\n<td>Q\/K normalization<\/td>\n<td style=\"text-align: right;\">enabled<\/td>\n<\/tr>\n<tr>\n<td>Gate RoPE<\/td>\n<td style=\"text-align: right;\">enabled<\/td>\n<\/tr>\n<tr>\n<td>Training sequence length<\/td>\n<td style=\"text-align: right;\">32,768 tokens<\/td>\n<\/tr>\n<tr>\n<td>Training data<\/td>\n<td style=\"text-align: right;\">OpenR1-Math-220k<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2>Strengths and Use Cases<\/h2>\n<p>The underlying base model (Qwen\/Qwen3-4B) features the ability to switch within a single model between a &#8220;thinking mode&#8221; for complex logical reasoning, mathematics, and coding, and a &#8220;non-thinking mode&#8221; for efficient general-purpose dialogue. It also features high agent capabilities in tool calling, support for over 100 languages and dialects, and native support for a 32,768-token context length (with extended context support also possible using YaRN).<\/p>\n<p>The introduction of SAS checkpoints enables efficient context processing and inference in an SGLang-based sparse attention environment.<\/p>\n<p><!-- lmw:hardware --><\/p>\n<h2>Hardware Requirements<\/h2>\n<p><strong>Estimated requirements (calculated by Local Model Watch)<\/strong> \u2014 4.0B parameters (taken from the base model Qwen\/Qwen3-4B)<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Your VRAM<\/th>\n<th>Quantization<\/th>\n<th>File size<\/th>\n<th>Est. memory needed<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>12GB (RTX 4070 \/ 3060 12GB, etc.)<\/td>\n<td>BF16<\/td>\n<td>7.5GB<\/td>\n<td>9.0GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model&#8217;s authors. Compare with other models in our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a>. What the quantization names mean: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">glossary<\/a>.<\/em><\/p>\n<p><!-- \/lmw:hardware --><\/p>\n<p><!-- lmw:runnability --><\/p>\n<h2>Can You Run It Locally?<\/h2>\n<p><strong>Not usable in Ollama, LM Studio and llama.cpp yet \u2014 we have found no GGUF build.<\/strong><\/p>\n<p>The publisher ships PyTorch (pth) only.<\/p>\n<p><em>Compiled by this site&#8217;s code from the published formats, converted builds we have found, and each engine&#8217;s own model registry. &#8220;Not found&#8221; means we have not seen such a build, not that none exists.<\/em><\/p>\n<p><!-- \/lmw:runnability --><\/p>\n<p><!-- lmw:lab --><\/p>\n<h2>Our Own Measurements<\/h2>\n<p>Values we checked ourselves on our server (no GPU) by actually reading this model&#8217;s files, without running the model \u2014 not figures copied from the model card. How we measure, and the results for every model: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/observations-en\/\">Our Measurements<\/a>.<\/p>\n<h3>Measurements We Did Not Take<\/h3>\n<p>This model does not state a license, so we cannot confirm permission to use it. Because this site carries advertising, we did not run the model (no CPU run, answers, quantization comparison, conversion or generation). Only values we checked without running the model, such as token counts and the GGUF header, are shown.<\/p>\n<h3>Japanese Token Efficiency<\/h3>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Tokenizer<\/th>\n<th>Tokens per 1,000 Japanese characters<\/th>\n<th>Ratio to the same text in English<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><strong>This model<\/strong><\/td>\n<td><strong>688<\/strong><\/td>\n<td><strong>1.26\u00d7<\/strong><\/td>\n<\/tr>\n<tr>\n<td>Qwen3<\/td>\n<td>688<\/td>\n<td>1.26\u00d7<\/td>\n<\/tr>\n<tr>\n<td>Llama 3.2<\/td>\n<td>744<\/td>\n<td>1.36\u00d7<\/td>\n<\/tr>\n<tr>\n<td>Gemma 3<\/td>\n<td>564<\/td>\n<td>1.03\u00d7<\/td>\n<\/tr>\n<tr>\n<td>gpt-oss<\/td>\n<td>795<\/td>\n<td>1.45\u00d7<\/td>\n<\/tr>\n<tr>\n<td>LLM-jp-3<\/td>\n<td>497<\/td>\n<td>0.85\u00d7<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>The same Japanese text takes about as many tokens as with the Qwen3 tokenizer.<\/p>\n<p>Counted with the <code>tokenizer.json<\/code> of <a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-4B\">Qwen\/Qwen3-4B<\/a> on a fixed text we wrote ourselves (876 Japanese characters across news, conversation, technical docs, a formal email, travel writing and a recipe) and its English translation. Fewer tokens mean more Japanese fits in the context window.<\/p>\n<p><!-- \/lmw:lab --><\/p>\n<p><!-- lmw:peers --><\/p>\n<h2>Recent Models in the Same Size Class<\/h2>\n<p><em>Models with <\/em><em>up to 4B<\/em><em> parameters that Local Model Watch covered recently, listed by code from our article log for comparison. VRAM tiers are this site&#8217;s estimates; licenses are as stated on the model cards.<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<p class=\"lmw-table-hint\" style=\"margin:0 0 4px;font-size:0.85em;opacity:0.7;\">\u2192 Scroll horizontally to see all columns<\/p>\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Model<\/th>\n<th>Parameters<\/th>\n<th>Smallest VRAM tier<\/th>\n<th>License<\/th>\n<th>Our article<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>LiquidAI\/LFM2.5-350M-Diffusion-Exp<\/td>\n<td>425M<\/td>\n<td>4GB<\/td>\n<td>lfm1.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/04\/liquidai-lfm2-5-350m-diffusion-exp\/\">LFM2.5-350M-Diffusion-Exp Text Generation Model: 4GB+ VRAM<\/a> (2026-10-03)<\/td>\n<\/tr>\n<tr>\n<td>togethercomputer\/Tev1-0.8B-experimental<\/td>\n<td>873M<\/td>\n<td>4GB<\/td>\n<td>\u2014<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/26\/tev1-08b-experimental-2\/\">Tev1-0.8B-experimental Text Generation Model: 4GB+ VRAM<\/a> (2026-09-25)<\/td>\n<\/tr>\n<tr>\n<td>pfnet\/plamo-3-610m-fin-instruct<\/td>\n<td>890M<\/td>\n<td>4GB<\/td>\n<td>plamo-community-license<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/24\/plamo-3-610m-fin-instruct-2\/\">plamo-3-610m-fin-instruct Text Generation Model: 4GB+ VRAM<\/a> (2026-09-24)<\/td>\n<\/tr>\n<tr>\n<td>harshatheg\/Qwen-2.5-1B-RLCD<\/td>\n<td>1.5B<\/td>\n<td>4GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/17\/qwen-25-1b-rlcd-mlx-constrained-decoding\/\">Qwen-2.5-1B-RLCD Text Generation Model: 4GB+ VRAM<\/a> (2026-09-16)<\/td>\n<\/tr>\n<tr>\n<td>openbmb\/MiniCPM5-2B<\/td>\n<td>2.5B<\/td>\n<td>4GB<\/td>\n<td>apache-2.0<\/td>\n<td><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/08\/minicpm5-2b-released\/\">MiniCPM5-2B Text Generation Model: Our Test Answers, 4GB+ VRAM<\/a> (2026-09-07)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:peers --><\/p>\n<h2>How to Get It<\/h2>\n<p>Use after cloning the repository and setting up the dedicated environment.<\/p>\n<pre><code class=\"language-bash\">git clone https:\/\/github.com\/Tencent-Hunyuan\/Simple-Attention-Sparsification.git\ncd Simple-Attention-Sparsification\n\ngit submodule update --init --recursive\n\ncurl -fsSL https:\/\/pixi.sh\/install.sh | bash\npixi install\n\ncd third_party\/sglang-blocksparse\npixi install\ncd..\/..\n<\/code><\/pre>\n<p>For inference, use the <code>seer_attn<\/code> backend within the <code>sglang-blocksparse<\/code> fork.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/18\/tencent-releases-wevisdoc-document-parsing-models\/\">Tencent Releases WeVisDoc-2B and WeVisDoc-4B Document Parsing Models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Find models by VRAM<\/strong> (This model runs from the 12GB tier) \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-12gb-en\/\">Other models that run on a 12GB GPU<\/a><\/li>\n<li><strong>Learn about the publisher<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/publisher-tencent-en\/\">Tencent: models, licenses and articles<\/a><\/li>\n<li><strong>Other models for the same task<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/models-by-task-en\/#task-text\">Other text generation models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification\">https:\/\/huggingface.co\/tencent\/Simple-Attention-Sparsification<\/a><\/li>\n<li><a href=\"https:\/\/huggingface.co\/Qwen\/Qwen3-4B\">https:\/\/huggingface.co\/Qwen\/Qwen3-4B<\/a><\/li>\n<\/ul>\n<p><!-- lmw:updates --><\/p>\n<h2>Update History<\/h2>\n<ul>\n<li>2026-09-24: Rebuilt the article: Added a &#8220;can you run it locally?&#8221; verdict from the distribution formats, converted builds, engine registries, license terms and measured compression<\/li>\n<li>2026-09-25: Changed the title to show what the article covers (VRAM requirements, file list, etc.).<\/li>\n<li>2026-09-28: Added our own measurements: Japanese token efficiency.<\/li>\n<\/ul>\n<p><!-- \/lmw:updates --><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tencent has released Simple Attention Sparsification (SAS) checkpoints for Qwen3 models, enabling efficient context processing and inference.<\/p>\n","protected":false},"author":1,"featured_media":3992,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[310],"tags":[136,375,169,1170,1172,117],"class_list":["post-623","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-new-models","tag-llm-en","tag-qwen3-en","tag-sglang-en","tag-sparse-attention-en","tag-tencent-en","tag--en"],"lang":"en","translations":{"en":623,"ja":621},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=623"}],"version-history":[{"count":27,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623\/revisions"}],"predecessor-version":[{"id":9721,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/623\/revisions\/9721"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/3992"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=623"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=623"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=623"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}