Qwen3-30B-A3B-MoE-SAE: 80GB+ VRAM

Qwen3-30B-A3B-MoE-SAE: 80GB+ VRAM

At a Glance

Item Value
Repository ByteDance-Seed/Qwen3-30B-A3B-MoE-SAE
Publisher guide ByteDance Seed: models and licenses
Published 2026-11-05
License apache-2.0
Source type Unverified (not confirmed by a primary source)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

It is reported that ByteDance-Seed has released a Sparse AutoEncoder (SAE) repository for the Qwen3-30B-A3B model, titled Qwen3-30B-A3B-MoE-SAE. Please note that this news is currently unverified and has not been officially confirmed.

This SAE is reportedly provided as a research output from the paper “Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization." It was created to analyze features output by the MoE (Mixture-of-Experts) blocks (layers 4, 30, and 46) of Qwen3-30B-A3B, with the goal of extracting and interpreting human-understandable, fine-grained expertise. It is intended for use in analyzing internal model representations and interpretability research.

Specifications

The main specifications for the Qwen3-30B-A3B-MoE-SAE and the base model are as follows:

  • Base Model: Qwen/Qwen3-30B-A3B
  • Parameter Count (Base Model): 30.5B total parameters, 3.3B active parameters (29.9B non-embedding parameters)
  • Architecture: Qwen3MoeForCausalLM (48 layers, 128 experts, 8 active experts)
  • Context Length: 32,768 tokens native (131,072 tokens with YaRN)
  • Latents: 2^20 (1,048,576)
  • TopK Setting: k = 100
  • Target Layers: MoE block outputs for layers 4, 30, and 46 (model.model.layers[L].mlp)
  • Training Data: Approximately 3.2B tokens per SAE (Olmo 3 training corpus: 5/8 Dolma 3 pre-training data, 1/4 Dolci-Think-SFT, 1/8 Dolci-Instruct-SFT)

Performance

The model card provides the Fraction of Variance Unexplained (FVU) for the reconstruction at each target layer.

Layer FVU
4 12.6%
30 25.1%
46 7.6%

Additionally, the specifications for the tensor structures and shapes included in the SAE for each layer are as follows:

Tensor Shape
W_enc [2048, 1048576]
b_enc [1048576]
W_dec [1048576, 2048], rows have unit norm
b_dec [2048]

Based on the FVU table, the SAE for layer 46, which is closer to the output, shows an FVU drop to 7.6%, indicating that the MoE block output can be reconstructed with very high precision. Conversely, the middle layer (layer 30) shows 25.1%, suggesting that reconstruction errors remain relatively higher compared to the 12.6% seen in layer 4.

These results suggest that internal representations in the middle layers of the network are more complex and ambiguous, which may lead to differences in how easily information can be captured, even with the same number of latents (1,048,576). Researchers should note that reconstruction accuracy characteristics vary by layer when conducting interpretability analyses.

Strengths and Use Cases

Qwen3-30B-A3B-MoE-SAE is described as a toolkit for decomposing and analyzing the internal behavior of the MoE blocks in the Qwen3-30B-A3B base model, allowing for visualization and research into human-understandable features. Primary use cases and analysis targets include:

  • Expert Expertise Analysis: By capturing the output of MoE blocks at layers 4, 30, and 46, one can track which concepts or language patterns each expert specializes in at a fine-grained level.
  • Interpretability Research: It is reported that the 1,048,576 latents can be used to represent the activation states of the neural network, facilitating research into decomposing and evaluating the model’s internal representation space.

Regarding the base model “Qwen3-30B-A3B" itself, the model card highlights the following strengths and use cases:

  • Switching Between Thinking and Non-Thinking Modes: The model can dynamically switch between a “thinking mode" for complex logical reasoning, mathematics, and programming, and a “non-thinking mode" for general conversation focused on response efficiency.
  • Advanced Agent and Tool Use: It supports external tool calls and complex agent tasks in coordination with Qwen-Agent and MCP (Model Context Protocol).
  • Long-Context and Multilingual Support: It supports long contexts of up to 131,072 tokens using YaRN technology and is capable of instruction following and translation in over 100 languages and dialects.

Using this SAE, it is said that one can perform more detailed verification and analysis of the dynamic behavior of experts within the model while it performs reasoning or tool usage.

Hardware Requirements

Estimated requirements (calculated by Local Model Watch) — 30.5B parameters (taken from the base model Qwen/Qwen3-30B-A3B)

Your VRAM Quantization File size Est. memory needed
80GB class (A100 / H100) BF16 56.9GB 68.2GB

Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model’s authors. Compare with other models in our VRAM quick reference. What the quantization names mean: glossary.

Can You Run It Locally?

License — apache-2.0 (Commercial use allowed): Permits commercial use, modification and redistribution. Redistribution requires including the license and stating changes; includes a patent grant.

Compiled by this site’s code from the published formats and the license field. License summaries are not legal advice — check the publisher’s original terms before relying on them.

How to Get It

It is reported that this SAE is released as open-source under the Apache 2.0 license on Hugging Face. There is no gated access, and it can be downloaded without agreeing to additional terms.

The distribution format includes safetensors and json files, with sae.safetensors and config.json located within each layer’s folder (layer_4, layer_30, layer_46).

Guidance is provided for using Python to retrieve the weights via huggingface_hub and safetensors, and then using transformers to register a forward hook on the mlp module of the loaded Qwen3-30B-A3B to extract and decode the MoE output. A specific code example is provided below:

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer

LAYER, K = 30, 100
DEVICE = "cuda"

sae = load_file(
    hf_hub_download("ByteDance-Seed/Qwen3-30B-A3B-MoE-SAE", f"layer_{LAYER}/sae.safetensors"),
    device = DEVICE,
)

def encode(y):
    pre_acts = y.float() @ sae["W_enc"] + sae["b_enc"]
    return torch.topk(torch.relu(pre_acts), K, dim = -1)

def decode(acts, ids):
    return torch.einsum("...k,...kd->...d", acts, sae["W_dec"][ids]) + sae["b_dec"]

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-30B-A3B", torch_dtype = torch.bfloat16, device_map = "auto")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-30B-A3B")

captured = {}
def hook(module, inputs, output):
    captured["y"] = output[0] if isinstance(output, tuple) else output

handle = model.model.layers[LAYER].mlp.register_forward_hook(hook)
inputs = tokenizer("def binary_search(arr, x):", return_tensors = "pt").to(model.device)
with torch.no_grad():
    model(**inputs)
handle.remove()

y = captured["y"][0].to(DEVICE)
acts, ids = encode(y)
y_hat = decode(acts, ids)

Note that because dense pre-activation processing is performed, it is recommended to split long sequences when encoding.

Related Articles

What to Read Next

Sources

This article contains unverified information. We will append an update note once it is confirmed by a primary source.