{"id":13819,"date":"2026-11-17T11:02:47","date_gmt":"2026-11-17T02:02:47","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=13819"},"modified":"2026-11-17T13:12:01","modified_gmt":"2026-11-17T04:12:01","slug":"qwen3-8-27b-nvfp4-64k-mtp","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/qwen3-8-27b-nvfp4-64k-mtp\/","title":{"rendered":"Qwen3.8-27B-NVFP4-DGX-SPARK-V2 NVFP4 Quantized 27B Model: 24GB+ VRAM"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/huggingface.co\/autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2\">autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-10-28<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Formats<\/td>\n<td>safetensors<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Unverified (not confirmed by a primary source)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>This information has not been officially confirmed. It is reported by autotrust that a repository combining Qwen3.8-27B NVFP4 checkpoints with a 64k MTP (Multi-Token Prediction) recipe has been released.<\/p>\n<p>This repository integrates the complete checkpoints of Qwen3.8-27B NVFP4 (including MTP tensors) with the &#8220;64k-vocabulary MTP recipe&#8221; provided by <code>autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK<\/code>. The 64k MTP optimization provided here is not written directly into the safetensors weights, but functions as a &#8220;runtime patch&#8221; loaded via an included vLLM plugin or an offline Python patch. This patch limits the draft head vocabulary size from 248,320 rows to 64,026 rows, and pads it to 64,064 rows for quantized GEMM kernels, significantly lightweighting the draft step. This mechanism is reported to make it possible to handle a larger number of speculative tokens. This configuration is designed targeting Blackwell GPUs (B200 and DGX Spark \/ GB10).<\/p>\n<h2>Specifications<\/h2>\n<ul>\n<li>Architecture: Qwen3_5ForConditionalGeneration<\/li>\n<li>Quantization: NVFP4<\/li>\n<li>MTP: 64k-vocabulary MTP recipe<\/li>\n<\/ul>\n<h2>Performance<\/h2>\n<p>The model card does not include comparison tables with other models or benchmark figures.<\/p>\n<p>However, according to the publisher&#8217;s report, applying this 64k MTP patch significantly reduces the size of the draft step, making it possible to handle a larger number of speculative tokens. As specific verification results, <code>num_speculative_tokens: 7<\/code> for the B200 environment and <code>8<\/code> for the DGX Spark environment are reported as reasonable starting points.<\/p>\n<p>This patch incorporates advanced quantization processing at runtime. Specifically, it collects draft heads from the target <code>lm_head<\/code> and performs quantization to Marlin NVFP4 W4A16 at runtime. Additionally, the five MTP linear projections are also quantized to Marlin NVFP4 at runtime. Note that this process is designed to have zero impact on the base target model itself and the verification <code>lm_head<\/code>, aiming to improve inference efficiency.<\/p>\n<h2>Strengths and Use Cases<\/h2>\n<p>This model is designed for high-speed inference on GPUs adopting the Blackwell architecture (B200 and DGX Spark \/ GB10). In particular, it features a configuration specialized for accelerating inference through speculative decoding using MTP (Multi-Token Prediction) technology.<\/p>\n<p>Specifically, it has the following features:<\/p>\n<ul>\n<li><strong>Efficient Speculative Decoding<\/strong>: Applying the 64k MTP patch significantly lightens the draft step and increases the number of speculative tokens that can be generated at once. Speculative token settings of about 7 tokens for B200 and 8 tokens for DGX Spark are recommended.<\/li>\n<li><strong>Acceleration via Advanced Quantization<\/strong>: Improves inference efficiency by quantizing draft heads and MTP linear projections to Marlin NVFP4 at runtime.<\/li>\n<li><strong>Support for Multimodal and Dialogue<\/strong>: According to tags, it supports text generation and conversational use cases, with support for English (en) and Chinese (zh).<\/li>\n<\/ul>\n<p><!-- lmw:hardware --><\/p>\n<h2>Hardware Requirements<\/h2>\n<p><strong>Estimated requirements (calculated by Local Model Watch)<\/strong><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Your VRAM<\/th>\n<th>Quantization<\/th>\n<th>File size<\/th>\n<th>Est. memory needed<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>24GB (RTX 4090 \/ 3090, etc.)<\/td>\n<td>NVFP4<\/td>\n<td>18.4GB<\/td>\n<td>22.1GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><strong>Inference engine support<\/strong> (architecture name matched against each project&#8217;s own model registry in its source code, checked 2026-11-17): llama.cpp: registered, vLLM: registered, MLX (mlx-lm): registered.<\/p>\n<p><em>Memory estimates add a 20% runtime overhead (KV cache, etc.) to the actual size of the distributed files. Actual usage varies with context length, batch size and inference engine. These figures are computed by this site from file sizes, not published by the model&#8217;s authors. Compare with other models in our <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-en\/\">VRAM quick reference<\/a>. What the quantization names mean: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">glossary<\/a>.<\/em><\/p>\n<p><!-- \/lmw:hardware --><\/p>\n<p><!-- lmw:runnability --><\/p>\n<h2>Can You Run It Locally?<\/h2>\n<p><strong>Not usable in Ollama, LM Studio and llama.cpp yet \u2014 we have found no GGUF build.<\/strong><\/p>\n<p>The publisher ships safetensors only. However, llama.cpp&#8217;s registry does list this architecture, so <strong>conversion to GGUF is possible<\/strong> and the model will run once someone publishes a converted build. Today it can be run with transformers or vLLM, using the memory figures in the table above.<\/p>\n<p><strong>License \u2014 <code>apache-2.0<\/code> (Commercial use allowed):<\/strong> Permits commercial use, modification and redistribution. Redistribution requires including the license and stating changes; includes a patent grant.<\/p>\n<p><em>Compiled by this site&#8217;s code from the published formats, converted builds we have found, and each engine&#8217;s own model registry. &#8220;Not found&#8221; means we have not seen such a build, not that none exists. License summaries are not legal advice \u2014 check the publisher&#8217;s original terms before relying on them.<\/em><\/p>\n<p><!-- \/lmw:runnability --><\/p>\n<h2>How to Get It<\/h2>\n<p>This model can be obtained from Hugging Face using the following steps. The distribution format is <code>safetensors<\/code>.<\/p>\n<pre><code class=\"language-bash\">export MODEL_DIR=&quot;$HOME\/models\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2&quot;\n\nHF_HUB_ENABLE_HF_TRANSFER=1 \\\nhf download autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2 \\\n  --local-dir &quot;$MODEL_DIR&quot;\n\ncd &quot;$MODEL_DIR&quot;\n<\/code><\/pre>\n<p>Operation requires installing a dedicated vLLM plugin. Build the environment with the following steps:<\/p>\n<pre><code class=\"language-bash\">python3 -m venv.venv\nsource.venv\/bin\/activate\npython -m pip install -U pip\n\npip install &quot;vllm==0.31.0&quot;\npip install -e.\/mtp64k\/plugin\n<\/code><\/pre>\n<p>Additionally, setting the following environment variables is recommended during execution:<\/p>\n<pre><code class=\"language-bash\">export SPARK_DRAFT_HOME=&quot;$PWD\/mtp64k\/scripts&quot;\nexport SPARK_DRAFT_VOCAB=&quot;$PWD\/mtp64k\/artifacts\/draft_vocab_64k.pt&quot;\nexport SPARK_DRAFT_HEAD_Q=marlin\nexport SPARK_DRAFT_MTP_Q=marlin\n<\/code><\/pre>\n<p>The supported inference engine is vLLM (version 0.31.0). SGLang and TensorRT-LLM are not supported.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/qwen38-flash-next-e384-fp4-local\/\">Qwen3.8-Flash-Next-E384-FP4-Local Vision-Language Model: ~96GB Memory<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/autotrust-tts-v5-overview\/\">TTS-V5 Chinese TTS Model: 4GB+ VRAM, File List<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/autotrust-tts-v4-chinese-call-center-tts\/\">TTS-V4 Chinese Text-to-Speech Model: 8GB+ VRAM, File List<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/autotrust-tts-v3-released\/\">TTS-V3 Phone-Dialogue Specialized TTS Model: 8GB+ VRAM, File List<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Find models by VRAM<\/strong> (This model runs from the 24GB tier) \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/vram-guide-24gb-en\/\">Other models that run on a 24GB GPU<\/a><\/li>\n<li><strong>Engines that run this model<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a><\/li>\n<li><strong>What NVFP4 mean and where to get this model<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/format-fp4-en\/\">NVFP4 \/ MXFP4 format guide and models<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/glossary-quantization-en\/\">Quantization and model-format glossary<\/a><\/li>\n<li><strong>Other models for the same task<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/models-by-task-en\/#task-text\">Other text generation models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/huggingface.co\/autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2\"><a href=\"https:\/\/huggingface.co\/autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2\">https:\/\/huggingface.co\/autotrust\/Qwen3.8-27B-NVFP4-DGX-SPARK-V2<\/a><\/a><\/li>\n<\/ul>\n<blockquote>\n<p><strong>This article contains unverified information.<\/strong> We will append an update note once it is confirmed by a primary source.<\/p>\n<\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Explore the unofficial release of Qwen3.8-27B NVFP4 combined with a 64k MTP recipe for accelerated speculative decoding on Blackwell GPUs.<\/p>\n","protected":false},"author":1,"featured_media":13818,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[310],"tags":[4024,813,1082,4252,209,1565,592,1555],"class_list":["post-13819","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-new-models","tag-autotrust-en","tag-nvfp4-en","tag-qwen3-8-27b-en","tag-qwen3-8-dgx-spark-v2-en","tag-safetensors-en","tag-unverified","tag-vllm-en","tag--en"],"lang":"en","translations":{"en":13819,"ja":13817},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13819","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=13819"}],"version-history":[{"count":3,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13819\/revisions"}],"predecessor-version":[{"id":18182,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/13819\/revisions\/18182"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/13818"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=13819"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=13819"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=13819"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}