{"id":12720,"date":"2026-11-17T11:01:33","date_gmt":"2026-11-17T02:01:33","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=12720"},"modified":"2026-11-17T11:10:44","modified_gmt":"2026-11-17T02:10:44","slug":"deepseek-open-sources-deepinfer","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/deepseek-open-sources-deepinfer\/","title":{"rendered":"DeepSeek Open-Sources High-Performance Inference Framework DeepInfer"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Published<\/td>\n<td>2026-10-20<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>Apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>DeepSeek has open-sourced &#8220;DeepInfer&#8221;, a high-performance inference framework that powers the company&#8217;s large-scale production services. DeepInfer is implemented in Python and licensed under Apache-2.0. It was originally built on top of vLLM, with inspiration also drawn from SGLang and TensorRT-LLM.<\/p>\n<p>This release provides an offline inference environment equipped with core technologies required for large-scale serving, currently supporting the &#8220;DeepSeek-V4.1-Flash&#8221; model. Featuring uniquely optimized kernels and an advanced parallel processing architecture, the biggest highlight is that it allows developers to directly leverage the internal stack used by the company to operate its large-scale models at maximum efficiency.<\/p>\n<h2>Key Changes<\/h2>\n<h3>Integration of Custom Optimized Kernels and Runtime<\/h3>\n<p>To maximize execution efficiency for model inference and samplers, multiple dedicated kernels have been integrated. Specifically, alongside FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect, proprietary TileLang kernels are incorporated. These optimized kernels accelerate computationally intensive attention operations and MoE (Mixture of Experts) dispatching processes. Developers using supported environments (NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs) can benefit from these out of the box without manually building or tuning kernels.<\/p>\n<h3>Flexible Parallel Processing Features (Attention DP \/ Balanced CP, MoE EP + EPLB)<\/h3>\n<p>To efficiently distribute and execute large-scale MoE models, a variety of parallelization methods are supported. For the attention mechanism, Attention DP (Data Parallelism) and balanced CP (Context Parallelism) are provided, while EP (Expert Parallelism) and load-balancing EPLB (Expert Parallelism Load Balancing) are implemented for MoE layers. This helps maintain high throughput while suppressing load imbalances among experts even when scaling out models in ultra-large cluster environments.<\/p>\n<h3>Decoding Acceleration via DSpark and Confidence Scheduling Verification<\/h3>\n<p>The speculative decoding mechanism &#8220;DSpark&#8221; is included, which accelerates decoding using draft tokens and schedules verification based on confidence prediction. To enable this feature, set the environment variable <code>DEEPINFER_ENABLE_MTP_VARLEN=1<\/code>. Additionally, the maximum draft length can be controlled via the environment variable <code>DEEPINFER_USE_NEXTN_DECODE<\/code>, with the default set to 5. This significantly reduces token generation latency while maintaining inference accuracy.<\/p>\n<h3>Reduction of Prefill Load via Decoder SWA Bounded Replay<\/h3>\n<p>&#8220;Decoder SWA bounded replay&#8221; has been introduced, which reduces computational load during decoder prefill processing by replaying only bounded suffixes. When used with the standard settings of DeepSeek-V4.1, set the environment variables <code>DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact<\/code> and <code>DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140<\/code>. The retained suffix must cover the range of the SWA (Sliding Window Attention) window width plus the maximum backtrack length.<\/p>\n<h3>Asynchronous Scheduling, Batch-Invariant Inference, and Deterministic Sampling<\/h3>\n<p>Asynchronous scheduling, which overlaps batch scheduling processing on the CPU side with execution processing on the device side to make scheduling overhead virtually zero, is enabled by default.<\/p>\n<p>Furthermore, as a feature to facilitate verification and debugging, &#8220;Batch-invariant inference&#8221; is provided to keep model output results identical regardless of different batch configurations. This can be utilized by specifying the environment variable <code>DEEPINFER_BATCH_INVARIANT=1<\/code> in an NVIDIA environment. In addition to this, by specifying <code>DEEPINFER_ENABLE_MTP_VARLEN=0<\/code> and passing a fixed <code>seq_seed<\/code> to <code>SamplingParams<\/code>, deterministic sampling is enabled, yielding completely identical outputs for the same input tokens and sampling parameters at all times.<\/p>\n<h3>Support for Structured Outputs<\/h3>\n<p>Structured output functionality is provided to constrain generated tokens according to regular expressions (regex), JSON schema, or EBNF grammars. When using this, passing <code>so_args={\"schema_type\":..., \"schema\":...}<\/code> to <code>SamplingParams<\/code> during inference allows you to obtain outputs strictly adhering to a specific format.<\/p>\n<h3>External Components Required for Production Serving<\/h3>\n<p>This open-source release focuses on core features centered around offline inference, and the following elements required to build a complete inference service are not included in this release:<\/p>\n<ul>\n<li>KV cache system<\/li>\n<li>VE\/Prefill\/Decode separation mechanism (most required engine-side features are already implemented)<\/li>\n<li>Load balancing between data parallel replicas in both prefill and decode<\/li>\n<\/ul>\n<p>When building a complete inference cluster for commercial environments, these peripheral systems must be prepared or integrated separately.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>The models and hardware environments supported by DeepInfer in this release are as follows:<\/p>\n<h3>Supported Models<\/h3>\n<ul>\n<li><strong>DeepSeek-V4.1-Flash<\/strong><\/li>\n<\/ul>\n<p>The standard checkpoint specified for this release is <code>deepseek-ai\/DeepSeek-V4.1-Flash-1017<\/code>. It is designed to automatically download to the Hugging Face cache upon initial execution, allowing verification to begin even without weight files prepared locally. It is also possible to use weights downloaded in advance to local storage.<\/p>\n<h3>Supported Hardware<\/h3>\n<p>To achieve ultra-high throughput, DeepInfer&#8217;s supported accelerators are optimized specifically for certain generations.<\/p>\n<ul>\n<li><strong>NVIDIA SM100 GPU<\/strong><\/li>\n<li><strong>Huawei Ascend 950 NPU<\/strong><\/li>\n<\/ul>\n<p>There are no descriptions guaranteeing operation on general consumer-facing GPUs or older-generation datacenter GPUs (such as pre-Hopper generations), and cutting-edge SM100 and Ascend 950 are explicitly specified as target hardware. Additionally, some advanced features, such as batch-invariant inference and deterministic sampling, are provided exclusively for NVIDIA environments.<\/p>\n<h2>How to Get It<\/h2>\n<p>DeepInfer is set up by building from source code. After meeting the prerequisites of the operating environment, follow the steps below to build the environment and run inference.<\/p>\n<h3>Prerequisites<\/h3>\n<p>Building and running requires the following operating system and toolchain:<\/p>\n<ul>\n<li>OS: Linux<\/li>\n<li>Compiler: GCC 13.0 or higher<\/li>\n<li>Language environment: Python 3.12<\/li>\n<li>Accelerator drivers and toolkits:<\/li>\n<li>For NVIDIA environments: CUDA 13 &#8211; For Huawei Ascend environments: CANN 9.2<\/li>\n<\/ul>\n<h3>Build Steps from Source Code<\/h3>\n<p>Before starting the build, create and activate a fresh, empty Python 3.12 environment. Then, set the environment variables according to the accelerator you are using and run the build script.<\/p>\n<p>The commands for using an NVIDIA GPU are as follows. These environment variables must remain exported not only during the build but also when running the inference script later.<\/p>\n<pre><code class=\"language-bash\"># For NVIDIA environments\nexport DEEPINFER_ACCELERATOR=nvidia\n.\/build.sh\n<\/code><\/pre>\n<p>When executing on the Huawei Ascend platform, perform the build by specifying <code>ascend<\/code> for the environment variable.<\/p>\n<pre><code class=\"language-bash\"># For Ascend environments\nexport DEEPINFER_ACCELERATOR=ascend\n.\/build.sh\n<\/code><\/pre>\n<h3>Running Offline Inference<\/h3>\n<p>Once the build is complete, you can try offline inference by running the provided sample script.<\/p>\n<pre><code class=\"language-bash\">python examples\/offline_inference.py\n<\/code><\/pre>\n<p>By default, the checkpoint <code>deepseek-ai\/DeepSeek-V4.1-Flash-1017<\/code> is automatically downloaded to the Hugging Face cache upon initial execution. If you have already downloaded the model to your local disk, specify its directory path in the environment variable <code>DEEPINFER_HF_MODEL_DIR<\/code> before running.<\/p>\n<pre><code class=\"language-bash\">export DEEPINFER_HF_MODEL_DIR=\/path\/to\/DeepSeek-V4.1-Flash-1017\npython examples\/offline_inference.py\n<\/code><\/pre>\n<p>For detailed settings and instructions on running sweep validations, please refer to the <code>examples<\/code> directory in the repository, <code>run_sweep.sh<\/code>, and <code>eval\/README.md<\/code>, which contains guides for evaluation benchmarks.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/vllm-v033-0-released\/\">vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/05\/vllm-v0310-released\/\">vLLM v0.31.0 Released: Fast Restart and Hardware Optimization<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/deepseek-v4-1-flash-1017-gguf-unsloth\/\">DeepSeek-V4.1-Flash-1017-GGUF 1M Context GGUF Model: ~560GB Memory<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/deepseek-v4-1-flash-1017-released\/\">DeepSeek-V4.1-Flash-1017 Vision-Language Model: ~570GB Memory<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Engines mentioned in this article<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/deepseek-ai\/DeepInfer\">DeepInfer GitHub Repository<\/a><\/li>\n<li><a href=\"https:\/\/news.ycombinator.com\/item?id=50163277\">Hacker News Discussion<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>DeepSeek has open-sourced DeepInfer, a high-performance inference framework supporting DeepSeek-V4.1-Flash with customized kernels.<\/p>\n","protected":false},"author":1,"featured_media":12719,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[3811,604,774,1547,592,1193],"class_list":["post-12720","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-deepinfer-en","tag-deepseek-en","tag-deepseek-v4-1-flash-en","tag-verified","tag-vllm-en","tag--en"],"lang":"en","translations":{"en":12720,"ja":12718},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12720","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=12720"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12720\/revisions"}],"predecessor-version":[{"id":17879,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/12720\/revisions\/17879"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/12719"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=12720"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=12720"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=12720"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}