DeepSeek Open-Sources High-Performance Inference Framework DeepInfer

- 1. At a Glance
- 2. Overview
- 3. Key Changes
- 3.1. Integration of Custom Optimized Kernels and Runtime
- 3.2. Flexible Parallel Processing Features (Attention DP / Balanced CP, MoE EP + EPLB)
- 3.3. Decoding Acceleration via DSpark and Confidence Scheduling Verification
- 3.4. Reduction of Prefill Load via Decoder SWA Bounded Replay
- 3.5. Asynchronous Scheduling, Batch-Invariant Inference, and Deterministic Sampling
- 3.6. Support for Structured Outputs
- 3.7. External Components Required for Production Serving
- 4. Supported Models and Hardware
- 5. How to Get It
- 6. Related Articles
- 7. What to Read Next
- 8. Sources
At a Glance
| Item | Value |
|---|---|
| Published | 2026-10-20 |
| License | Apache-2.0 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
DeepSeek has open-sourced “DeepInfer", a high-performance inference framework that powers the company’s large-scale production services. DeepInfer is implemented in Python and licensed under Apache-2.0. It was originally built on top of vLLM, with inspiration also drawn from SGLang and TensorRT-LLM.
This release provides an offline inference environment equipped with core technologies required for large-scale serving, currently supporting the “DeepSeek-V4.1-Flash" model. Featuring uniquely optimized kernels and an advanced parallel processing architecture, the biggest highlight is that it allows developers to directly leverage the internal stack used by the company to operate its large-scale models at maximum efficiency.
Key Changes
Integration of Custom Optimized Kernels and Runtime
To maximize execution efficiency for model inference and samplers, multiple dedicated kernels have been integrated. Specifically, alongside FlashMLA, DeepGEMM, DeepEP, TileKernels, and DeepSelect, proprietary TileLang kernels are incorporated. These optimized kernels accelerate computationally intensive attention operations and MoE (Mixture of Experts) dispatching processes. Developers using supported environments (NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs) can benefit from these out of the box without manually building or tuning kernels.
Flexible Parallel Processing Features (Attention DP / Balanced CP, MoE EP + EPLB)
To efficiently distribute and execute large-scale MoE models, a variety of parallelization methods are supported. For the attention mechanism, Attention DP (Data Parallelism) and balanced CP (Context Parallelism) are provided, while EP (Expert Parallelism) and load-balancing EPLB (Expert Parallelism Load Balancing) are implemented for MoE layers. This helps maintain high throughput while suppressing load imbalances among experts even when scaling out models in ultra-large cluster environments.
Decoding Acceleration via DSpark and Confidence Scheduling Verification
The speculative decoding mechanism “DSpark" is included, which accelerates decoding using draft tokens and schedules verification based on confidence prediction. To enable this feature, set the environment variable DEEPINFER_ENABLE_MTP_VARLEN=1. Additionally, the maximum draft length can be controlled via the environment variable DEEPINFER_USE_NEXTN_DECODE, with the default set to 5. This significantly reduces token generation latency while maintaining inference accuracy.
Reduction of Prefill Load via Decoder SWA Bounded Replay
“Decoder SWA bounded replay" has been introduced, which reduces computational load during decoder prefill processing by replaying only bounded suffixes. When used with the standard settings of DeepSeek-V4.1, set the environment variables DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. The retained suffix must cover the range of the SWA (Sliding Window Attention) window width plus the maximum backtrack length.
Asynchronous Scheduling, Batch-Invariant Inference, and Deterministic Sampling
Asynchronous scheduling, which overlaps batch scheduling processing on the CPU side with execution processing on the device side to make scheduling overhead virtually zero, is enabled by default.
Furthermore, as a feature to facilitate verification and debugging, “Batch-invariant inference" is provided to keep model output results identical regardless of different batch configurations. This can be utilized by specifying the environment variable DEEPINFER_BATCH_INVARIANT=1 in an NVIDIA environment. In addition to this, by specifying DEEPINFER_ENABLE_MTP_VARLEN=0 and passing a fixed seq_seed to SamplingParams, deterministic sampling is enabled, yielding completely identical outputs for the same input tokens and sampling parameters at all times.
Support for Structured Outputs
Structured output functionality is provided to constrain generated tokens according to regular expressions (regex), JSON schema, or EBNF grammars. When using this, passing so_args={"schema_type":..., "schema":...} to SamplingParams during inference allows you to obtain outputs strictly adhering to a specific format.
External Components Required for Production Serving
This open-source release focuses on core features centered around offline inference, and the following elements required to build a complete inference service are not included in this release:
- KV cache system
- VE/Prefill/Decode separation mechanism (most required engine-side features are already implemented)
- Load balancing between data parallel replicas in both prefill and decode
When building a complete inference cluster for commercial environments, these peripheral systems must be prepared or integrated separately.
Supported Models and Hardware
The models and hardware environments supported by DeepInfer in this release are as follows:
Supported Models
- DeepSeek-V4.1-Flash
The standard checkpoint specified for this release is deepseek-ai/DeepSeek-V4.1-Flash-1017. It is designed to automatically download to the Hugging Face cache upon initial execution, allowing verification to begin even without weight files prepared locally. It is also possible to use weights downloaded in advance to local storage.
Supported Hardware
To achieve ultra-high throughput, DeepInfer’s supported accelerators are optimized specifically for certain generations.
- NVIDIA SM100 GPU
- Huawei Ascend 950 NPU
There are no descriptions guaranteeing operation on general consumer-facing GPUs or older-generation datacenter GPUs (such as pre-Hopper generations), and cutting-edge SM100 and Ascend 950 are explicitly specified as target hardware. Additionally, some advanced features, such as batch-invariant inference and deterministic sampling, are provided exclusively for NVIDIA environments.
How to Get It
DeepInfer is set up by building from source code. After meeting the prerequisites of the operating environment, follow the steps below to build the environment and run inference.
Prerequisites
Building and running requires the following operating system and toolchain:
- OS: Linux
- Compiler: GCC 13.0 or higher
- Language environment: Python 3.12
- Accelerator drivers and toolkits:
- For NVIDIA environments: CUDA 13 – For Huawei Ascend environments: CANN 9.2
Build Steps from Source Code
Before starting the build, create and activate a fresh, empty Python 3.12 environment. Then, set the environment variables according to the accelerator you are using and run the build script.
The commands for using an NVIDIA GPU are as follows. These environment variables must remain exported not only during the build but also when running the inference script later.
# For NVIDIA environments
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
When executing on the Huawei Ascend platform, perform the build by specifying ascend for the environment variable.
# For Ascend environments
export DEEPINFER_ACCELERATOR=ascend
./build.sh
Running Offline Inference
Once the build is complete, you can try offline inference by running the provided sample script.
python examples/offline_inference.py
By default, the checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017 is automatically downloaded to the Hugging Face cache upon initial execution. If you have already downloaded the model to your local disk, specify its directory path in the environment variable DEEPINFER_HF_MODEL_DIR before running.
export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017
python examples/offline_inference.py
For detailed settings and instructions on running sweep validations, please refer to the examples directory in the repository, run_sweep.sh, and eval/README.md, which contains guides for evaluation benchmarks.
Related Articles
- vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades
- vLLM v0.31.0 Released: Fast Restart and Hardware Optimization
- DeepSeek-V4.1-Flash-1017-GGUF 1M Context GGUF Model: ~560GB Memory
- DeepSeek-V4.1-Flash-1017 Vision-Language Model: ~570GB Memory
What to Read Next
- Engines mentioned in this article → vLLM

