vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades

vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades

At a Glance

Item Value
Repository vllm-project/vllm
Version v0.33.0rc3
Published 2026-11-02
License Apache-2.0
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

vLLM, a high-speed and high-efficiency inference and serving engine for LLMs, has released its latest version, “v0.33.0". vLLM is an open-source (Apache-2.0 license) inference framework developed in Python.

This version updates to PyTorch 2.14.0 and moves exclusively to CUDA 13.0, alongside enhanced speculative decoding features that significantly speed up inference, the addition of new methods, and support for numerous new models. Since official build distributions have dropped CUDA 12.9 binaries and older speculative methods such as Medusa have been deprecated, existing environment operators should check their dependency environments before updating.

Breaking Changes and Deprecations

This update deprecates certain execution environments and removes older speculative decoding methods. Environments using the affected configurations will encounter errors upon updating, so please perform migration work.

Item Old Specification New Specification / Migration Target
Official CUDA Build CUDA 12.9 wheel / images provided CUDA 12.9 support dropped, changed to CUDA 13.0 exclusive
Speculative Decoding Method medusa, mlp_speculator Removed (migrate to EAGLE, MTP, DFlash, DSpark, etc.)
Ray Execution Engine Legacy Ray compiled-graph executor Removed (consolidated to RayExecutorV2)
Mamba Cache Setting --mamba-cache-mode Deprecated (does nothing if specified)
Behavior When Generation Paused Undefined Requests sent during generation pause return HTTP 503
Structured Output Backend LM Format Enforcer Deprecated (opt-in in v0.34.0, scheduled for removal in v0.35.0)
Intermediate Representation Priority --ir-op-priority Deprecated (scheduled for removal in v0.34.0)
NIXL Connector Older version support NIXL 1.5 or later required

These changes affect environments deployed by directly specifying CUDA 12.9 containers or wheels, configurations specifying medusa or mlp_speculator for speculative decoding, or environments specifying older execution backends in a Ray setup.

Key Changes

Overhaul of Speculative Decoding and Introduction of New Methods “XPress" and “FLy"

For block-diffusion draft models, “XPress" has been added, which performs causal refinement processing. To solve the issue where block-diffusion draft models like DFlash struggle to increase acceptance lengths due to predicting independently without referencing the immediate preceding tokens, it corrects blocks via Jacobi iteration using a small bottleneck layer and a causal mixer. Fixed-shape iterative processing allows it to be captured as a CUDA Graph, running with low overhead.

In addition, an opt-in verification policy “FLy" that allows for loss tolerance has been implemented. While standard verification aborts at the first mismatching token, FLy allows a single mismatch if the target model’s entropy is high, continuing acceptance if subsequent tokens pass verification. Benchmarks confirm decode throughput improvements of 1.055x to 1.310x on AMD MI355X and 1.041x to 1.273x on NVIDIA B300. Because the output distribution is not completely preserved, using this after weighing the trade-off between quality and speed is recommended.

Furthermore, draft slot reservations on the scheduler have been lifted in DFlash and DSpark. In tests running DeepSeek-V4.1-Flash and DSpark on GB300, output throughput significantly improved from 1373 tok/s to 3390 tok/s alongside the increase in the maximum concurrent sequence limit. Alongside this, draft_lm_head_quantization (supporting NVFP4 and FP8) to quantize the shared lm_head for drafts, and a lookahead block hashing feature that retains tail blocks upon EAGLE or MTP prefix cache hits have been introduced.

Acceleration of Decode Performance in NVIDIA and AMD Environments

Numerous hardware-specific optimizations have been incorporated. In NVIDIA environments, low-latency decoding for Kimi-K3 (1.5x to 2x with TP8/TP16) and DeepSeek-V4.1-Flash (decode steps 18.3% faster at batch size 1) was added for GB300, as well as a path to execute DeepSeek-V4/V4.1 MXFP4 experts as block FP8 via Humming on the Hopper architecture (9% to 21% decode performance improvement on 4x H200).

In AMD ROCm environments, the default distribution migrated to ROCm 10.0 (traditional ROCm 7.2 is also provided). --monokernel decoding for DeepSeek-V4.1-Flash and MiniMax-M3 on MI300X/MI325X/MI355X, ROCM_OCTAVE (3-4bit KV cache) capable of handling 4.6x more KV tokens compared to FP16 on RX 7900 XTX, and optimizations for gfx1250 have been enabled.

API Feature Expansion and Stability Enhancements

On the API and serving side, the OpenAI-compatible /v1/decisions endpoint was added and is enabled by default for supported generative models. Additionally, token embedding outputs and late chunking support in /v1/embeddings, and support for output_mode="derender" in /inference/v1/generate have been added.

To improve operational stability, local worker parallel startup during tensor parallelism (TP > 1) initialization (reducing startup time by about 50 seconds for TP8) and a watchdog feature to output stack traces for stuck engines and workers were introduced. Furthermore, functional parity for the Rust frontend was advanced, resolving HTTP 500 errors during long conversations and detecting unsupported flags at startup as errors.

Supported Models and Hardware

v0.33.0 broadly implements support for numerous latest model architectures and hardware-specific optimizations.

Newly Supported Models

Inference and serving are now newly supported for the following model architectures and derivative models:

  • Reflection Beam-1.0: A 501B MoE (23B active) model supported with DFlash2 draft models, Blackwell-optimized fused kernels, and an inference/tool parser.
  • Yandex AliceAI-Foundation-80B-A3B: Supported model featuring a Kimi Delta Attention hybrid structure, 512-expert MoE, and 262K context.
  • Recursive Flint: A 40B MoE (2.5B active) model supporting BF16 and NVFP4 GGUF checkpoints, as well as small-batch decode kernels for Blackwell.
  • Muse Spark Open: Supported including inference and tool parsers.
  • Step-5 Preview: Supported for operation on the NVIDIA SM100 architecture and with LoRA.
  • NVIDIA Alpamayo 2 Super / Nemotron-Labs Dory1.0-4B / Nemotron Parse 3.5: Nemotron Parse 3.5 supports MTP proposals on Model Runner V2 (MRV2).
  • MiniCPM-OCR 1.0 / Domyn Edge / ColUltraVBERT: Supported various models including document OCR and visual document retrievers.
  • Other Models: Supported embedding model LFM2.5-Embedding-350M, Kolibri 1 via the Transformers backend, Decision 3.0, and Cloudflare Clef-Flash.

Functional Enhancements of Existing Models and Multimodal Improvements

Significant functional expansions have also been made to existing major models:

  • DeepSeek-V4 / V4.1: Added decode indexer block selection reducing about 850 microseconds per step in 1M context, direct loading into DeepGEMM layout, and FlashMLA sparse prefill for SM90 environments.
  • GLM: FP8 KV cache became the default setting for GLM-5.3-Flash (9.8% improvement in end-to-end throughput in benchmarks), and Blackwell-optimized mHC TileLang decode kernels were added.
  • Qwen: Supported FP8 QSA KV cache in Ampere environments for Qwen3.8-Flash-Next, and NVFP4 PLE embeddings to fit within a single DGX Spark.
  • Multimodal: Added batch vision encoders (34% to 85% throughput improvement) and compile_mm_encoder for PaddleOCR-VL, full CUDA Graph conversion for DeepSeek-VL2’s ViT, and bug fixes related to image processing in EXAONE-4.5.

Hardware and Accelerator Support

Dedicated features and optimizations have also been added for accelerators and CPUs from various companies:

  • NVIDIA GPU: Numerous optimizations were introduced for the Blackwell generation (SM100, B300, GB300), alongside low-latency BF16 GEMM for SM107 and CUDA Graph execution in FlashInfer speculative decoding for the SM80 generation.
  • AMD ROCm: The official default environment migrated to ROCm 10.0 (ROCm 7.2 continues to be provided). The gfx1250 architecture has been enabled, supporting operation for gpt-oss-120b, DeepSeek-R1, DeepSeek-V4-Flash, and MiniMax-M3. In CDNA4 environments, DeepSeek-V4 MXFP4 indexer K-cache is now enabled by default.
  • AWS EFA Container: A new official Docker image (vllm/vllm-openai:v0.33.0-efa, x86_64 and aarch64) bundled with the AWS EFA library and Mooncake transfer engine has been provided, facilitating KV transfers and multi-node inference on EC2 instances.
  • Intel XPU: Supported SYCL FlashAttention, SYCL GDN prefill backends, and the torch MXFP4 linear backend.
  • CPU: zentorch 2.14.0.1 was introduced for Zen-series CPUs (AVX512-FP16) adding float16 support, and compile wait times on the first request (11 to 20 seconds) were resolved via MRV2 Triton pre-warmup.

How to Get It

Depending on your platform and installation method, run the following commands to update to the latest version.

Installation via uv or pip

The procedure when using the recommended uv is as follows. Wheels for the CUDA 13.0 environment will be installed.

uv pip install vllm --torch-backend=auto
vllm download-kernels

When using standard pip, run the following commands.

pip install vllm
vllm download-kernels

To reduce startup time, running vllm download-kernels after installation to fetch pre-compiled kernels is recommended.

For AMD ROCm Environments

In ROCm 10.0 or ROCm 7.2 environments, specify the dedicated repository URL for installation.

# For ROCm 10.0
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm100

# For ROCm 7.2
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm72

Using Docker Images

When using a container environment, pull the latest image suited to your use case. Starting with this version, short-name aliases (vllm/vllm) are also available.

# CUDA 13.0 environment (standard)
docker pull vllm/vllm-openai:v0.33.0
# Or short name
docker pull vllm/vllm:v0.33.0

# AWS EFA environment
docker pull vllm/vllm-openai:v0.33.0-efa

# ROCm 10.0 environment
docker pull vllm/vllm-openai-rocm:v0.33.0

# CPU environment
docker pull vllm/vllm-openai-cpu:v0.33.0

Related Articles

What to Read Next

Sources