vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades

At a Glance
| Item | Value |
|---|---|
| Repository | vllm-project/vllm |
| Version | v0.33.0rc3 |
| Published | 2026-11-02 |
| License | Apache-2.0 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
vLLM, a high-speed and high-efficiency inference and serving engine for LLMs, has released its latest version, “v0.33.0". vLLM is an open-source (Apache-2.0 license) inference framework developed in Python.
This version updates to PyTorch 2.14.0 and moves exclusively to CUDA 13.0, alongside enhanced speculative decoding features that significantly speed up inference, the addition of new methods, and support for numerous new models. Since official build distributions have dropped CUDA 12.9 binaries and older speculative methods such as Medusa have been deprecated, existing environment operators should check their dependency environments before updating.
Breaking Changes and Deprecations
This update deprecates certain execution environments and removes older speculative decoding methods. Environments using the affected configurations will encounter errors upon updating, so please perform migration work.
| Item | Old Specification | New Specification / Migration Target |
|---|---|---|
| Official CUDA Build | CUDA 12.9 wheel / images provided | CUDA 12.9 support dropped, changed to CUDA 13.0 exclusive |
| Speculative Decoding Method | medusa, mlp_speculator |
Removed (migrate to EAGLE, MTP, DFlash, DSpark, etc.) |
| Ray Execution Engine | Legacy Ray compiled-graph executor | Removed (consolidated to RayExecutorV2) |
| Mamba Cache Setting | --mamba-cache-mode |
Deprecated (does nothing if specified) |
| Behavior When Generation Paused | Undefined | Requests sent during generation pause return HTTP 503 |
| Structured Output Backend | LM Format Enforcer | Deprecated (opt-in in v0.34.0, scheduled for removal in v0.35.0) |
| Intermediate Representation Priority | --ir-op-priority |
Deprecated (scheduled for removal in v0.34.0) |
| NIXL Connector | Older version support | NIXL 1.5 or later required |
These changes affect environments deployed by directly specifying CUDA 12.9 containers or wheels, configurations specifying medusa or mlp_speculator for speculative decoding, or environments specifying older execution backends in a Ray setup.
Key Changes
Overhaul of Speculative Decoding and Introduction of New Methods “XPress" and “FLy"
For block-diffusion draft models, “XPress" has been added, which performs causal refinement processing. To solve the issue where block-diffusion draft models like DFlash struggle to increase acceptance lengths due to predicting independently without referencing the immediate preceding tokens, it corrects blocks via Jacobi iteration using a small bottleneck layer and a causal mixer. Fixed-shape iterative processing allows it to be captured as a CUDA Graph, running with low overhead.
In addition, an opt-in verification policy “FLy" that allows for loss tolerance has been implemented. While standard verification aborts at the first mismatching token, FLy allows a single mismatch if the target model’s entropy is high, continuing acceptance if subsequent tokens pass verification. Benchmarks confirm decode throughput improvements of 1.055x to 1.310x on AMD MI355X and 1.041x to 1.273x on NVIDIA B300. Because the output distribution is not completely preserved, using this after weighing the trade-off between quality and speed is recommended.
Furthermore, draft slot reservations on the scheduler have been lifted in DFlash and DSpark. In tests running DeepSeek-V4.1-Flash and DSpark on GB300, output throughput significantly improved from 1373 tok/s to 3390 tok/s alongside the increase in the maximum concurrent sequence limit. Alongside this, draft_lm_head_quantization (supporting NVFP4 and FP8) to quantize the shared lm_head for drafts, and a lookahead block hashing feature that retains tail blocks upon EAGLE or MTP prefix cache hits have been introduced.
Acceleration of Decode Performance in NVIDIA and AMD Environments
Numerous hardware-specific optimizations have been incorporated. In NVIDIA environments, low-latency decoding for Kimi-K3 (1.5x to 2x with TP8/TP16) and DeepSeek-V4.1-Flash (decode steps 18.3% faster at batch size 1) was added for GB300, as well as a path to execute DeepSeek-V4/V4.1 MXFP4 experts as block FP8 via Humming on the Hopper architecture (9% to 21% decode performance improvement on 4x H200).
In AMD ROCm environments, the default distribution migrated to ROCm 10.0 (traditional ROCm 7.2 is also provided). --monokernel decoding for DeepSeek-V4.1-Flash and MiniMax-M3 on MI300X/MI325X/MI355X, ROCM_OCTAVE (3-4bit KV cache) capable of handling 4.6x more KV tokens compared to FP16 on RX 7900 XTX, and optimizations for gfx1250 have been enabled.
API Feature Expansion and Stability Enhancements
On the API and serving side, the OpenAI-compatible /v1/decisions endpoint was added and is enabled by default for supported generative models. Additionally, token embedding outputs and late chunking support in /v1/embeddings, and support for output_mode="derender" in /inference/v1/generate have been added.
To improve operational stability, local worker parallel startup during tensor parallelism (TP > 1) initialization (reducing startup time by about 50 seconds for TP8) and a watchdog feature to output stack traces for stuck engines and workers were introduced. Furthermore, functional parity for the Rust frontend was advanced, resolving HTTP 500 errors during long conversations and detecting unsupported flags at startup as errors.
Supported Models and Hardware
v0.33.0 broadly implements support for numerous latest model architectures and hardware-specific optimizations.
Newly Supported Models
Inference and serving are now newly supported for the following model architectures and derivative models:
- Reflection Beam-1.0: A 501B MoE (23B active) model supported with DFlash2 draft models, Blackwell-optimized fused kernels, and an inference/tool parser.
- Yandex AliceAI-Foundation-80B-A3B: Supported model featuring a Kimi Delta Attention hybrid structure, 512-expert MoE, and 262K context.
- Recursive Flint: A 40B MoE (2.5B active) model supporting BF16 and NVFP4 GGUF checkpoints, as well as small-batch decode kernels for Blackwell.
- Muse Spark Open: Supported including inference and tool parsers.
- Step-5 Preview: Supported for operation on the NVIDIA SM100 architecture and with LoRA.
- NVIDIA Alpamayo 2 Super / Nemotron-Labs Dory1.0-4B / Nemotron Parse 3.5: Nemotron Parse 3.5 supports MTP proposals on Model Runner V2 (MRV2).
- MiniCPM-OCR 1.0 / Domyn Edge / ColUltraVBERT: Supported various models including document OCR and visual document retrievers.
- Other Models: Supported embedding model LFM2.5-Embedding-350M, Kolibri 1 via the Transformers backend, Decision 3.0, and Cloudflare Clef-Flash.
Functional Enhancements of Existing Models and Multimodal Improvements
Significant functional expansions have also been made to existing major models:
- DeepSeek-V4 / V4.1: Added decode indexer block selection reducing about 850 microseconds per step in 1M context, direct loading into DeepGEMM layout, and FlashMLA sparse prefill for SM90 environments.
- GLM: FP8 KV cache became the default setting for GLM-5.3-Flash (9.8% improvement in end-to-end throughput in benchmarks), and Blackwell-optimized mHC TileLang decode kernels were added.
- Qwen: Supported FP8 QSA KV cache in Ampere environments for Qwen3.8-Flash-Next, and NVFP4 PLE embeddings to fit within a single DGX Spark.
- Multimodal: Added batch vision encoders (34% to 85% throughput improvement) and
compile_mm_encoderfor PaddleOCR-VL, full CUDA Graph conversion for DeepSeek-VL2’s ViT, and bug fixes related to image processing in EXAONE-4.5.
Hardware and Accelerator Support
Dedicated features and optimizations have also been added for accelerators and CPUs from various companies:
- NVIDIA GPU: Numerous optimizations were introduced for the Blackwell generation (SM100, B300, GB300), alongside low-latency BF16 GEMM for SM107 and CUDA Graph execution in FlashInfer speculative decoding for the SM80 generation.
- AMD ROCm: The official default environment migrated to ROCm 10.0 (ROCm 7.2 continues to be provided). The gfx1250 architecture has been enabled, supporting operation for gpt-oss-120b, DeepSeek-R1, DeepSeek-V4-Flash, and MiniMax-M3. In CDNA4 environments, DeepSeek-V4 MXFP4 indexer K-cache is now enabled by default.
- AWS EFA Container: A new official Docker image (
vllm/vllm-openai:v0.33.0-efa, x86_64 and aarch64) bundled with the AWS EFA library and Mooncake transfer engine has been provided, facilitating KV transfers and multi-node inference on EC2 instances. - Intel XPU: Supported SYCL FlashAttention, SYCL GDN prefill backends, and the torch MXFP4 linear backend.
- CPU: zentorch 2.14.0.1 was introduced for Zen-series CPUs (AVX512-FP16) adding float16 support, and compile wait times on the first request (11 to 20 seconds) were resolved via MRV2 Triton pre-warmup.
How to Get It
Depending on your platform and installation method, run the following commands to update to the latest version.
Installation via uv or pip
The procedure when using the recommended uv is as follows. Wheels for the CUDA 13.0 environment will be installed.
uv pip install vllm --torch-backend=auto
vllm download-kernels
When using standard pip, run the following commands.
pip install vllm
vllm download-kernels
To reduce startup time, running vllm download-kernels after installation to fetch pre-compiled kernels is recommended.
For AMD ROCm Environments
In ROCm 10.0 or ROCm 7.2 environments, specify the dedicated repository URL for installation.
# For ROCm 10.0
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm100
# For ROCm 7.2
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.33.0/rocm72
Using Docker Images
When using a container environment, pull the latest image suited to your use case. Starting with this version, short-name aliases (vllm/vllm) are also available.
# CUDA 13.0 environment (standard)
docker pull vllm/vllm-openai:v0.33.0
# Or short name
docker pull vllm/vllm:v0.33.0
# AWS EFA environment
docker pull vllm/vllm-openai:v0.33.0-efa
# ROCm 10.0 environment
docker pull vllm/vllm-openai-rocm:v0.33.0
# CPU environment
docker pull vllm/vllm-openai-cpu:v0.33.0
Related Articles
- vLLM v0.34.0 Released with Extensible KV Cache and New Models
- vLLM v0.31.0 Released: Fast Restart and Hardware Optimization
- vLLM v0.32.0 Released: Major Updates and Enhancements
- vLLM v0.30.0 Released: Fast Start Weight Caching and New Models
What to Read Next
- Follow this tool → vLLM overview and release history (101 releases tracked)
- Other inference engines and runtimes → llama.cpp / Ollama / SGLang
Sources
- vLLM v0.33.0 Release Notes
- GitHub PR #62575: [CI/Build] Drop CUDA 12.9 from the release pipeline
- GitHub PR #54448: [Core] Add XPress speculative decoding method
- GitHub PR #53987: [Spec Decode][ROCm] Add FLy: entropy-gated deferred verification for draft-model speculative decoding
- GitHub PR #59468: [Spec Decode] Don’t reserve DFlash/DSpark draft slots in the scheduler token budget
- GitHub PR #59973: [MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head
- GitHub PR #61589: [Core][Spec Decode] Lookahead block hashes for EAGLE prefix caching
- GitHub PR #61270: [Model] Support MTP drafts in the Transformers modeling backend
- GitHub PR #61701: [Spec Decode] Remove Medusa and MLP speculator support

