vLLM v0.34.0 Released with Extensible KV Cache and New Models

At a Glance
| Item | Value |
|---|---|
| Repository | vllm-project/vllm |
| Version | v0.34.0 |
| Published | 2026-11-17 |
| License | Apache-2.0 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
vLLM v0.34.0 has been released. vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs (primary language: Python, license: Apache-2.0). According to the release notes, this version is built from 1,061 commits by 412 contributors.
The change with the largest impact for users is that the Extensible KV cache is now enabled by default, and the default value for gpu_memory_utilization is set to 1.0. Since the KV cache size is determined based on the actually free memory after warmup, it is reported that cache capacity increases and throughput improves for many models. On the other hand, Model Runner V1 has been removed, and the default value for --block-size has been changed. If you have fixed startup options or environment variables, please check the following before updating.
Breaking Changes and Deprecations
| Old | New (Action Required) | PR |
|---|---|---|
VLLM_USE_V2_MODEL_RUNNER=0 (Model Runner V1) |
V1 has been removed. Setting this will cause startup to fail. Please remove the environment variable. V1 logits processors need to be ported to vllm.v1.worker.gpu.sample.logits_processor |
#63373 |
--block-size default 16 |
Default 64. Previous behavior can be used with --block-size 16 (paged attention in ROCm remains 16) |
#63231 |
| Legacy KV cache estimation | Extensible KV cache enabled by default, gpu_memory_utilization default 1.0. To use legacy sizing, specify --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache |
#56492 |
ROCm 7.2 image (-rocm72) |
Discontinued. Use ROCm 10.0 wheels and images | #61939 |
Legacy huggingface_hub |
huggingface_hub >= 2.0.0 is required. tokenizers >= 0.23.3 and timm >= 1.0.30 are also required |
#64779 |
QuTLASS / fp_quant |
Removed. Checkpoints using fp_quant cannot be loaded |
#65230 |
Marlin W4A8 and VLLM_MARLIN_INPUT_DTYPE |
Removed. Use --quantization humming and VLLM_HUMMING_INPUT_QUANT_CONFIG instead |
#65542 |
| Keye-VL / Keye-VL-1.5 | Removed (loading results in an error) | #64287 |
Users who explicitly specified Model Runner V1, use custom logits processors, use the ROCm 7.2 image, run Marlin W4A8 or fp_quant checkpoints, or run Keye-VL will be affected.
For Marlin, configuration is specified using VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"int8"}' (or float8e4m3). According to the PR, setting VLLM_MARLIN_INPUT_DTYPE results in an error with migration instructions. For W4A8-FP8 compressed-tensors checkpoints, HummingLinearKernel is selected on SM89/SM12x. According to the PR, fp_quant was already unable to be loaded on main.
Key Changes
Defaulting to Extensible KV Cache
On CUDA and ROCm, the KV cache is allocated from actually free memory after CUDA graph capture and warmup. As an example, it is reported that MiniMax-M3 on H200 increased from 433k to 672k tokens. As a trade-off, TPOT may degrade by 6 to 18% at a parallelism of 128 for some models on H200. If latency is your priority, please use the opt-out mentioned above. If an explicit gpu_memory_utilization is specified, a warning will be issued at startup (#66704).
Suffix Decoding Works with Model Runner V2
Previously, method="suffix" required Arctic Inference, forced CPU operation and Model Runner V1, and disabled async scheduling. It now runs on MRV2 as a GPU speculator, and Arctic is no longer required. According to the PR, acceptance rates are comparable to Arctic (1.937 vs 1.923/1.948 tokens/step). The default for num_speculative_tokens is now 8 (previously 24 via suffix_decoding_max_tree_depth). Additionally, a register_speculative_method plugin API for custom speculators has been added (#63205). The draft KV cache now matches the target’s dtype, meaning if the target is FP8, the draft will also be FP8.
Performance Improvements
These are figures stated in the release notes. NVFP4 MoE on Step5 (4xGB200) achieves up to 1.95x decoding speedup, DeepGEMM FP8 MegaMoE on GLM-5.3-Flash delivers 1.6x output throughput, and long prompts on GLM-5.3 see a 1.24 to 1.40x TTFT speedup (GB200, 32K to 128K). Qwen3.5 reduces decoding ITL by 3.0 to 8.8% thanks to breakable CUDA graphs. Dedicated decoding for Muse Spark Open and Kimi-K3 is opt-in.
Operational Features
- Checkpoint/restore: Uses CRIU and
cuda-checkpoint, operated via/checkpoint/prepare,/checkpoint/status, and/checkpoint/restore(#62570). - DP+EP scale-down: Evicts dead ranks and reconfigures experts without restarting. Uses
POST /v1/fault_tolerance/apply. According to the PR, external LB mode (--api-server-count=1 --data-parallel-size-local=1) is required, and recovery is orchestrator-driven (#46370). - NUMA binding: Enabled by default on a best-effort basis. Can be disabled with
--no-numa-bind.
Notes
Foundational work for Kimi K3.1 has been merged, but according to the release notes, K3.1 cannot be run in this version.
Supported Models and Hardware
Newly Supported Models
According to the release notes, there are 10 new models. The main ones are as follows:
- Mistral Large 4: 1T total / 52B active (in FP8 and NVFP4). Can be run on NVIDIA Blackwell / Hopper and AMD MI350. Recipes are available at https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B (#64534).
- MiniMax-M3.1-Flash: Added support for Blackwell NVFP4, ROCm kernels, and DSpark draft (#64111).
- NVIDIA Nemotron-3.5-Transcribe: A 30B-A4B audio model. Supports audio chat and
/v1/audio/transcriptions(#64282). - Ricursive Ripple-65B Preview: Supports BF16 and NVFP4, and includes
ripplereasoning and tool parsers (#65745). - Zyphra ZAYA1: Added support for ZAYA1-8B and ZAYA1-74B-preview (#65302).
- Support was also added for Baidu Unlimited-OCR-2 (#64016), Liquid AI LFM3 / LFM3-VL / LFM3-Audio (#65826, #65827, #65828), NVIDIA Canary-1B-v2 (ASR and X↔En translation, #64215), Ming3-ASR (#63585), and NVIDIA Parakeet TDT (via vllm-parakeet-plugin, #65005).
Kimi K3.1 has foundational support only and cannot be run in this version (unsupported).
Hardware
- ROCm: Release wheels and images are provided exclusively for ROCm 10.0. Dedicated decoding (opt-in) for Kimi-K3 has been added for MI355X. MoRI EPv2 can also be used via
--all2all-backend mori. TRITON_ATTN is now the default attention backend on RDNA3 / RDNA4 (#60418). - Blackwell: Numerous optimizations for GB200 / GB300 have been included.
- XPU: Wheels and images continue to be provided.
How to Get It
The README recommends using uv.
uv pip install vllm
vllm download-kernels # precompiled kernels for faster startup
The release notes also list environment-specific commands.
# CUDA 13.0 (PyPI)
pip install vllm
# ROCm 10.0
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100
For Docker, the default image for CUDA 13.0 is vllm/vllm-openai:v0.34.0. For ROCm, it is vllm/vllm-openai-rocm:v0.34.0.
Before updating, please review the “Breaking Changes and Deprecations" table above. huggingface_hub >= 2.0.0 is now mandatory. A TODO comment remains in the release notes regarding whether the XPU install line can still be resolved. XPU users should refer to the release page if they encounter issues.
Related Articles
- vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades
- vLLM v0.31.0 Released: Fast Restart and Hardware Optimization
- vLLM v0.32.0 Released: Major Updates and Enhancements
- vLLM v0.30.0 Released: Fast Start Weight Caching and New Models
What to Read Next
- Follow this tool → vLLM overview and release history (101 releases tracked)
- Other inference engines and runtimes → llama.cpp / Ollama / SGLang

