vLLM v0.34.0 Released with Extensible KV Cache and New Models

vLLM v0.34.0 Released with Extensible KV Cache and New Models

At a Glance

Item Value
Repository vllm-project/vllm
Version v0.34.0
Published 2026-11-17
License Apache-2.0
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

vLLM v0.34.0 has been released. vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs (primary language: Python, license: Apache-2.0). According to the release notes, this version is built from 1,061 commits by 412 contributors.

The change with the largest impact for users is that the Extensible KV cache is now enabled by default, and the default value for gpu_memory_utilization is set to 1.0. Since the KV cache size is determined based on the actually free memory after warmup, it is reported that cache capacity increases and throughput improves for many models. On the other hand, Model Runner V1 has been removed, and the default value for --block-size has been changed. If you have fixed startup options or environment variables, please check the following before updating.

Breaking Changes and Deprecations

Old New (Action Required) PR
VLLM_USE_V2_MODEL_RUNNER=0 (Model Runner V1) V1 has been removed. Setting this will cause startup to fail. Please remove the environment variable. V1 logits processors need to be ported to vllm.v1.worker.gpu.sample.logits_processor #63373
--block-size default 16 Default 64. Previous behavior can be used with --block-size 16 (paged attention in ROCm remains 16) #63231
Legacy KV cache estimation Extensible KV cache enabled by default, gpu_memory_utilization default 1.0. To use legacy sizing, specify --gpu-memory-utilization 0.9 or --no-enable-extensible-kv-cache #56492
ROCm 7.2 image (-rocm72) Discontinued. Use ROCm 10.0 wheels and images #61939
Legacy huggingface_hub huggingface_hub >= 2.0.0 is required. tokenizers >= 0.23.3 and timm >= 1.0.30 are also required #64779
QuTLASS / fp_quant Removed. Checkpoints using fp_quant cannot be loaded #65230
Marlin W4A8 and VLLM_MARLIN_INPUT_DTYPE Removed. Use --quantization humming and VLLM_HUMMING_INPUT_QUANT_CONFIG instead #65542
Keye-VL / Keye-VL-1.5 Removed (loading results in an error) #64287

Users who explicitly specified Model Runner V1, use custom logits processors, use the ROCm 7.2 image, run Marlin W4A8 or fp_quant checkpoints, or run Keye-VL will be affected.

For Marlin, configuration is specified using VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"int8"}' (or float8e4m3). According to the PR, setting VLLM_MARLIN_INPUT_DTYPE results in an error with migration instructions. For W4A8-FP8 compressed-tensors checkpoints, HummingLinearKernel is selected on SM89/SM12x. According to the PR, fp_quant was already unable to be loaded on main.

Key Changes

Defaulting to Extensible KV Cache

On CUDA and ROCm, the KV cache is allocated from actually free memory after CUDA graph capture and warmup. As an example, it is reported that MiniMax-M3 on H200 increased from 433k to 672k tokens. As a trade-off, TPOT may degrade by 6 to 18% at a parallelism of 128 for some models on H200. If latency is your priority, please use the opt-out mentioned above. If an explicit gpu_memory_utilization is specified, a warning will be issued at startup (#66704).

Suffix Decoding Works with Model Runner V2

Previously, method="suffix" required Arctic Inference, forced CPU operation and Model Runner V1, and disabled async scheduling. It now runs on MRV2 as a GPU speculator, and Arctic is no longer required. According to the PR, acceptance rates are comparable to Arctic (1.937 vs 1.923/1.948 tokens/step). The default for num_speculative_tokens is now 8 (previously 24 via suffix_decoding_max_tree_depth). Additionally, a register_speculative_method plugin API for custom speculators has been added (#63205). The draft KV cache now matches the target’s dtype, meaning if the target is FP8, the draft will also be FP8.

Performance Improvements

These are figures stated in the release notes. NVFP4 MoE on Step5 (4xGB200) achieves up to 1.95x decoding speedup, DeepGEMM FP8 MegaMoE on GLM-5.3-Flash delivers 1.6x output throughput, and long prompts on GLM-5.3 see a 1.24 to 1.40x TTFT speedup (GB200, 32K to 128K). Qwen3.5 reduces decoding ITL by 3.0 to 8.8% thanks to breakable CUDA graphs. Dedicated decoding for Muse Spark Open and Kimi-K3 is opt-in.

Operational Features

  • Checkpoint/restore: Uses CRIU and cuda-checkpoint, operated via /checkpoint/prepare, /checkpoint/status, and /checkpoint/restore (#62570).
  • DP+EP scale-down: Evicts dead ranks and reconfigures experts without restarting. Uses POST /v1/fault_tolerance/apply. According to the PR, external LB mode (--api-server-count=1 --data-parallel-size-local=1) is required, and recovery is orchestrator-driven (#46370).
  • NUMA binding: Enabled by default on a best-effort basis. Can be disabled with --no-numa-bind.

Notes

Foundational work for Kimi K3.1 has been merged, but according to the release notes, K3.1 cannot be run in this version.

Supported Models and Hardware

Newly Supported Models

According to the release notes, there are 10 new models. The main ones are as follows:

  • Mistral Large 4: 1T total / 52B active (in FP8 and NVFP4). Can be run on NVIDIA Blackwell / Hopper and AMD MI350. Recipes are available at https://recipes.vllm.ai/mistralai/Mistral-Large-4-1T-A52B (#64534).
  • MiniMax-M3.1-Flash: Added support for Blackwell NVFP4, ROCm kernels, and DSpark draft (#64111).
  • NVIDIA Nemotron-3.5-Transcribe: A 30B-A4B audio model. Supports audio chat and /v1/audio/transcriptions (#64282).
  • Ricursive Ripple-65B Preview: Supports BF16 and NVFP4, and includes ripple reasoning and tool parsers (#65745).
  • Zyphra ZAYA1: Added support for ZAYA1-8B and ZAYA1-74B-preview (#65302).
  • Support was also added for Baidu Unlimited-OCR-2 (#64016), Liquid AI LFM3 / LFM3-VL / LFM3-Audio (#65826, #65827, #65828), NVIDIA Canary-1B-v2 (ASR and X↔En translation, #64215), Ming3-ASR (#63585), and NVIDIA Parakeet TDT (via vllm-parakeet-plugin, #65005).

Kimi K3.1 has foundational support only and cannot be run in this version (unsupported).

Hardware

  • ROCm: Release wheels and images are provided exclusively for ROCm 10.0. Dedicated decoding (opt-in) for Kimi-K3 has been added for MI355X. MoRI EPv2 can also be used via --all2all-backend mori. TRITON_ATTN is now the default attention backend on RDNA3 / RDNA4 (#60418).
  • Blackwell: Numerous optimizations for GB200 / GB300 have been included.
  • XPU: Wheels and images continue to be provided.

How to Get It

The README recommends using uv.

uv pip install vllm
vllm download-kernels  # precompiled kernels for faster startup

The release notes also list environment-specific commands.

# CUDA 13.0 (PyPI)
pip install vllm

# ROCm 10.0
pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.34.0/rocm100

For Docker, the default image for CUDA 13.0 is vllm/vllm-openai:v0.34.0. For ROCm, it is vllm/vllm-openai-rocm:v0.34.0.

Before updating, please review the “Breaking Changes and Deprecations" table above. huggingface_hub >= 2.0.0 is now mandatory. A TODO comment remains in the release notes regarding whether the XPU install line can still be resolved. XPU users should refer to the release page if they encounter issues.

Related Articles

What to Read Next

Sources