{"id":16568,"date":"2026-11-17T11:04:49","date_gmt":"2026-11-17T02:04:49","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=16568"},"modified":"2026-11-17T11:14:18","modified_gmt":"2026-11-17T02:14:18","slug":"vllm-v0-34-0-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/vllm-v0-34-0-released\/","title":{"rendered":"vLLM v0.34.0 Released with Extensible KV Cache and New Models"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/github.com\/vllm-project\/vllm\">vllm-project\/vllm<\/a><\/td>\n<\/tr>\n<tr>\n<td>Version<\/td>\n<td><a href=\"https:\/\/github.com\/vllm-project\/vllm\/releases\/tag\/v0.34.0\">v0.34.0<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-17<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>Apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>vLLM v0.34.0 has been released. vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs (primary language: Python, license: Apache-2.0). According to the release notes, this version is built from 1,061 commits by 412 contributors.<\/p>\n<p>The change with the largest impact for users is that <strong>the Extensible KV cache is now enabled by default, and the default value for <code>gpu_memory_utilization<\/code> is set to 1.0<\/strong>. Since the KV cache size is determined based on the actually free memory after warmup, it is reported that cache capacity increases and throughput improves for many models. On the other hand, Model Runner V1 has been removed, and the default value for <code>--block-size<\/code> has been changed. If you have fixed startup options or environment variables, please check the following before updating.<\/p>\n<h2>Breaking Changes and Deprecations<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Old<\/th>\n<th>New (Action Required)<\/th>\n<th>PR<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>VLLM_USE_V2_MODEL_RUNNER=0<\/code> (Model Runner V1)<\/td>\n<td>V1 has been removed. Setting this will cause startup to fail. Please remove the environment variable. V1 logits processors need to be ported to <code>vllm.v1.worker.gpu.sample.logits_processor<\/code><\/td>\n<td>#63373<\/td>\n<\/tr>\n<tr>\n<td><code>--block-size<\/code> default 16<\/td>\n<td>Default 64. Previous behavior can be used with <code>--block-size 16<\/code> (paged attention in ROCm remains 16)<\/td>\n<td>#63231<\/td>\n<\/tr>\n<tr>\n<td>Legacy KV cache estimation<\/td>\n<td>Extensible KV cache enabled by default, <code>gpu_memory_utilization<\/code> default 1.0. To use legacy sizing, specify <code>--gpu-memory-utilization 0.9<\/code> or <code>--no-enable-extensible-kv-cache<\/code><\/td>\n<td>#56492<\/td>\n<\/tr>\n<tr>\n<td>ROCm 7.2 image (<code>-rocm72<\/code>)<\/td>\n<td>Discontinued. Use ROCm 10.0 wheels and images<\/td>\n<td>#61939<\/td>\n<\/tr>\n<tr>\n<td>Legacy <code>huggingface_hub<\/code><\/td>\n<td><code>huggingface_hub &gt;= 2.0.0<\/code> is required. <code>tokenizers &gt;= 0.23.3<\/code> and <code>timm &gt;= 1.0.30<\/code> are also required<\/td>\n<td>#64779<\/td>\n<\/tr>\n<tr>\n<td>QuTLASS \/ <code>fp_quant<\/code><\/td>\n<td>Removed. Checkpoints using <code>fp_quant<\/code> cannot be loaded<\/td>\n<td>#65230<\/td>\n<\/tr>\n<tr>\n<td>Marlin W4A8 and <code>VLLM_MARLIN_INPUT_DTYPE<\/code><\/td>\n<td>Removed. Use <code>--quantization humming<\/code> and <code>VLLM_HUMMING_INPUT_QUANT_CONFIG<\/code> instead<\/td>\n<td>#65542<\/td>\n<\/tr>\n<tr>\n<td>Keye-VL \/ Keye-VL-1.5<\/td>\n<td>Removed (loading results in an error)<\/td>\n<td>#64287<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>Users who explicitly specified Model Runner V1, use custom logits processors, use the ROCm 7.2 image, run Marlin W4A8 or <code>fp_quant<\/code> checkpoints, or run Keye-VL will be affected.<\/p>\n<p>For Marlin, configuration is specified using <code>VLLM_HUMMING_INPUT_QUANT_CONFIG='{\"dtype\":\"int8\"}'<\/code> (or <code>float8e4m3<\/code>). According to the PR, setting <code>VLLM_MARLIN_INPUT_DTYPE<\/code> results in an error with migration instructions. For W4A8-FP8 compressed-tensors checkpoints, <code>HummingLinearKernel<\/code> is selected on SM89\/SM12x. According to the PR, <code>fp_quant<\/code> was already unable to be loaded on main.<\/p>\n<h2>Key Changes<\/h2>\n<h3>Defaulting to Extensible KV Cache<\/h3>\n<p>On CUDA and ROCm, the KV cache is allocated from actually free memory after CUDA graph capture and warmup. As an example, it is reported that MiniMax-M3 on H200 increased from 433k to 672k tokens. As a trade-off, TPOT may degrade by 6 to 18% at a parallelism of 128 for some models on H200. If latency is your priority, please use the opt-out mentioned above. If an explicit <code>gpu_memory_utilization<\/code> is specified, a warning will be issued at startup (#66704).<\/p>\n<h3>Suffix Decoding Works with Model Runner V2<\/h3>\n<p>Previously, <code>method=\"suffix\"<\/code> required Arctic Inference, forced CPU operation and Model Runner V1, and disabled async scheduling. It now runs on MRV2 as a GPU speculator, and Arctic is no longer required. According to the PR, acceptance rates are comparable to Arctic (1.937 vs 1.923\/1.948 tokens\/step). The default for <code>num_speculative_tokens<\/code> is now 8 (previously 24 via <code>suffix_decoding_max_tree_depth<\/code>). Additionally, a <code>register_speculative_method<\/code> plugin API for custom speculators has been added (#63205). The draft KV cache now matches the target&#8217;s dtype, meaning if the target is FP8, the draft will also be FP8.<\/p>\n<h3>Performance Improvements<\/h3>\n<p>These are figures stated in the release notes. NVFP4 MoE on Step5 (4xGB200) achieves up to 1.95x decoding speedup, DeepGEMM FP8 MegaMoE on GLM-5.3-Flash delivers 1.6x output throughput, and long prompts on GLM-5.3 see a 1.24 to 1.40x TTFT speedup (GB200, 32K to 128K). Qwen3.5 reduces decoding ITL by 3.0 to 8.8% thanks to breakable CUDA graphs. Dedicated decoding for Muse Spark Open and Kimi-K3 is opt-in.<\/p>\n<h3>Operational Features<\/h3>\n<ul>\n<li><strong>Checkpoint\/restore<\/strong>: Uses CRIU and <code>cuda-checkpoint<\/code>, operated via <code>\/checkpoint\/prepare<\/code>, <code>\/checkpoint\/status<\/code>, and <code>\/checkpoint\/restore<\/code> (#62570).<\/li>\n<li><strong>DP+EP scale-down<\/strong>: Evicts dead ranks and reconfigures experts without restarting. Uses <code>POST \/v1\/fault_tolerance\/apply<\/code>. According to the PR, external LB mode (<code>--api-server-count=1 --data-parallel-size-local=1<\/code>) is required, and recovery is orchestrator-driven (#46370).<\/li>\n<li><strong>NUMA binding<\/strong>: Enabled by default on a best-effort basis. Can be disabled with <code>--no-numa-bind<\/code>.<\/li>\n<\/ul>\n<h3>Notes<\/h3>\n<p>Foundational work for Kimi K3.1 has been merged, but according to the release notes, <strong>K3.1 cannot be run in this version<\/strong>.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<h3>Newly Supported Models<\/h3>\n<p>According to the release notes, there are 10 new models. The main ones are as follows:<\/p>\n<ul>\n<li><strong>Mistral Large 4<\/strong>: 1T total \/ 52B active (in FP8 and NVFP4). Can be run on NVIDIA Blackwell \/ Hopper and AMD MI350. Recipes are available at <a href=\"https:\/\/recipes.vllm.ai\/mistralai\/Mistral-Large-4-1T-A52B\">https:\/\/recipes.vllm.ai\/mistralai\/Mistral-Large-4-1T-A52B<\/a> (#64534).<\/li>\n<li><strong>MiniMax-M3.1-Flash<\/strong>: Added support for Blackwell NVFP4, ROCm kernels, and DSpark draft (#64111).<\/li>\n<li><strong>NVIDIA Nemotron-3.5-Transcribe<\/strong>: A 30B-A4B audio model. Supports audio chat and <code>\/v1\/audio\/transcriptions<\/code> (#64282).<\/li>\n<li><strong>Ricursive Ripple-65B Preview<\/strong>: Supports BF16 and NVFP4, and includes <code>ripple<\/code> reasoning and tool parsers (#65745).<\/li>\n<li><strong>Zyphra ZAYA1<\/strong>: Added support for ZAYA1-8B and ZAYA1-74B-preview (#65302).<\/li>\n<li>Support was also added for <strong>Baidu Unlimited-OCR-2<\/strong> (#64016), <strong>Liquid AI LFM3 \/ LFM3-VL \/ LFM3-Audio<\/strong> (#65826, #65827, #65828), <strong>NVIDIA Canary-1B-v2<\/strong> (ASR and X\u2194En translation, #64215), <strong>Ming3-ASR<\/strong> (#63585), and <strong>NVIDIA Parakeet TDT<\/strong> (via vllm-parakeet-plugin, #65005).<\/li>\n<\/ul>\n<p>Kimi K3.1 has foundational support only and cannot be run in this version (unsupported).<\/p>\n<h3>Hardware<\/h3>\n<ul>\n<li><strong>ROCm<\/strong>: Release wheels and images are provided exclusively for ROCm 10.0. Dedicated decoding (opt-in) for Kimi-K3 has been added for MI355X. MoRI EPv2 can also be used via <code>--all2all-backend mori<\/code>. TRITON_ATTN is now the default attention backend on RDNA3 \/ RDNA4 (#60418).<\/li>\n<li><strong>Blackwell<\/strong>: Numerous optimizations for GB200 \/ GB300 have been included.<\/li>\n<li><strong>XPU<\/strong>: Wheels and images continue to be provided.<\/li>\n<\/ul>\n<h2>How to Get It<\/h2>\n<p>The README recommends using <code>uv<\/code>.<\/p>\n<pre><code class=\"language-bash\">uv pip install vllm\nvllm download-kernels  # precompiled kernels for faster startup\n<\/code><\/pre>\n<p>The release notes also list environment-specific commands.<\/p>\n<pre><code class=\"language-bash\"># CUDA 13.0 (PyPI)\npip install vllm\n\n# ROCm 10.0\npip install vllm --extra-index-url https:\/\/wheels.vllm.ai\/rocm\/0.34.0\/rocm100\n<\/code><\/pre>\n<p>For Docker, the default image for CUDA 13.0 is <code>vllm\/vllm-openai:v0.34.0<\/code>. For ROCm, it is <code>vllm\/vllm-openai-rocm:v0.34.0<\/code>.<\/p>\n<p>Before updating, please review the &#8220;Breaking Changes and Deprecations&#8221; table above. <code>huggingface_hub &gt;= 2.0.0<\/code> is now mandatory. A TODO comment remains in the release notes regarding whether the XPU install line can still be resolved. XPU users should refer to the release page if they encounter issues.<\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/vllm-v033-0-released\/\">vLLM v0.33.0 Released: CUDA 13.0, Speculative Decoding Upgrades<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/05\/vllm-v0310-released\/\">vLLM v0.31.0 Released: Fast Restart and Hardware Optimization<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/vllm-v0320-released-2\/\">vLLM v0.32.0 Released: Major Updates and Enhancements<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/22\/vllm-v0-30-0-released\/\">vLLM v0.30.0 Released: Fast Start Weight Caching and New Models<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Follow this tool<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM overview and release history (101 releases tracked)<\/a><\/li>\n<li><strong>Other inference engines and runtimes<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-llama-cpp-en\/\">llama.cpp<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-ollama-en\/\">Ollama<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-sglang-en\/\">SGLang<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/vllm-project\/vllm\/releases\/tag\/v0.34.0\">https:\/\/github.com\/vllm-project\/vllm\/releases\/tag\/v0.34.0<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/vllm-project\/vllm\/pull\/64534\">https:\/\/github.com\/vllm-project\/vllm\/pull\/64534<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/vllm-project\/vllm\/pull\/61939\">https:\/\/github.com\/vllm-project\/vllm\/pull\/61939<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>vLLM v0.34.0 is out with Extensible KV cache enabled by default, new model support, and breaking changes. Read the translation of the release notes.<\/p>\n","protected":false},"author":1,"featured_media":16567,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[968,1187,163,1573,1547,592],"class_list":["post-16568","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-blackwell-en","tag-cuda-en","tag-gguf-en","tag-rocm-en","tag-verified","tag-vllm-en"],"lang":"en","translations":{"en":16568,"ja":16566},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/16568","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=16568"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/16568\/revisions"}],"predecessor-version":[{"id":18139,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/16568\/revisions\/18139"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/16567"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=16568"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=16568"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=16568"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}