{"id":14796,"date":"2026-11-17T11:03:14","date_gmt":"2026-11-17T02:03:14","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=14796"},"modified":"2026-11-17T11:12:42","modified_gmt":"2026-11-17T02:12:42","slug":"sglang-v0-5-23-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/sglang-v0-5-23-released\/","title":{"rendered":"SGLang v0.5.23 Released: Major Performance and Feature Updates"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/github.com\/sgl-project\/sglang\">sgl-project\/sglang<\/a><\/td>\n<\/tr>\n<tr>\n<td>Version<\/td>\n<td><a href=\"https:\/\/github.com\/sgl-project\/sglang\/releases\/tag\/v0.5.23\">v0.5.23<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-03<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>Apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p>SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models. Its latest version, v0.5.23, has been released.<\/p>\n<p>The main highlight of this update is the dramatic improvement in inference performance. In particular, it is reported that when running DeepSeek-V4.1 on GB300, the prefill throughput under load is improved by up to 1.5x, and the time-to-first-token latency for 32K to 64K prompts is reduced by approximately 28%.<\/p>\n<h2>Breaking Changes and Deprecations<\/h2>\n<p>Several changes have been made that affect existing configurations and commands.<\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Old<\/th>\n<th>New<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Inference option<\/td>\n<td><code>--speculative-skip-dp-mlp-sync<\/code><\/td>\n<td>Removed (will cause an error if used)<\/td>\n<\/tr>\n<tr>\n<td>Attention backend<\/td>\n<td><code>tokenspeed_mla<\/code><\/td>\n<td><code>cutedsl_mla<\/code> (<code>tokenspeed_mla<\/code> remains as a deprecated alias)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p>The <code>--speculative-skip-dp-mlp-sync<\/code> option has been removed because this flag caused servers to crash under specific configurations when Attention DP was in use. Environments currently using DP attention should run without specifying this flag.<\/p>\n<p>Additionally, the <code>tokenspeed_mla<\/code> backend has been removed, and <code>cutedsl_mla<\/code> now takes its place. If existing code specifies <code>tokenspeed_mla<\/code>, a warning will be displayed and it will be mapped to <code>cutedsl_mla<\/code>, but users are encouraged to explicitly migrate to the new backend name.<\/p>\n<h2>Key Changes<\/h2>\n<h3>Substantial Inference Performance Improvements<\/h3>\n<p>For MoE (Mixture-of-Experts) models such as DeepSeek-V4.1 using H200, the first token generation is 30% to 58% faster for 8K prompts. Furthermore, the &#8220;low-latency mode,&#8221; which optionally utilizes NVIDIA&#8217;s megakernel and AMD&#8217;s monokernel plugins, accelerates the decode speed of DeepSeek-V4.1-Flash by up to 44% on GB300 and up to 75% on MI350X. These improvements provide massive benefits to users looking to leverage the latest GPU hardware for low-latency inference.<\/p>\n<h3>Memory Savings and Speedups via Paged Experts<\/h3>\n<p>The optional &#8220;Paged Experts&#8221; feature now allows MoE models that exceed GPU memory capacity to run on a single consumer-grade GPU. This feature operates up to 30x faster compared to CPU offloading, serving as an extremely effective option for users who want to run massive models with limited VRAM resources.<\/p>\n<h3>Multimodal and Diffusion Model Optimizations<\/h3>\n<p>Image and video generation models such as Cosmos3, Qwen-Image, MOVA, and HunyuanVideo have received optimizations that reduce host syncs within the denoising loop and per-step data transfers. This reduces the time the host spends waiting while the GPU performs computations, improving the efficiency of the overall generation process. Users running image and video generation models on SGLang can benefit from these model-specific optimizations.<\/p>\n<h3>Support for New Hardware Platforms<\/h3>\n<p>Support for Google TPU7x has been introduced, realized through the PyTorch-native plugin &#8220;SGLang-TorchTPU.&#8221; A new optional LoRA engine has also been introduced, enabling the serving of LoRA adapters for quantized MoE models (FP8 and NVFP4). This is an important update for engineers looking to deploy new hardware or apply LoRA to quantized models.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>This version adds support for several new LLM, multimodal model, and diffusion model architectures and quantization methods.<\/p>\n<ul>\n<li><strong>LLM \/ VLM:<\/strong> Muse Spark Open, Beam-1.0, MiniMax-M3.1-Flash, MiniCPM-V 4.7, Kev<\/li>\n<li><strong>Diffusion:<\/strong> Cosmos3-Nano-Sim (Bimanual, Transfer, Auto), FastH3 Omni, FLUX.3 Image Klein, HY-Nomas, MOSS-Image<\/li>\n<li><strong>Quantized Models:<\/strong> Added the MiniMax-M3.1 NVFP4 MoE method, and enabled serving LoRA adapters for quantized MoE models (FP8 and NVFP4).<\/li>\n<li><strong>Hardware:<\/strong> Added support for Google TPU7x via &#8220;SGLang-TorchTPU,&#8221; supporting execution in Google TPU environments in addition to NVIDIA and AMD.<\/li>\n<\/ul>\n<p>These models can now undergo GGUF conversion, inference, and cookbook usage, expanding the choices of supported hardware.<\/p>\n<h2>How to Get It<\/h2>\n<p>For installation in a development environment, use the following command:<\/p>\n<pre><code class=\"language-bash\">uv pip install --prerelease=allow -e &quot;python&quot;\n<\/code><\/pre>\n<p>When setting up an environment using Docker, the following images prepared for each platform can be utilized:<\/p>\n<ul>\n<li><strong>NVIDIA (CUDA 13):<\/strong> <code>docker pull lmsysorg\/sglang:v0.5.23<\/code><\/li>\n<li><strong>AMD MI35x:<\/strong> <code>docker pull lmsysorg\/sglang:v0.5.23-rocm10-mi35x<\/code><\/li>\n<li><strong>AMD MI30x:<\/strong> <code>docker pull lmsysorg\/sglang:v0.5.23-rocm10-mi30x<\/code><\/li>\n<li><strong>Intel GPU:<\/strong> <code>docker pull lmsysorg\/sglang:v0.5.23-xpu<\/code><\/li>\n<li><strong>Intel CPU:<\/strong> <code>docker pull lmsysorg\/sglang:v0.5.23-xeon<\/code><\/li>\n<li><strong>Google TPU7x (TorchTPU):<\/strong> <code>docker pull lmsysorg\/sglang:v0.1.0-tpu<\/code><\/li>\n<\/ul>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/19\/sglang-v0-5-20-released\/\">SGLang v0.5.20 Released: New Models and Optimizations<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/sglang-v0-5-22-released-2\/\">SGLang v0.5.22 Released: 10x Faster Loading and DeepSeek-V4.1 Boost<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/02\/sglang-v0-5-21-released-2\/\">SGLang v0.5.21 Released with Dynamic PD Role Switching<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/26\/deepseek-v4-pro-0813-released\/\">DeepSeek-V4-Pro-0813 Text Generation Model: ~998GB Memory, GGUF Builds<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Follow this tool<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-sglang-en\/\">SGLang overview and release history (63 releases tracked)<\/a><\/li>\n<li><strong>Other inference engines and runtimes<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-llama-cpp-en\/\">llama.cpp<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-ollama-en\/\">Ollama<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/releases\/tag\/v0.5.23\">sgl-project\/sglang v0.5.23 Release Notes<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/45043\">Pull Request #45043: Remove &#8211;speculative-skip-dp-mlp-sync<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/44803\">Pull Request #44803: Remove the tokenspeed_mla backend and dependency<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/45432\">Pull Request #45432: Drop the fixed 64-item multimodal cap in favor of &#8211;limit-mm-data-per-request<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/45967\">Pull Request #45967: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/46046\">Pull Request #46046: Remove per-step host syncs and uploads in image models<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/46047\">Pull Request #46047: Remove per-step host syncs and uploads in video models<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/45170\">Pull Request #45170: Remove blocking seq_lens D2H from graph replay<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/sgl-project\/sglang\/pull\/44393\">Pull Request #44393: Move the remaining CUDA build paths off Python 3.10<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>SGLang v0.5.23 is released with dramatic inference performance gains, Paged Experts, multi-platform support, and breaking changes.<\/p>\n","protected":false},"author":1,"featured_media":14795,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[604,937,165,169,1547],"class_list":["post-14796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-deepseek-en","tag-gpu-en","tag-moe-en","tag-sglang-en","tag-verified"],"lang":"en","translations":{"en":14796,"ja":14794},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=14796"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14796\/revisions"}],"predecessor-version":[{"id":18026,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14796\/revisions\/18026"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/14795"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=14796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=14796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=14796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}