SGLang v0.5.23 Released: Major Performance and Feature Updates

SGLang v0.5.23 Released: Major Performance and Feature Updates

At a Glance

Item Value
Repository sgl-project/sglang
Version v0.5.23
Published 2026-11-03
License Apache-2.0
Source type Primary source (the publisher itself)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models. Its latest version, v0.5.23, has been released.

The main highlight of this update is the dramatic improvement in inference performance. In particular, it is reported that when running DeepSeek-V4.1 on GB300, the prefill throughput under load is improved by up to 1.5x, and the time-to-first-token latency for 32K to 64K prompts is reduced by approximately 28%.

Breaking Changes and Deprecations

Several changes have been made that affect existing configurations and commands.

Item Old New
Inference option --speculative-skip-dp-mlp-sync Removed (will cause an error if used)
Attention backend tokenspeed_mla cutedsl_mla (tokenspeed_mla remains as a deprecated alias)

The --speculative-skip-dp-mlp-sync option has been removed because this flag caused servers to crash under specific configurations when Attention DP was in use. Environments currently using DP attention should run without specifying this flag.

Additionally, the tokenspeed_mla backend has been removed, and cutedsl_mla now takes its place. If existing code specifies tokenspeed_mla, a warning will be displayed and it will be mapped to cutedsl_mla, but users are encouraged to explicitly migrate to the new backend name.

Key Changes

Substantial Inference Performance Improvements

For MoE (Mixture-of-Experts) models such as DeepSeek-V4.1 using H200, the first token generation is 30% to 58% faster for 8K prompts. Furthermore, the “low-latency mode," which optionally utilizes NVIDIA’s megakernel and AMD’s monokernel plugins, accelerates the decode speed of DeepSeek-V4.1-Flash by up to 44% on GB300 and up to 75% on MI350X. These improvements provide massive benefits to users looking to leverage the latest GPU hardware for low-latency inference.

Memory Savings and Speedups via Paged Experts

The optional “Paged Experts" feature now allows MoE models that exceed GPU memory capacity to run on a single consumer-grade GPU. This feature operates up to 30x faster compared to CPU offloading, serving as an extremely effective option for users who want to run massive models with limited VRAM resources.

Multimodal and Diffusion Model Optimizations

Image and video generation models such as Cosmos3, Qwen-Image, MOVA, and HunyuanVideo have received optimizations that reduce host syncs within the denoising loop and per-step data transfers. This reduces the time the host spends waiting while the GPU performs computations, improving the efficiency of the overall generation process. Users running image and video generation models on SGLang can benefit from these model-specific optimizations.

Support for New Hardware Platforms

Support for Google TPU7x has been introduced, realized through the PyTorch-native plugin “SGLang-TorchTPU." A new optional LoRA engine has also been introduced, enabling the serving of LoRA adapters for quantized MoE models (FP8 and NVFP4). This is an important update for engineers looking to deploy new hardware or apply LoRA to quantized models.

Supported Models and Hardware

This version adds support for several new LLM, multimodal model, and diffusion model architectures and quantization methods.

  • LLM / VLM: Muse Spark Open, Beam-1.0, MiniMax-M3.1-Flash, MiniCPM-V 4.7, Kev
  • Diffusion: Cosmos3-Nano-Sim (Bimanual, Transfer, Auto), FastH3 Omni, FLUX.3 Image Klein, HY-Nomas, MOSS-Image
  • Quantized Models: Added the MiniMax-M3.1 NVFP4 MoE method, and enabled serving LoRA adapters for quantized MoE models (FP8 and NVFP4).
  • Hardware: Added support for Google TPU7x via “SGLang-TorchTPU," supporting execution in Google TPU environments in addition to NVIDIA and AMD.

These models can now undergo GGUF conversion, inference, and cookbook usage, expanding the choices of supported hardware.

How to Get It

For installation in a development environment, use the following command:

uv pip install --prerelease=allow -e "python"

When setting up an environment using Docker, the following images prepared for each platform can be utilized:

  • NVIDIA (CUDA 13): docker pull lmsysorg/sglang:v0.5.23
  • AMD MI35x: docker pull lmsysorg/sglang:v0.5.23-rocm10-mi35x
  • AMD MI30x: docker pull lmsysorg/sglang:v0.5.23-rocm10-mi30x
  • Intel GPU: docker pull lmsysorg/sglang:v0.5.23-xpu
  • Intel CPU: docker pull lmsysorg/sglang:v0.5.23-xeon
  • Google TPU7x (TorchTPU): docker pull lmsysorg/sglang:v0.1.0-tpu

Related Articles

What to Read Next

Sources