ExLlamaV3 v1.4.9 Released: Faster CPU Offloading
At a Glance
| Item | Value |
|---|---|
| Repository | turboderp-org/exllamav3 |
| Version | v1.4.9 |
| Published | 2026-09-10 |
| License | MIT |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code at collection time. Dates are JST.
Overview
ExLlamaV3 v1.4.9, the latest version of ExLlamaV3, has been released. ExLlamaV3 is an optimized quantization and inference library designed for running LLMs locally on modern consumer GPUs.
This update speeds up CPU offloading and strengthens support for Intel CPUs (AVX512BW tier), significantly improving performance when running large models in memory-constrained environments.
Breaking Changes and Deprecations
It has been announced that support for PyTorch 2.7 will be retired starting from the next release.
| Item | Change |
|---|---|
| PyTorch 2.7 | Support will be retired starting from the next release |
Users currently relying on environments dependent on PyTorch 2.7 are recommended to prepare for migration to PyTorch 2.12 or 2.13 beforehand, as it may stop working in future updates.
Key Changes
Support for New Model Architectures
The following model architectures have been newly added:
– DeepseekV4ForCausalLM (supports DeepseekV4-Flash-Vision-Exp)
– Lfm2ForCausalLM (LFM2.5 dense variant)
– Spark2_5ForCausalLM (Spark-X2.5)
– Glm4MoeLiteForCausalLM (GLM4.7-Flash)
Engineers who want to use models with these new architectures can perform inference through ExLlamaV3 by updating to this version.
Faster CPU Offloading and Intel Optimizations
CPU offloading speeds have been improved, and support for Intel CPUs (AVX512BW tier) has been enhanced.
This is expected to improve inference speeds for users who cannot fit large models entirely into GPU memory and run them by offloading parts to the CPU, as well as users on Intel environments supporting the AVX512BW instruction set. This is especially beneficial for running large models on single-GPU setups with limited VRAM capacity.
Fixes for VRAM Leaks and Timeout Issues
The VRAM leak issue that occurred after offloading the vision tower, and a potential worker timeout issue when offloading large models to the CPU, have been fixed.
Users utilizing models with vision capabilities or running huge models that do not fit into GPU memory alongside CPU offloading will experience improved memory management stability and operational reliability.
Drastic Reduction in Import Times
By vendoring (embedding within the library) a subset of flash-linear-attention, the library import time has been significantly reduced. This eliminates the need to initialize transformers or the entire flash-linear-attention, streamlining the startup process.
This affects all users looking to reduce waiting times before model loading begins. In particular, it improves startup responsiveness in development environments where scripts are frequently executed or multiple models are switched and used.
Improved Autosplit Calculation Accuracy
The accuracy of autosplit calculations when splitting and placing models has been improved.
This allows for more accurate memory allocation when using multiple GPUs or distributing models across both GPUs and CPUs. As a result, out-of-memory errors are more easily avoided, enabling inference that makes more optimal use of limited hardware resources.
Supported Models and Hardware
Support for the AVX512BW instruction set on Intel CPUs has been enhanced. This improves CPU offloading performance on Intel environments supporting the AVX512BW tier.
How to Get It
Before installing ExLlamaV3, make sure that the appropriate version of PyTorch (CUDA 12.4 or later) is installed. PyTorch dependencies are not automatically handled by pip, so prior preparation is required.
The main installation methods are as follows:
Installation from PyPI
pip install exllamav3
Note that the PyPI package does not include pre-built extensions. Therefore, the CUDA toolkit and build prerequisites (such as VS Build Tools on Windows, or gcc and python-dev headers on Linux) must be installed on your system.
Installation Tailored to a Specific CUDA Environment
This method involves installing the CUDA-compatible version of torch matching your environment first, followed by installation from the current directory.
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install.
Setting Build Environment Variables
You can control behavior by setting the following environment variables during compilation or installation:
MAX_JOBS: By default, the build tool ninja may try to spawn many processes in parallel and exhaust system memory. If you experience out-of-memory issues, set this to an appropriate value such as4.EXLLAMA_NOCOMPILE: Set this to install without compiling C++/CUDA extensions. Using this setting allows Torch to build and load them at runtime.
Related Articles
Sources
Update History
- 2026-09-19: Rewrote the article from re-collected sources and restored it from draft to published.