{"id":418,"date":"2026-09-10T20:13:42","date_gmt":"2026-09-10T11:13:42","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/09\/10\/exllamav3-v149-released\/"},"modified":"2026-09-20T17:37:17","modified_gmt":"2026-09-20T08:37:17","slug":"exllamav3-v149-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/09\/10\/exllamav3-v149-released\/","title":{"rendered":"ExLlamaV3 v1.4.9 Released: Faster CPU Offloading"},"content":{"rendered":"

<\/p>\n

At a Glance<\/h2>\n
\n\n\n\n\n\n\n\n\n\n
Item<\/th>\nValue<\/th>\n<\/tr>\n<\/thead>\n
Repository<\/td>\nturboderp-org\/exllamav3<\/a><\/td>\n<\/tr>\n
Version<\/td>\nv1.4.9<\/a><\/td>\n<\/tr>\n
Published<\/td>\n2026-09-10<\/td>\n<\/tr>\n
License<\/td>\nMIT<\/td>\n<\/tr>\n
Source type<\/td>\nPrimary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Values determined by this site’s code at collection time. Dates are JST.<\/em><\/p>\n

<\/p>\n

Overview<\/h2>\n

ExLlamaV3 v1.4.9, the latest version of ExLlamaV3, has been released. ExLlamaV3 is an optimized quantization and inference library designed for running LLMs locally on modern consumer GPUs.<\/p>\n

This update speeds up CPU offloading and strengthens support for Intel CPUs (AVX512BW tier), significantly improving performance when running large models in memory-constrained environments.<\/p>\n

Breaking Changes and Deprecations<\/h2>\n

It has been announced that support for PyTorch 2.7 will be retired starting from the next release.<\/p>\n

\n\n\n\n\n\n
Item<\/th>\nChange<\/th>\n<\/tr>\n<\/thead>\n
PyTorch 2.7<\/td>\nSupport will be retired starting from the next release<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Users currently relying on environments dependent on PyTorch 2.7 are recommended to prepare for migration to PyTorch 2.12 or 2.13 beforehand, as it may stop working in future updates.<\/p>\n

Key Changes<\/h2>\n

Support for New Model Architectures<\/h3>\n

The following model architectures have been newly added:
\n– DeepseekV4ForCausalLM (supports DeepseekV4-Flash-Vision-Exp)
\n– Lfm2ForCausalLM (LFM2.5 dense variant)
\n– Spark2_5ForCausalLM (Spark-X2.5)
\n– Glm4MoeLiteForCausalLM (GLM4.7-Flash)<\/p>\n

Engineers who want to use models with these new architectures can perform inference through ExLlamaV3 by updating to this version.<\/p>\n

Faster CPU Offloading and Intel Optimizations<\/h3>\n

CPU offloading speeds have been improved, and support for Intel CPUs (AVX512BW tier) has been enhanced.<\/p>\n

This is expected to improve inference speeds for users who cannot fit large models entirely into GPU memory and run them by offloading parts to the CPU, as well as users on Intel environments supporting the AVX512BW instruction set. This is especially beneficial for running large models on single-GPU setups with limited VRAM capacity.<\/p>\n

Fixes for VRAM Leaks and Timeout Issues<\/h3>\n

The VRAM leak issue that occurred after offloading the vision tower, and a potential worker timeout issue when offloading large models to the CPU, have been fixed.<\/p>\n

Users utilizing models with vision capabilities or running huge models that do not fit into GPU memory alongside CPU offloading will experience improved memory management stability and operational reliability.<\/p>\n

Drastic Reduction in Import Times<\/h3>\n

By vendoring (embedding within the library) a subset of flash-linear-attention<\/code>, the library import time has been significantly reduced. This eliminates the need to initialize transformers<\/code> or the entire flash-linear-attention<\/code>, streamlining the startup process.<\/p>\n

This affects all users looking to reduce waiting times before model loading begins. In particular, it improves startup responsiveness in development environments where scripts are frequently executed or multiple models are switched and used.<\/p>\n

Improved Autosplit Calculation Accuracy<\/h3>\n

The accuracy of autosplit<\/code> calculations when splitting and placing models has been improved.<\/p>\n

This allows for more accurate memory allocation when using multiple GPUs or distributing models across both GPUs and CPUs. As a result, out-of-memory errors are more easily avoided, enabling inference that makes more optimal use of limited hardware resources.<\/p>\n

Supported Models and Hardware<\/h2>\n

Support for the AVX512BW instruction set on Intel CPUs has been enhanced. This improves CPU offloading performance on Intel environments supporting the AVX512BW tier.<\/p>\n

How to Get It<\/h2>\n

Before installing ExLlamaV3, make sure that the appropriate version of PyTorch (CUDA 12.4 or later) is installed. PyTorch dependencies are not automatically handled by pip<\/code>, so prior preparation is required.<\/p>\n

The main installation methods are as follows:<\/p>\n

Installation from PyPI<\/h3>\n
pip install exllamav3\n<\/code><\/pre>\n
Note that the PyPI package does not include pre-built extensions. Therefore, the CUDA toolkit and build prerequisites (such as VS Build Tools on Windows, or gcc and python-dev<\/code> headers on Linux) must be installed on your system.<\/p>\n
Installation Tailored to a Specific CUDA Environment<\/h3>\n
This method involves installing the CUDA-compatible version of torch matching your environment first, followed by installation from the current directory.<\/p>\n
pip install torch --index-url https:\/\/download.pytorch.org\/whl\/cu128\npip install.\n<\/code><\/pre>\n
Setting Build Environment Variables<\/h3>\n
You can control behavior by setting the following environment variables during compilation or installation:<\/p>\n