Magnitude CLI 0.2.8 Released with VRAM Fixes and Gemma 4 Support

- 1. At a Glance
- 2. Overview
- 3. Key Changes
- 3.1. Gemma 4 Draft Model Integration and Optimization Improvements
- 3.2. Memory Reduction via Streaming Embedding Tables
- 3.3. Stabilization of Long Prompt Processing and Model Retention on 8 GB GPUs
- 3.4. Optimization of GPU Memory Judgment and “Fits" Label Display
- 3.5. Prevention of Memory Explosions via Codex Request Expansion Size Limits
- 3.6. Bug Fixes for Apple Silicon, Windows, and Integrated GPU Environments
- 4. Supported Models and Hardware
- 5. How to Get It
- 6. Related Articles
- 7. What to Read Next
- 8. Sources
At a Glance
| Item | Value |
|---|---|
| Repository | magnitudedev/magnitude |
| Version | @magnitudedev/cli@0.2.8 |
| Published | 2026-10-13 |
| License | Apache-2.0 |
| Source type | Primary source (the publisher itself) |
Values determined by this site’s code when the information was collected. Dates are JST.
Overview
A new version of the CLI tool for the hardware-auto-optimized inference engine “magnitude", @magnitudedev/cli@0.2.8, has been released. Magnitude is an open-source inference engine developed in Rust under the Apache-2.0 license. It compiles and tunes kernels directly on the device according to the execution environment’s hardware, allowing open models to run at high speeds across diverse environments such as Apple Silicon, NVIDIA, AMD, and CPUs.
The most significant changes in this version are improvements to VRAM management, focusing on 8 GB GPU environments, and a reduction in memory consumption. The adequacy check has been revised to correctly exclude areas reserved by the GPU driver. Additionally, streaming embedding tables reduces model memory usage, significantly mitigating model unloads and various crashes caused by tight VRAM capacity. Users who have struggled with stable operation in such environments are encouraged to update.
Key Changes
Gemma 4 Draft Model Integration and Optimization Improvements
Gemma 4 models are now paired with assistant models in the catalog, enabling them to be utilized as a draft head for faster text generation. Furthermore, optimization processing during the initial load has been improved to fit within the set time limit, resolving an issue where parts would operate unoptimized after a version update. This further accelerates Gemma’s decoding process after the initial load completes, benefiting users who want to run Gemma 4 locally at high speeds.
Memory Reduction via Streaming Embedding Tables
The approach has been changed from keeping model embedding tables continuously in memory to loading them via streaming. This reduces memory consumption when using large-scale models. Specifically, a memory usage reduction of approximately 600 MB has been confirmed with the Qwen3.8 27B model. For users running large-scale models in environments with limited VRAM or main memory capacity, this improvement provides more operational headroom.
Stabilization of Long Prompt Processing and Model Retention on 8 GB GPUs
An issue where models would automatically unload when executing multiple requests simultaneously after a long prompt on NVIDIA and AMD GPUs with 8 GB of VRAM has been fixed. Additionally, the prompt length that can be handled in these 8 GB environments has been significantly increased; processing that previously failed at around 26K tokens with the MiniCPM5 2B model can now handle up to approximately 78K tokens. Furthermore, an issue where processing would hang when handling long prompts on 8 GB AMD GPUs has also been resolved.
Optimization of GPU Memory Judgment and “Fits" Label Display
In NVIDIA and AMD GPU environments, a fix was made to prevent memory areas reserved by the GPU driver itself from being counted as available space. As a result, model recommendation displays and the “Fits" label judgment on the screen will precisely match the conditions under which models can actually be successfully loaded onto the GPU. This prevents issues where labels indicated models would fit, but loads failed due to out-of-memory errors.
Prevention of Memory Explosions via Codex Request Expansion Size Limits
When processing Codex requests compressed with zstd, a restriction was established to immediately halt decoding once the routing upper limit of 128 MB is exceeded (PR #182). Previously, the entire request was expanded before checking the upper limit, creating a risk that a few kilobytes of compressed data could consume gigabytes of memory and pressure the gateway. Moving forward, processing will stop once 128 MB is exceeded, returning an HTTP 413 status.
Bug Fixes for Apple Silicon, Windows, and Integrated GPU Environments
Numerous fixes have also been applied to hardware-specific errors. First, on Mac environments equipped with M1 Max and M1 Ultra, an issue where running Qwen 3.5, 3.6, and 3.8 models failed with a “sampling distribution is nonfinite" error was fixed. In addition, issues where AMD GPUs on Windows crashed when loading models, or where a “target rows differ" error occasionally occurred when executing multiple requests simultaneously, have been resolved. Furthermore, on PCs with integrated GPUs that do not meet Magnitude’s operation requirements, such as the Intel UHD 770, the system has been improved to fall back to the CPU to run the model rather than failing to load and terminating.
Supported Models and Hardware
The model architectures and hardware environments where normal operation and optimization have been confirmed and enhanced in this version are as follows:
Supported Models
- Gemma 4: Added support for pairing with assistant models within the catalog. This allows assistant models to be called as a draft head during inference to improve generation speed. Additionally, the optimization process during initial load has been improved, making decoding processing faster.
- Qwen 3.5 / 3.6 / 3.8: Fixed sampling calculation errors on Apple Silicon environments (M1 Max / M1 Ultra), enabling stable inference on these high-performance chips.
- Qwen3.8 27B: Runtime memory consumption has been reduced by approximately 600 MB through streaming embedding tables, allowing large models to run with fewer resources.
- MiniCPM5 2B: Maximum token count has been significantly expanded from roughly 26K to approximately 78K due to improved memory management in 8 GB GPU environments.
Supported Hardware and Accelerators
- Apple Silicon (M1 Max / M1 Ultra): Resolved nonfinite sampling distribution errors that occurred in specific model architectures.
- NVIDIA GPUs (Models with 8 GB VRAM): Supported accurate memory judgment taking GPU driver reserved memory into account. Stability during long prompt processing has also improved, preventing unintended model unloads.
- AMD GPUs (Models with 8 GB VRAM): Fixed load-time crashes in Windows environments and hang-ups during long prompt processing, improving stability.
- Intel Integrated GPUs (Intel UHD 770, etc.): Added a feature to automatically fall back to CPU inference without error termination in integrated GPU environments that do not meet Magnitude’s requirements, enabling model execution even without a dedicated GPU.
- Multi-Request Environments: Fixed an issue where executing multiple requests simultaneously caused “target rows differ" errors and request failures, improving reliability during parallel execution.
How to Get It
This release documentation does not list specific installation commands like pip install or npm install. To adopt the latest version, @magnitudedev/cli@0.2.8, please check the official release page.
Magnitude is a project written in Rust. To use the latest version, users typically need to retrieve the latest source code from the official repository and rebuild it to match their hardware environment. Additionally, please note that because optimization processing during the initial load has been improved in this version, time must be allowed for optimization during the first execution after updating.
Related Articles
- Magnitude CLI 0.2.7 Released: Major Speedups and Fixes
- Magnitude CLI 0.2.14 Released: Faster CPU Decoding and More
- Magnitude @magnitudedev/cli@0.2.12 Released
- Magnitude CLI 0.2.11 Released with Custom GGUF Support
What to Read Next
- Follow this tool → Magnitude overview and release history (16 releases tracked)
- Other inference engines and runtimes → llama.cpp / Ollama / vLLM

