{"id":9202,"date":"2026-10-03T00:19:12","date_gmt":"2026-10-02T15:19:12","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/2026\/10\/03\/magnitude-0-2-4-released\/"},"modified":"2026-10-03T00:20:34","modified_gmt":"2026-10-02T15:20:34","slug":"magnitude-0-2-4-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/10\/03\/magnitude-0-2-4-released\/","title":{"rendered":"Magnitude 0.2.4 Released: Faster Inference and Lower Memory"},"content":{"rendered":"

<\/p>\n

At a Glance<\/h2>\n
\n\n\n\n\n\n\n\n\n\n
Item<\/th>\nValue<\/th>\n<\/tr>\n<\/thead>\n
Repository<\/td>\nmagnitudedev\/magnitude<\/a><\/td>\n<\/tr>\n
Version<\/td>\n@magnitudedev\/cli@0.2.4<\/a><\/td>\n<\/tr>\n
Published<\/td>\n2026-10-02<\/td>\n<\/tr>\n
License<\/td>\nApache-2.0<\/td>\n<\/tr>\n
Source type<\/td>\nPrimary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

Values determined by this site’s code when the information was collected. Dates are JST.<\/em><\/p>\n

<\/p>\n

Overview<\/h2>\n

Version 0.2.4 (@magnitudedev\/cli@0.2.4) of “magnitudedev\/magnitude<\/a>” has been released<\/a>. magnitudedev\/magnitude is an open-source inference engine written in Rust and designed for agents, licensed under Apache-2.0. By compiling and tuning kernels directly on your device and optimizing them for the hardware, it can run open models up to 2x faster compared to llama.cpp. It supports Apple Silicon, NVIDIA, AMD, and CPU-only operation.<\/p>\n

The most significant impacts of this update for users are the drastic reduction in memory required to run models and the reduction of initial load kernel tuning times to a maximum of about 1 minute across all devices. This makes it more comfortable to handle larger models and longer contexts while dramatically eliminating startup wait time stress.<\/p>\n

Key Changes<\/h2>\n

Significant Reduction in Memory Usage<\/h3>\n

Memory required by models beyond their weights has been reduced, enabling the operation of larger models and longer contexts. As a specific example, the working memory required when running Qwen3.5-4B has been reduced from 3.7 GB to 120 MB, and the reserved memory for execution has decreased from 10.0 GB to 5.9 GB. There is no drop in inference speed resulting from this memory reduction. Additionally, image processing memory is now allocated when the first image is processed and released when idling.<\/p>\n

Limiting Initial Tuning Time to a Maximum of About 1 Minute<\/h3>\n

An issue where initial model loading took about 20 to 50 minutes on CPU-only machines and appeared to hang has been fixed. Previously, the lower the processing capability of the device, the longer the tuning took, but with this change, tuning completes in a maximum of about 1 minute on all devices, accompanied by progress display. For the initial load of Qwen3.5-4B on an M4 Max environment, the tuning time was reduced to about 50 seconds (formerly 149 seconds on GPU and 279 seconds on CPU), while maintaining subsequent inference speeds. Even for large models like Gemma 4 26B, test data preparation is consolidated into a single step, and kernels that do not finish within 1 minute retain their default settings, keeping the tuning time within 1 minute.<\/p>\n

Speeding Up Q6_K Models on x86 CPUs<\/h3>\n

In x86 architecture CPU environments, inference speeds for models with Q6_K weights have been improved. This is because the processing was changed to convert half-precision scales without routing through slow processor paths. Users running Q6_K quantized models on x86 CPU environments will directly benefit from this.<\/p>\n

Improved Error Handling on Request Failures<\/h3>\n

A bug where requests failed midway\u2014such as running out of memory during a long conversation\u2014yet returned an empty response and appeared to succeed has been fixed. After the update, it returns a 503 error code, a Retry-After<\/code> header, and an error message. This allows agent harnesses like Pi, OpenCode, and Claude Code to automatically retry. Furthermore, memory pressure and request failures from the inference engine that previously failed silently are now logged.<\/p>\n

Fix for Load Failures on M1\/M2 Macs<\/h3>\n

On Macs equipped with M1 and M2 chips, a bug where the default kernel settings exceeded the chip’s allowed thread count and caused model loading to fail has been fixed. With this fix, tuning now begins from the closest workable setting. This is an effective fix for users with M1 or M2 Macs who previously experienced startup failures.<\/p>\n

Fix for Startup Errors via SSH<\/h3>\n

A problem where the magnitude serve<\/code> command failed to start when accessing a Mac via SSH with no console user logged in has been resolved. This improves convenience for developers who start and use servers on Macs remotely.<\/p>\n

Supported Models and Hardware<\/h2>\n

In this version 0.2.4, operational stability and performance in specific model architectures and hardware environments have been substantially improved.<\/p>\n

Supported Models<\/h3>\n

Operation and optimization for the following models have been confirmed in this version:<\/p>\n