{"id":14923,"date":"2026-11-17T11:03:17","date_gmt":"2026-11-17T02:03:17","guid":{"rendered":"https:\/\/localmodelwatch.tsuchitsuchi.com\/?p=14923"},"modified":"2026-11-17T11:12:49","modified_gmt":"2026-11-17T02:12:49","slug":"magnitude-0-2-16-released","status":"publish","type":"post","link":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/magnitude-0-2-16-released\/","title":{"rendered":"Magnitude v0.2.16 Released: Faster Parallel Requests and Docker"},"content":{"rendered":"<p><!-- lmw:facts --><\/p>\n<h2>At a Glance<\/h2>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Item<\/th>\n<th>Value<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Repository<\/td>\n<td><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\">magnitudedev\/magnitude<\/a><\/td>\n<\/tr>\n<tr>\n<td>Version<\/td>\n<td><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/releases\/tag\/@magnitudedev\/cli@0.2.16\">@magnitudedev\/cli@0.2.16<\/a><\/td>\n<\/tr>\n<tr>\n<td>Published<\/td>\n<td>2026-11-04<\/td>\n<\/tr>\n<tr>\n<td>License<\/td>\n<td>Apache-2.0<\/td>\n<\/tr>\n<tr>\n<td>Source type<\/td>\n<td>Primary source (the publisher itself)<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><em>Values determined by this site&#8217;s code when the information was collected. Dates are JST.<\/em><\/p>\n<p><!-- \/lmw:facts --><\/p>\n<h2>Overview<\/h2>\n<p><code>magnitudedev\/magnitude<\/code> has released <code>@magnitudedev\/cli@0.2.16<\/code>. This project is an open-source inference engine that aims to run each model at maximum speed by tuning kernels to match the hardware. It runs on Apple Silicon, NVIDIA, AMD, Intel, or CPU, and is reported to be twice as fast as Ollama.<\/p>\n<p>The most important change in this update is a massive improvement in handling concurrent requests that share prompt prefixes. This is expected to bring dramatic performance improvements for use cases such as agent-based parallel processing.<\/p>\n<h2>Key Changes<\/h2>\n<h3>Faster Shared Prefixes in Concurrent Requests<\/h3>\n<p>The mechanism for reusing existing computation results in concurrent requests sharing the same long prompt prefix (such as parallel sub-agents) has been improved. This allows requests to wait for shared prefix computations and reuse the results, avoiding redundant calculations.<\/p>\n<p>This change benefits users running multiple agents in parallel. For example, using an NVIDIA A10G, the execution time for 8 concurrent requests sharing a 15K-token prompt dropped significantly from 226 s to 3.5 s each. In addition, the shared prefix is kept in the cache even after a batch finishes, enabling fast execution upon repeated use.<\/p>\n<h3>Added Docker Support<\/h3>\n<p>The official Docker image <code>magnitudedev\/magnitude<\/code> for x64 and ARM64 Linux is now available. Docker Compose, Kubernetes, and Podman (including rootless) are also supported.<\/p>\n<p>This affects users who want to use containers in a Linux environment. NVIDIA GPUs are supported via NVIDIA Container Toolkit, and AMD and Intel GPUs are supported via Vulkan. However, since Metal (GPU) cannot be used via containers on a Mac, installing the native macOS app is required to run at full speed. Models and settings are stored in the <code>\/data<\/code> volume, so data is preserved after upgrades.<\/p>\n<h3>Expanded GGUF Quantization Formats and Performance Improvements<\/h3>\n<p>New quantization formats Q1_0, IQ1_S, IQ1_M, IQ2_S, and TQ1_0 models can now be loaded and run. Furthermore, existing formats Q2_K, Q3_K, Q4_1, IQ3_S, IQ4_XS, IQ4_NL, and TQ2_0 are now kept without expanding in size upon loading, making it possible to fit larger models into memory.<\/p>\n<p>This affects users running models in memory-constrained environments. Decode speeds have also improved, with CUDA reporting 13\u201323% faster speeds for Q4_1, Q5_0, Q5_1, Q2_K, Q4_K, and Q5_K, and 2.5x faster speeds for IQ3_XXS. In Metal environments, small batches for Q8_0, Q5_0, and several IQ formats are reported to be 29\u201335% faster.<\/p>\n<h3>Improved Model Optimization Process<\/h3>\n<p>The one-time optimization process that runs after downloading a model has been improved. Previously, starting a chat during optimization could interrupt the process and cause it to run with untuned settings, but optimizations will now complete during loading or when the device is idle.<\/p>\n<p>This affects all users. Statuses such as &#8220;Optimizing&#8221; and &#8220;Optimization pending&#8221; are now displayed in the UI. Note that after applying this update, previously downloaded models will be re-optimized, which may add about 1 minute of extra time during the next load.<\/p>\n<h3>Fix for NVIDIA GPU Decode Speed in Linux Environments<\/h3>\n<p>An issue where memory checks during decoding caused slowdowns when using NVIDIA GPUs on Linux has been fixed. Previously, memory budgets for all GPUs were re-verified on every decode step, but this has been changed to check only the device in use.<\/p>\n<p>This affects users running NVIDIA GPUs on Linux. As a result, the decode step for Qwen3.5 4B using an A100 was reduced from 13.0 ms to 6.7 ms, and prefill became 32% faster.<\/p>\n<h3>Default Chat Settings Saved per Model<\/h3>\n<p>Chat settings (thinking level, system prompt, sampling settings, seed, stop sequences, etc.) can now be saved as defaults for each model.<\/p>\n<p>This affects all users. When starting a new chat or switching models, the default settings for that model are automatically applied. The settings screen allows users to distinguish whether the current chat is using standard settings, the model&#8217;s default settings, or individually modified settings.<\/p>\n<h2>Supported Models and Hardware<\/h2>\n<p>GGUF models with the new quantization formats Q1_0, IQ1_S, IQ1_M, IQ2_S, and TQ1_0 can now be loaded and executed. Additionally, formats Q2_K, Q3_K, Q4_1, IQ3_S, IQ4_XS, IQ4_NL, and TQ2_0 are now preserved without expanding in size upon loading, allowing larger models to fit into memory.<\/p>\n<p>Regarding hardware, operation using the official Docker image is supported on x64 and ARM64 Linux environments. NVIDIA GPUs are available via NVIDIA Container Toolkit, and AMD and Intel GPUs are available via Vulkan. Note that when using containers in a Mac environment, Metal (GPU) routing is not supported, so using the native macOS app is recommended for full-speed operation.<\/p>\n<h2>How to Get It<\/h2>\n<p>When using Docker in a Linux environment, you can use the official image with the following command:<\/p>\n<pre><code class=\"language-bash\">docker pull magnitudedev\/magnitude\n<\/code><\/pre>\n<p>Containers do not update automatically, so please run <code>docker pull<\/code> according to update notifications.<\/p>\n<p><!-- lmw:releases --><\/p>\n<h2>Releases Since Our Last Article<\/h2>\n<p><em>Compiled by Local Model Watch from the project&#8217;s GitHub releases: the versions between this release and the last one we covered, which did not get separate articles.<\/em> <em>Full history: <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-magnitude-en\/\">release tracker<\/a>.<\/em><\/p>\n<div class=\"lmw-table-scroll\" tabindex=\"0\" style=\"overflow-x:auto;-webkit-overflow-scrolling:touch;max-width:100%;\">\n<table style=\"width:max-content;min-width:100%;border-collapse:collapse;\">\n<thead>\n<tr>\n<th>Version<\/th>\n<th>Released<\/th>\n<th>Release notes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>@magnitudedev\/cli@0.2.15<\/td>\n<td>2026-10-30<\/td>\n<td><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/releases\/tag\/%40magnitudedev\/cli%400.2.15\">GitHub<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><!-- \/lmw:releases --><\/p>\n<p><!-- lmw:related --><\/p>\n<h2>Related Articles<\/h2>\n<ul>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/magnitude-cli-0-2-17-released\/\">Magnitude CLI v0.2.17 Released with Dynamic Context Scaling<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/magnitude-cli-0-2-14-released\/\">Magnitude CLI 0.2.14 Released: Faster CPU Decoding and More<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/magnitude-cli-0-2-12-release\/\">Magnitude @magnitudedev\/cli@0.2.12 Released<\/a><\/li>\n<li><a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/2026\/11\/17\/magnitude-cli-0-2-11-released\/\">Magnitude CLI 0.2.11 Released with Custom GGUF Support<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:related --><\/p>\n<p><!-- lmw:next-steps --><\/p>\n<h2>What to Read Next<\/h2>\n<ul>\n<li><strong>Follow this tool<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-magnitude-en\/\">Magnitude overview and release history (16 releases tracked)<\/a><\/li>\n<li><strong>Other inference engines and runtimes<\/strong> \u2192 <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-llama-cpp-en\/\">llama.cpp<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-ollama-en\/\">Ollama<\/a> \/ <a href=\"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/engine-vllm-en\/\">vLLM<\/a><\/li>\n<\/ul>\n<p><!-- \/lmw:next-steps --><\/p>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/releases\/tag\/%40magnitudedev%2Fcli%400.2.16\">magnitudedev\/magnitude @magnitudedev\/cli@0.2.16<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/commit\/b4eb689ee384cfdc16fd628de0727f3044eddc7f\">b4eb689<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/commit\/b7645534de59b20de335800e241eac58f8f67399\">b764553<\/a><\/li>\n<\/ul>\n<h2>Sources<\/h2>\n<ul>\n<li><a href=\"https:\/\/github.com\/magnitudedev\/magnitude\/releases\/tag\/%40magnitudedev\/cli%400.2.16\">https:\/\/github.com\/magnitudedev\/magnitude\/releases\/tag\/%40magnitudedev\/cli%400.2.16<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Magnitude v0.2.16 is out with major speedups for shared prompt prefixes, official Docker support, and new GGUF quantization formats.<\/p>\n","protected":false},"author":1,"featured_media":14922,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[105],"tags":[1187,1475,163,4545,2791,1547],"class_list":["post-14923","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engines-and-tools","tag-cuda-en","tag-docker-en","tag-gguf-en","tag-inference-engine-en","tag-magnitude-en","tag-verified"],"lang":"en","translations":{"en":14923,"ja":14921},"_links":{"self":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14923","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/comments?post=14923"}],"version-history":[{"count":2,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14923\/revisions"}],"predecessor-version":[{"id":18036,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/posts\/14923\/revisions\/18036"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media\/14922"}],"wp:attachment":[{"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/media?parent=14923"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/categories?post=14923"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/localmodelwatch.tsuchitsuchi.com\/en\/wp-json\/wp\/v2\/tags?post=14923"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}