llama-modchip v2.4.0 Released for CPU Inference of Large MoE

llama-modchip v2.4.0 Released for CPU Inference of Large MoE

At a Glance

Item Value
Published 2026-11-08
License Apache-2.0
Source type Unverified (not confirmed by a primary source)

Values determined by this site’s code when the information was collected. Dates are JST.

Overview

Please note that the contents covered in this article have not undergone independent verification by official sources or third parties, and include unconfirmed information based on reports from the developers.

Version v2.4.0 of “synchopate/llama-modchip" has been released. llama-modchip is an Apache-2.0 licensed project developed primarily in Python, serving as a tool and patch collection designed to run ultra-large MoE (Mixture of Experts) models with large context lengths on general PC and server CPU environments without GPUs.

The standout feature of this version is that, according to reports, it makes it possible to run inference with the 284.33B parameter “DeepSeek-V4-Flash" at a 256K context length in a pure CPU environment equipped with 64GB of main memory. It reportedly introduces proprietary compression technology and cache optimization, presenting a new option for running large local models in environments without GPUs.

Breaking Changes and Deprecations

Regarding the proprietary tensor format introduced in this version, attention must be paid to compatibility with existing inference engines.

Item Traditional Environment (Unpatched llama.cpp / Ollama / LM Studio) llama-modchip v2.4.0 Environment
Tensor Type Recognition Fails to load, throwing unknown tensor type 40 errors Operates normally via GGML_TYPE_Q2_0 (type ID 40) and dedicated kernels
Execution Binary Standard llama-server, etc. Dedicated patched single binary llama-server-modchip-linux-x86_64 or ./modchip

This change affects all users utilizing GGUF models in the “Fractal-64" format provided by this project. Model files cannot be loaded directly from unpatched tools such as llama.cpp, Ollama, or LM Studio, and require the use of dedicated scripts or the bundled pre-built binaries.

Key Changes

Adapting the 284B MoE Model to 64GB RAM Environments

It is reported that the “DeepSeek-V4-Flash" model (284.33B total parameters, 13.17B active) can now be run with an on-disk size of 56.56 GiB and a peak VmRSS of 57.61 to 58.9 GiB at a full 256K context. This combines analytical second-order Hessian compression “SwiGLU-H2", Fractal-64 (Q2_0, 3.75 bpw) MLA transcoding, AVX-512BW registers and L1D/L2/L3 full model tiling, and L2 quadrature MoE pruning. Operating on an AMD Ryzen 9 9955HX environment with 64GB of RAM, it is claimed to achieve a prefill of 43.16 to 44.36 tok/s and decoding of 9.06 to 10.74 tok/s (11.5 to 14.7 tok/s with PVS applied). This affects users who want to run massive models on desktops or servers with 64GB of RAM.

Accelerating CPU Inference for 30B MoE Models

“NVIDIA-Nemotron-3.5-30B-A3B" (30.2B / 3.0B active) can be stored with a disk footprint of 14.35 GiB and executed in a pure CPU environment at 25.0 to 60.4 tok/s (Zen 5 16-core) or 111.62 tok/s (Threadripper). This implements a 1.64MB L2/L3 drafter using “Stockfish-PVS v2" and “RouteCrack" (adapted from chess engine search techniques) alongside L3 cache isolation (HARPCache) via _MM_HINT_NTA. This affects users wishing to perform high-speed agent inference on laptops, mini PCs, and workstations equipped with 16 to 32GB of RAM.

Fast KV Cache Restoration During Claude Code Integration

When integrating with the agent CLI tool “Claude Code" (supporting 40 types of tools), a warm KV cache restoration feature using Slot-0 in /dev/shm has been added. Context restoration is reported to take 18.9 ms for Nemotron-30B and 13.1 to 32.6 ms for DeepSeek-284B, significantly reducing prompt re-evaluation overhead in CPU environments. This affects engineers using local LLMs as coding agents.

Tuning for Processors and Hybrid Environments

Hardware-specific optimization patches and scripts have been added and streamlined, targeting Zen 5, Zen 4 Multi-CCD (Threadripper), AMD 3D V-Cache and Radeon (Vulkan) hybrid setups, Windows environments with NVIDIA RTX (CUDA 12.4) and CPU hybrid distribution (--n-cpu-moe), Apple Silicon (Metal), and AWS Graviton3 (ARM64). CPU core pinning and automatic cache size adjustments are performed according to each environment.

Supported Models and Hardware

Below are the main models and supported hardware platforms where operation and optimization are reported in this version.

Supported Model Architectures

  • DeepSeek-V4-Flash-0731-Fractal
    An ultra-large MoE model with 284.33B total parameters and 13.17B active parameters. By utilizing proprietary technologies such as SwiGLU-H2 and Fractal-64 quantization (3.75 bpw), the GGUF file size on disk is reduced to 56.56 GiB, enabling inference at full 256K context lengths in environments equipped with 64GB of RAM.
  • NVIDIA-Nemotron-3.5-30B-A3B
    A model featuring 30.2B total parameters and 3.0B active parameters (23 SSM + 23 MoE + 6 GQA). It supports Fractal-64 (3.75 bpw), Fractal-MXFP4 (4.25 bpw), and Q4_0 quantization formats, resulting in GGUF file sizes of 14.35 to 17.59 GiB. It is capable of high-speed CPU-only operation at 25.0 to 60.4 tok/s in 16 to 32GB RAM environments.
  • Qwen3-30B-A3B and Qwen3-14B
    Supports quantization formats such as Q3_K_M and Q4_K_M. Optimizations are reported, including norm-preserving dynamic MoE Top-K control (changed from 8 to 4) and SWA safety locks alternating at 3:1.
  • NVIDIA-Nemotron-H-8B-128K
    An 8.0B parameter dense model supporting Q8_0 and Q4_K_M formats. It uses a 48-layer Mamba-2 SSM backbone and is said to allow 100% VRAM retention on GPUs with 8 to 11GB of VRAM.

Supported Processors and Hardware Platforms

The project reportedly provides kernels and auto-tuning tailored to various processor architectures and hybrid configurations.

  • AMD Zen 5 (Ryzen 9 9955HX / 9950X, EPYC 9005 Turin)
    Optimizations leveraging single-cycle 512-bit AVX-512 instructions (VPDPBUSD, VPRORVQ, VBMI, BF16), 48 KB L1D 3D tiling, and 2MB Transparent HugePages (THP) are applied.
  • AMD Zen 4 Multi-CCD (Threadripper PRO 7985WX)
    Utilizes dynamic L3 CCD topology detection via sysfs and deterministic stride specification per CCD (-t 32) to avoid Infinity Fabric bandwidth saturation and improve processing performance.
  • AMD 3D V-Cache + Radeon Vulkan Hybrid (Ryzen 9 7945HX3D + RX 9070 XT)
    Supports a hybrid configuration where attention portions are allocated to the GPU (Vulkan SSM_SCAN shaders) while routed MoE experts are handled on the CPU DDR5 memory side.
  • Intel / AMD CPU + NVIDIA RTX (Windows 10/11)
    Provides a mechanism to automatically detect CUDA 12.4 and Vulkan binaries, calculate the optimal --n-cpu-moe split ratio based on available VRAM, and execute accordingly.
  • Apple Silicon (M1 / M2 / M3 / M4)
    Combines Metal shaders leveraging Unified Memory with ARM64 NEON/AMX instructions in environments like Apple M4 Pro to process long-context prompts without triggering swapping.
  • ARM64 Cloud / Edge (AWS Graviton3 m7g.2xlarge)
    Achieves high decoding speeds through optimizations such as ARM8.4-A DotProd (SDOT) instructions, 256-Bit SVE, and vCPU core pinning (-t 8).

How to Get It

For Linux and macOS environments, users are guided to clone the repository and run the included script to perform everything from automatic setup to launching an interactive chat in one go. The specific steps are as follows:

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

Additionally, procedures are provided to standalone-launch GGUF models downloaded directly from Hugging Face using a precompiled Linux x86_64 single binary (llama-server-modchip-linux-x86_64), or to launch the Web chat UI in Windows environments by double-clicking windows/AVVIA_MODCHIP.bat.

Sources

This article contains unverified information. We will append an update note once it is confirmed by a primary source.