Pruning LLMs Like a Physicist: Constrained Binary Optimization

September 21, 2026

At a Glance

Values determined by this site’s code at collection time. Dates are JST.

Overview

Multiverse Computing has announced “CBO (Constrained Binary Optimization)," a method that formulates depth pruning—removing Transformer blocks from large language models (LLMs) to speed them up—as an Ising model optimization problem in physics. This method treats whether to remove each block as a binary variable (spin) and takes inter-block interactions into account, demonstrating that it can maintain model performance at a higher compression rate than traditional methods based on independent metrics.

Claims and Evidence

The presenters claim that the impact of block removal on model loss is not independent for each block, but varies depending on the combination of which blocks are removed simultaneously. To account for this interaction, the model loss was second-order Taylor expanded with respect to the binary variables, constructing a Hessian matrix that indicates the coupling between blocks. This converts the problem of finding the optimal block combination into an energy minimization problem for an Ising glass.

According to the presenters’ measurements in verification using Llama-3.3-70B-Instruct, even when 50% of the blocks were removed (40/80 blocks) without retraining, CBO maintained performance significantly higher than existing prominent methods (Block influence). Specifically, on MMLU, a metric measuring general knowledge, the existing method drops to 54.0, while CBO records 76.9, creating a difference of about 23 points.

Llama-3.3-70B-Instruct Benchmark Results (Without Retraining)

Furthermore, in the case where 12/40 blocks were removed in Qwen3-14B, CBO showed a lead of about 10 points in MMLU. In addition, it was shown to be applicable without retraining even to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, which has a heterogeneous architecture where Mamba2, Attention, and Mixture-of-Experts (MoE) layers are mixed, discovering configurations that outperform existing methods in AIME25 and GPQA Diamond, which measures difficult scientific questions.

A feature of this method is that it is possible to easily explore not only the ground state (minimum value) of energy, but also low-energy “excited states." In experiments using Llama-3.1-8B-Instruct, it was confirmed that the 17th excited state configuration showed superior performance across multiple benchmarks after light retraining compared to the ground state. This result overturns the common assumption that removing a single contiguous group of blocks is best.

In terms of computational cost, the calculation of the Hessian matrix is completed in a single forward-backward pass using a small calibration dataset. After that, the cost of calculating the energy of a specific configuration is extremely low, and a brute-force search of all 29,000 million (29 billion) configurations when removing 8 blocks from Llama-3.3-70B was completed in about two days on a single GPU. It is also stated that by using quantum-inspired algorithms such as open-source Tabu search solvers, low-energy states can be reached in seconds even in huge search spaces where brute-force search is impossible.

Prerequisites

The conditions for applying this method and obtaining results equivalent to those in the documentation are as follows.

• Target Models: Llama-3.1-8B-Instruct, Qwen3-14B, Llama-3.3-70B-Instruct, and NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 (hybrid models mixing Mamba2, Attention, and MoE layers).
• Hardware:
  • Hessian matrix calculation and small-scale search (e.g., searching 29 billion configurations for removing 8 blocks from Llama-3.3-70B): A single GPU (specific model name not listed). – Large-scale search: Classical computers, or quantum/quantum-inspired solvers (QAOA, quantum annealing, Tabu search, etc.).
• Software:
  • Public repository: github.com/CompactifAI/Block_removal_through_constrained_binary_optimization. – Solvers: Open-source Tabu search solvers and dedicated solver groups provided by Multiverse Computing.
• Settings and Data:
  • Calibration: A single forward and backward pass using a small calibration dataset is required to calculate inter-block coupling coefficients (Hessian matrix). – Training settings: Basically operates without retraining (zero-shot), but “light retraining" may be combined to maximize performance.

What Can Be Replicated Locally

Readers can apply this depth pruning method to their own models using the published code and a general GPU environment.

• Code Usage: Implementation code is publicly available on GitHub, making it possible to calculate the Hessian matrix for one’s own model and search for the optimal block removal configuration.
• Running the Search: It is reported that for millions of configurations, it takes only a few seconds, and even for a brute-force search of 29 billion configurations removing 8 blocks in Llama-3.3-70B, it takes about 2 days on a single GPU. Once calculated, the Hessian matrix can be reused for different reduction target numbers (M).
• Efficient Optimization: Even when the search space is too wide for a brute-force search, using the open-source Tabu solver included in the repository makes it possible to identify good low-energy states (combinations of blocks to remove) within seconds. According to the documentation, it is not strictly necessary to find the exact ground state (minimum energy), and an approach of generating and verifying several low-energy states is effective.
• Application to Heterogeneous Models: It has been shown that the method can be applied as-is without special modifications not only to dense Transformer models but also to complex architectures including MoE and Mamba2.

What the Source Does Not Mention

Information that is important for evaluating the effectiveness of the method but is not described in the documentation is as follows.

• Specific Hardware Specifications: The specific model numbers of the GPU or CPU used for the “2 days" required for brute-force search or the execution of the Tabu solver, as well as the required video memory (VRAM) capacity.
• Calibration Data Details: The specific number of samples, number of tokens, and dataset composition of the “small calibration dataset" required to calculate the Hessian matrix.
• Specific Improvements in Inference Speed: Although it is stated that making the model shorter via block removal provides “predictable inference speed improvements," specific measurement data on throughput (tokens/sec) or latency in a real machine environment are not shown.
• Cost of Retraining (Healing): While examples are shown where performance is improved by “light retraining," details on the computational resources, time, and amount of data used for that retraining are not provided.
• Verification with Other Model Families: Effectiveness or differences in behavior in major models other than Llama, Qwen, and Nemotron (e.g., Mistral, Gemma, Phi, etc.).

Sources

• https://huggingface.co/blog/MultiverseComputingCAI/pruning-llms-like-a-physicist-block-removal-as-an