Multiverse Computing researchers have reformulated transformer block removal as an Ising glass optimization problem, Hugging Face reported on September 21, 2026. The technique targets large language model depth pruning, which deletes entire layers to reduce memory usage and accelerate inference without restricting removals to consecutive blocks.
The team treats block selection as a constrained binary optimization task by assigning a binary variable to each transformer block, where zero keeps the block and one removes it. A second-order Taylor expansion of model loss yields an approximate Hessian matrix. The diagonal measures each block's individual effect, while off-diagonal values capture pairwise interactions between blocks. Forward and backward passes on a small calibration dataset generate the Hessian once, allowing engineers to evaluate different compression targets using cheap energy calculations without running the underlying model.
Benchmark evaluations without retraining showed marked performance differences under heavy compression. On Llama-3.3-70B-Instruct, cutting 40 of 80 blocks yielded a 76.9 score on MMLU under the optimization method, compared to 54.0 for a block-influence baseline. At 32 cuts, the model scored 76.6 versus 59.3 for the baseline, against an uncompressed score of 82.2. On Qwen3-14B with 12 of 40 blocks removed, the technique led MMLU results by approximately 10 points.
Hardware demands varied depending on the size of the configuration space. Researchers evaluated candidate prunings through brute-force calculations on a single GPU up to tens of billions of states. Assessing 29 billion combinations when removing eight blocks from Llama-3.3-70B required roughly two days. For larger spaces, the team converted the problem to quadratic unconstrained binary optimization format and used an open-source tabu solver to identify low-energy states in seconds. Across several tests, the highest performance came from low-lying excited states rather than the ground state; on Llama-3.1-8B-Instruct with 16 of 32 blocks cut, the 17th excited state removed an early block and outperformed the ground state across benchmarks after light retraining.
Tests extended beyond dense models to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid architecture combining Mamba2, attention, and mixture-of-experts layers. Without retraining, removing two to three mixture-of-experts layers or two attention layers outperformed block-influence methods on AIME25 and GPQA benchmarks.
Multiverse released the implementation code under the CompactifAI repository on GitHub.
