Breaking the 1.58-bit Barrier for Ternary LLMs with BITCOS
Overview
The research paper and benchmark results covered in this article are unconfirmed information that has not been officially verified. We avoid definitive statements and present this as reported content.
A research paper titled “Breaking the 1.58-bit Barrier for Ternary LLMs" has reportedly been submitted and is currently drawing attention in the technical community for allegedly breaking the 1.58 bits/weight barrier—considered the theoretical information entropy limit in ternary Large Language Models (LLMs)—by achieving 1.485 bits/weight. The research focuses on the bias in weight distributions within actual ternary LLMs and proposes a new data layout scheme called “BITCOS" that leverages the property that zero elements exist with high probability.
While traditional implementations generally pack five ternary weights into a single byte, it is reported that using BITCOS enables higher-density compression and faster unpacking processing. However, these figures and effects are based on the authors’ reports, and sufficient verification by third parties is unconfirmed.
Why It’s Buzzing
On communities such as Hacker News, posts discussing this paper have garnered attention, gathering a score of 150 and 21 comments.
Ternary LLMs are models where each weight is represented by one of three values ${-1, 0, +1}$, and the information-theoretic lower bound cost is stated to be $\log_2 3 \approx 1.585$ bits/weight. However, in the 5-trit packing used in practical deployments (a method storing five ternary weights in one byte), due to power-of-two sizing constraints, it was effectively rounded up to 1.625 bits/weight. This design assumed that the three values appear with equal probability.
The research team measured the symbol distribution of 29 ternary LLM models and reportedly found that zeros (0) account for up to 51.5% of all weights in actual implementations. The proposed BITCOS is a distribution-adaptive layout that achieves a storage cost of $2 – z$ bits per weight element according to this zero density $z$. It is reported that for the sparsest model, this reduces the size down to 1.485 bits/weight, and with unpacking processes optimized for AVX-512, AVX2, and Intel Xe2 GPUs, speedups of up to 1.28x for matrix-vector multiplication in practical kernels, up to 1.18x for end-to-end CPU LLM decode throughput, and up to 1.27x for GPU were achieved.
For engineers who want to run local models in environments with strict VRAM or main memory constraints, there is growing interest in this approach as a way to combine reduced effective storage with improved inference speed.
Discussion Points
Organizing the discussions taking place within the community highlights three main points:
Memory Deployment and Hardware Computation Efficiency
The question of whether the compression format by BITCOS remains merely a file format on storage, or if it is a format that can be handled directly during in-memory or hardware calculations.
Validity of the Ternary Quantization Approach Itself
Observations regarding whether the approach of performing ternary quantization is optimal in the first place, and its superiority compared to other post-training quantization (PTQ) methods.
Application to Embedded Environments and Edge Devices, and Further Compression Methods
Discussions on practical benefits in environments with limited VRAM such as 16GB or edge devices, and the possibility of applying other compression techniques such as arithmetic coding.
Community Reactions
On Memory Deployment and Hardware Computation Efficiency
Participants have praised the concept of breaking the 1.58-bit barrier by leveraging the real-world distribution where 51% of effective weights are zeros. In particular, expectations have been expressed that if dedicated hardware such as custom silicon or ASICs could directly support the BITCOS format in the future, it might achieve overwhelming power efficiency and processing performance for on-device inference.
On the other hand, some voices raise doubts about the behavior during actual inference. Pointing out that calculations in memory might ultimately require expanding back into traditional 1.58-bit formats (5 trits per byte), discussions took place regarding memory bandwidth and unpacking processing overhead.
On the Validity of the Ternary Quantization Approach Itself
Opinions questioning the ternary quantization approach itself have also been raised. Some participants expressed critical views suggesting that vector quantization or trellis-based methods might exhibit superior performance compared to ternary quantization for post-training quantization (PTQ) in this bit-width domain.
On Application to Embedded Environments and Edge Devices, and Further Compression Methods
Engineers running models in resource-constrained environments have shared favorable reactions regarding practical aspects. Users trying to fit the latest quantized models into 16GB of VRAM or wishing to run LLMs on edge devices welcomed such compression techniques as essential. Views were expressed that this could dramatically reduce LLM sizes for embedded systems and significantly improve portability.
As a more advanced idea, some posts suggested that combining information-theoretic techniques such as arithmetic coding with the presence bitmap could squeeze the size down by a few more fractional bits.