Google Unveils TurboQuant to Reduce LLM Memory Requirements

When large language models generate text, they use a key-value, or KV, cache to store vectors for previous tokens. Longer contexts increase memory use and inference costs. Google Research developed TurboQuant to address capacity bottlenecks in LLM inference and large-scale vector search, using PolarQuant to quantize vectors and 1-bit QJL to correct residual errors.

Google Research published the findings on March 24, 2026. The paper was first uploaded to arXiv on April 28, 2025, and was listed as an ICLR 2026 paper. In LongBench tests using Llama-3.1-8B-Instruct, TurboQuant reduced the KV cache by at least sixfold while maintaining downstream performance. On an NVIDIA H100, it accelerated 4-bit attention-score computation by as much as eightfold.