TurboQuant Cuts LLM Memory Use Six Times Without Retraining

When a large language model serves a long conversation, most of the GPU's working memory ends up holding what is called the key-value cache. It is the model's record of every prior token in the conversation, and it grows linearly with the length of the chat. A model that comfortably fits on one accelerator at the start of a session can spill into a second one a few thousand tokens later. The memory cost is the bill that pays for context.

Google researchers have a new tool for that bill. TurboQuant, published this spring at ICLR 2026 by Zandieh and colleagues, compresses the KV cache by a factor of six and speeds up attention computation by roughly eight times. It does so without retraining the underlying model and without calibration data.

How it works

The technique is a two-stage method the authors call PolarQuant. Before quantizing each key and value vector down to three bits, the algorithm applies a random rotation matrix. The rotation does not change the mathematical content of the vector. What it does change is the spread of values across the vector's coordinates: variance gets redistributed evenly, which is exactly the property that makes aggressive low-bit quantization safe.

In practice, that means a 32-bit float gets replaced with a three-bit integer, the cache shrinks proportionally, and attention reads run faster because the data is now small enough to live closer to the compute. The team reports zero accuracy degradation on standard benchmarks across the Gemma and Mistral families.

The community has already shipped it

Independent implementations have already started showing up. A community port to llama.cpp landed on GitHub within weeks of the paper, claiming a 5.2x memory reduction with near-lossless quality, and a vLLM integration with Triton kernels followed shortly after. The open-weight ecosystem tends to absorb this kind of inference-time optimization quickly because it slots in beneath whatever model the user is already running.

The wider context is that inference, not training, is now the dominant cost line for organisations running LLMs at scale. Anything that meaningfully cuts the memory footprint of long-context inference is real money. Coming on the heels of Tufts' neuro-symbolic energy paper, which suggested orders-of-magnitude gains on the model side, TurboQuant is the parallel play on the system side.