A New Memory Standard Takes Aim at AI's Bottleneck

The hardest part of running a large AI model is often not the math. It is getting enough of the model close enough to the processor, fast enough, to keep the expensive chips busy. On August 3, SanDisk and SK hynix published the first open standard for a technology meant to ease that squeeze, a new class of memory they call High Bandwidth Flash.

They released the specification through the Open Compute Project, the industry group that standardizes data center hardware, and presented it at the Future of Memory and Storage conference in Santa Clara. The pitch is simple. Today's fastest AI memory, known as HBM, is quick but small and costly. Flash storage is large and cheap but slow. High Bandwidth Flash tries to sit in the gap, stacking NAND flash dies to reach bandwidth close to HBM while holding far more data.

What the spec describes

The first version supports up to 512 gigabytes in a single package, built from stacks of 8 or 16 NAND dies, with three speed grades running from about 0.4 to 3.0 terabytes per second. It connects to processors over UCIe, an emerging standard for linking chiplets inside a package, which means a GPU could in principle address a large pool of this memory almost as if it sat right next to the compute. For inference, where a model's weights have to be read over and over, capacity at speed is exactly the currency that matters.

Not just two vendors

What makes this more than a product announcement is the company it keeps. The HBF workstream under the Open Compute Project formed in February, and by the summer had drawn in Google and the AI chip startup Tenstorrent to help validate the design. When a hyperscaler and a rival silicon designer agree to test a memory maker's standard, it usually means they want a second source and an open specification, not a proprietary part they cannot shop around.

The memory wall, again

None of this ships tomorrow. A first specification is a long way from parts in a rack, and flash is slower to write and wears out in ways HBM does not, which limits where it fits. But the direction is telling. The industry has spent two years pouring money into compute, and it is now reckoning with the humbler problem of feeding those processors data. The memory wall has been a known obstacle for decades. This is the industry trying to build a door in it.