AI Inference Reshapes Memory Stack With HBF, SSD PODs and SRAM

The bottleneck in AI inference is shifting from computing power to memory capacity. Although mixture-of-experts models activate only a subset of experts for computation, all model weights must remain resident in GPU memory. KV cache requirements for long-context workloads also grow linearly with conversation length. Expensive high-bandwidth memory cannot handle all data on its own, making a redesigned storage hierarchy critical to reducing costs and expanding capacity.

TrendForce published a report on April 23, 2026, which TechNews carried on May 5. It proposed using HBM for hot data and HBF for warm data. HBF combines advanced packaging with the capacity and cost advantages of NAND flash, but commercialization remains constrained by packaging processes and the characteristics of NAND. The report did not disclose investment amounts, a mass-production timeline or specific technical specifications.

The focus of AI computing is shifting from model training to inference at scale, broadening memory requirements beyond bandwidth to include capacity for long contexts and large volumes of intermediate state. Existing HBM is constrained by capacity and cost, making HBF—with its high capacity and low power consumption—a potential key technology for reshaping AI server storage architecture.

Turing Award winner David Patterson recently said HBF could follow HBM as the next core technology for easing the AI inference memory bottleneck. The architecture uses stacked NAND flash for high-capacity data storage while HBM handles high-speed data transfers, supporting the growing volume of contextual data and computational state required during inference.