Understand
High Bandwidth Flash (HBF): A Potential Companion or Challenger to HBM in AI Memory
Overview
The memory industry is facing exponentially rising technical demands from larger AI models and massive KV Caches. To address these challenges, memory makers are developing High Bandwidth Flash (HBF)—stacking and connecting flash dies using advanced packaging similar to High Bandwidth Memory (HBM). While HBF offers vastly increased per-stack memory capacity and competitive bandwidth, it brings drawbacks including slower read and write speeds, endurance limits, and thermal challenges. Industry stakeholders are exploring HBF's role in the memory hierarchy, potentially storing read-mostly data like model weights or shared prompts, though extensive software support and careful integration will be required to avoid pitfalls similar to Intel's past Optane technology.
Key points
- AI inference challenges involve prefill and decode computational regimes, with growing model weights and massive KV Caches creating heavy memory demands.
- High Bandwidth Flash (HBF) stacks flash dies using TSVs similarly to HBM, providing 8 to 16 times higher per-stack capacity.
- HBF trade-offs include significantly slower read and write speeds compared to DRAM, endurance limits, and higher power and thermal requirements.
- Sandisk, SK hynix, Google, and Tenstorrent are collaborating within an HBF consortium to standardize specifications, with engineering samples expected around late 2027 and high-volume manufacturing targeted for 2029.
- System architects are evaluating HBF as a Tier 1 or Tier 2 memory solution to store read-mostly data such as model weights, though success depends heavily on software support and workload characteristics.