Per-token pricing has become the default commercial language of generative AI, and practically every budget conversation now happens in those terms (tokens consumed, tokens generated, cost per million tokens, etc., etc.).
The problem is that this model treats every AI interaction as a discrete transaction, separate from the infrastructure required to process it. Yes, tokens are a billing unit, but the work they represent also has a physical footprint in memory and storage, and as agentic AI drives context windows into the hundreds of thousands of tokens, that footprint has outgrown the GPU.
So why does this matter? Well, according to recent analysis published in Harvard Business Review (HBR), corporate AI is about to experience a massive “cost shock”, as vendors move away from unmetered or bundled AI features and begin passing on the cost of the infrastructure required to deliver them.
As HBR points out, organizations are currently being paid, in effect, to adopt AI, with major software vendors quietly absorbing the steep costs of GPUs, inference and tokens on their customers’ behalf. That subsidy is not built to last.
Take memory and storage costs, for example. DRAM prices have risen sharply as manufacturers focus on High Bandwidth Memory (HBM) production for AI accelerators, tightening supply for conventional server memory. Meanwhile, demand for AI is exerting sustained pressure on the availability and pricing of high-capacity SSDs.
This is totally unsustainable, and there was always going to come a point at which vendors could no longer continue covering these costs themselves and would need to pass more of them on to customers. That point is now rapidly approaching.
The scale of the shift is hard to overstate. Google was processing 9.7 trillion tokens a month in 2024; by mid-2025 that had risen to 480 trillion, and at this year’s I/O, Sundar Pichai put the figure at 3.2 quadrillion tokens a month. Goldman Sachs projects industry-wide token consumption will multiply another 24 times by 2030. Whatever an organization budgeted for inference last year, the volume assumption behind it is already out of date, and a growing share of that volume is input, not output: agentic workflows that plan, call tools and re-read context on every step are pushing input tokens, not generated ones, to the top of the cost stack.
Your AI budget has a serious NAND problem
The clearest sign of where this is heading came at CES 2026, when Nvidia unveiled what it first called its Inference Context Memory Storage Platform, built around its coming BlueField-4 DPU, and has since formalized as the NVIDIA CMX context memory storage platform. CMX standardizes the offload of the model’s working memory, the KV cache, onto NVMe SSDs, extending GPU memory into a shared, pod-level context tier and making it persistent across inference runs. Nvidia says the approach can deliver up to five times greater power efficiency and up to five times higher tokens-per-second throughput, and it already has more than a dozen storage partners lined up to support it, including Dell, IBM, NetApp, VAST and WEKA.
That is a significant validation point: several storage vendors were already extending GPU memory onto flash before ICMSP existed. Nvidia standardizing the approach across its ecosystem confirms this is now an architectural direction for the industry, not a niche workaround. But it also means the economics of inference are no longer determined solely by GPU pricing; they are now directly exposed to storage media pricing.
That exposure matters because flash pricing is anything but stable. According to VDURA’s own Flash Volatility Index, 30TB TLC SSD pricing rose 472% between Q2 2025 and Q1 2026, and the price multiple between high-capacity QLC flash and hard drives widened from roughly 4.9x to more than 22x over the same period, while HDD pricing moved about 35%. If cost-per-token now depends on flash-resident context tiers, then token costs inherit flash volatility, and organizations running all-flash inference infrastructure are carrying unhedged exposure to the NAND market on every query.
Instead, flash should be reserved for hot context data where its performance genuinely affects inference. This is because not all types of data generated or retained by AI workloads have the same latency requirements. For instance, data retained for future use, such as checkpoints, does not necessarily need flash performance. The hyperscalers worked this out long before the current price cycle: Google, Meta and Microsoft do not run all-flash storage; they run mixed fleets, with just enough flash to saturate compute and intelligent tiering draining everything else to high-density disk. A mixed-fleet architecture separates performance capacity from bulk capacity, giving organizations a structural hedge against volatile flash prices.
Tread carefully
If they aren’t doing so already, infrastructure leaders should model token-cost sensitivity to media pricing before committing capital. Do not forget that agentic workloads can disproportionately increase token consumption on the input and context sides. As a result, cost models must account for how much context requires high-performance flash capacity and how much can be stored on lower-cost tiers. Teams should regularly test how projected AI costs change as CMX-style architectures push more context onto NVMe, and as SSD pricing rises further while context requirements grow.
That all being said, HBR’s concluding analysis is extremely apposite: as long as executives are being asked to redesign their organizations around a technology with unpredictable pricing, they should tread carefully.