The hidden bottleneck in the next generation of reasoning models

The arithmetic is moving off the chip

Reasoning workloads do not simply perform more operations. They repeatedly move model weights and a growing key-value cache through memory. As context and concurrency rise, that movement can leave expensive arithmetic units waiting for data.

NVIDIA’s Rubin architecture makes the direction unusually visible. Its HBM4 subsystem is specified at up to 22 TB/s of bandwidth and 288 GB per GPU. NVIDIA says HBM4 doubles the memory-interface width, explicitly tying the change to larger contexts and high-concurrency inference.

Cheaper intelligence still needs a wider road

The broader economics are improving quickly. Stanford’s AI Index documents a steep decline in the inference cost required to reach a fixed capability level. That makes new products possible, but it also encourages teams to run more tokens, more agents, and more simultaneous sessions.

The result is a useful paradox: lower inference prices increase demand for the part of the system that is hardest to scale cheaply. Memory capacity, bandwidth, packaging, and interconnect are becoming one design problem rather than four separate specifications.

What to watch next

Look beyond peak FLOPS. The more revealing measures will be tokens per watt under long context, sustained bandwidth under real concurrency, and how often systems must offload their cache. The next accelerator race will be judged by how well the whole machine keeps data moving.