New to this? Read it in simple words
- AI chips can calculate very fast. But they must wait when memory cannot send data quickly enough.
- NVIDIA says its Rubin GPU will use HBM4, a high-bandwidth memory. It lists speeds of up to 22 terabytes per second for each GPU.
- Memory speed is becoming as important as the power of the main AI chip. Cheaper AI also creates more demand for memory, which is still expensive.
- GPU
- A chip that does many calculations at once. Most AI runs on GPUs.
- High-bandwidth memory
- Memory that can move very large amounts of data to a chip quickly.
- Terabyte
- A unit of computer data. One terabyte is about one thousand gigabytes.
AI chips can calculate very fast. But they still have to wait when memory cannot send data quickly enough.
Why memory can slow a fast chip
AI reasoning moves a large amount of data. It often loads model weights and recent conversation data again and again. This can leave the chip waiting instead of calculating.
NVIDIA says its Rubin GPU will use HBM4, which means high-bandwidth memory. The company lists speeds of up to 22 terabytes per second and 288 gigabytes of memory for each GPU.
Peak compute numbers only show the best possible speed. Real speed also depends on the model, prompt length, number of users, and software. Two chips with similar compute power can therefore give very different results.
Memory size matters too. A model runs faster when its data fits close to the GPU. More memory may also support more users, but good scheduling is still needed.
Sources1
The diagram shows computing power growing faster than memory speed during long AI tasks.
Cheaper AI creates more memory demand
Stanford’s AI Index shows that the price of using capable AI models has fallen sharply. Lower prices make new products possible. They also encourage people to use more tokens and run more agents at the same time.
This creates a difficult balance. AI becomes cheaper, but demand grows for memory that is still expensive. Memory size, memory speed, chip packaging, and network links must now work as one system.
Software can reduce some pressure. Quantization stores a model with fewer bits. Speculative decoding tries to create text more efficiently. Better cache management reuses recent data. These methods help, but data still has to move.
Many companies affect the final result. Chip makers, memory suppliers, cloud providers, and software teams all help decide how much useful output a user gets for each dollar.
Sources2
What teams should measure
Buyers should look beyond the highest compute number. Useful measures include tokens per watt, memory speed during busy periods, and how often data must move to slower storage.
Teams should test the work they really plan to run. Training, a short chat, and a long agent task use the system in different ways. Tests should also include busy periods, not only quiet ones.
Product design matters here. Keeping every old message or running many agents may cost too much. A good system decides what to keep, shorten, or find again later.
Sources
Every fact in this story comes from the sources below. Open them to check our work.
- 1
- 2
NVIDIA’s performance numbers are company claims, not tested by Silicon AI News.
Memory speed can slow down the next AI models, because fast chips are left waiting for data, even as AI gets cheaper. How we rate the mood