After this explainer you canExplain what happens between your question and the answer, why every answer has a running cost, and compare what different models cost.
The short answerWhen you ask a question, it travels over the internet to a data centre, a building full of computers. Special chips run the model there and send the answer back in seconds.
In simple words
- Using a trained model to get an answer is called inference.
- It runs on special chips in big data centres.
- Businesses that use AI models pay for every token, in the question and in the answer.
- Data centres use a lot of electricity.
The short answerEvery request runs the trained model on chips in a data centre to produce an answer. This step is called inference: training happens once, inference happens every time someone uses the model.
In simple words
- Inference means using a trained model to answer a request.
- It runs on chips such as GPUs, in large data centres.
- Businesses pay per token, so speed and price matter.
- Data centres need a lot of electricity, which keeps them in the news.
The short answerServing a model splits into a parallel prefill over the prompt and a sequential decode that emits one token per step from a key-value cache. Memory bandwidth, batching and quantisation set the cost per token.
Key points
- Prefill reads the whole prompt in parallel; decode writes one token per step.
- Decode reuses a key-value cache and is usually limited by memory bandwidth.
- Batching, quantisation and speculative decoding cut the cost or the time per token.
- Output tokens usually cost more than input tokens, and hidden reasoning tokens are billed as output.
Cost of 1,000 such answers
List prices per million tokens from each company’s pricing page, as in our model comparison of November 16, 2026. Output tokens cost more than input, so long answers cost more quickly. Reasoning models also bill their hidden thinking as output. As a guide, Google counts 100 tokens as about 60 to 80 English words.
Words to know
- Prompt
- What you type or send to a model.
- Data centre
- A building full of computers that run models.
- Inference
- Using a trained model to get an answer.
Your question takes a trip
Your question, also called a prompt, goes over the internet to a data centre, a building full of computers. The model runs there and sends back the answer, piece by piece. That is why answers often appear word by word on your screen.
The model was trained once, before you used it. Running it to answer you is called inference, and it happens every time anyone asks anything.
Some small models can also run on a phone or laptop, so your words stay on your device.
Paying by the piece
AI companies charge businesses and developers by tokens, small pieces of text. Longer questions and longer answers cost more, and some models cost far more than others.
Chat apps usually charge people a monthly fee instead, or nothing.
Data centres need a lot of electricity
The chips that run models need a lot of electricity and cooling. That is why power deals, chips and new data centres appear in AI news so often.
Try it yourself
In the cost calculator above, set the answer to 4,000 tokens. Which models cost at least ten times more than the cheapest one?
Check yourself3 quick questions
Why does a chat app’s answer often appear on your screen word by word?
Two companies use the same AI model and pay for each token. Which one pays more for one answer?
A tech company plans a huge new data centre. Why does it also make a deal to buy electricity?
Words to know
- Prompt
- What you type or send to a model.
- Data centre
- A building full of computers that run models.
- Inference
- Using a trained model to get an answer.
Training once, inference every time
A model is trained once, at great cost. After that, every question, image or agent step needs the model to run again. That is inference.
Because it happens for every user and every step, the cost of inference decides how cheap, fast and widely used AI can be.
Tokens and prices
AI services measure work in tokens. For Google’s Gemini models, 100 tokens are about 60 to 80 English words.
Businesses often use a model through an API, a connection that lets their own software use it. They pay for the tokens they send in and the tokens the model writes back, and prices differ a lot between models.
Reasoning models also think before they answer, and those hidden thinking tokens are billed as output too. Our model comparison lists prices next to independent test results, so you can see what you get for the money.
Chips and data centres
Models run on specialised chips: Nvidia’s GPUs (graphics processing units), Google’s TPUs (tensor processing units) and a growing number of custom chips. Thousands of them sit in data centres that need a lot of power and cooling.
That is why chip export rules, power deals and new data centres appear in the news so often.
Labels on what comes out
Some laws cover the output itself. In the EU, AI that creates text, images, video or sound must mark its output in a way computers can detect. California requires hidden data in AI-made images, video and audio that shows where they came from.
Try it yourself
In the cost calculator above, set the answer to 4,000 tokens. Which models cost at least ten times more than the cheapest one?
Check yourself3 quick questions
A model was trained last year. Today, a million people ask it questions. Which step runs a million times today?
You send the same questions to two models. Model A’s price per token is ten times Model B’s, and the answers are the same length. What happens to your bill?
Why do chip export rules and power deals keep appearing in AI news?
Prefill and decode
Inference has two phases. Prefill processes the whole prompt in parallel and fills a key-value (KV) cache. Decode then generates one token per step, reading that cache instead of recomputing the past.
Prefill is limited mainly by raw computing. Decode is usually limited by memory bandwidth, because every step reads the weights and the KV cache to produce one token per request.
Prefill time largely sets how long you wait for the first token; decode speed sets how fast the rest streams. Artificial Analysis measures both for many models.
Serving tricks
Providers batch many users’ requests on the same chips to keep them busy, and manage the KV cache carefully, as in the vLLM project’s PagedAttention method.
Quantisation stores weights in fewer bits, such as 8 or 4, to save memory and cost, sometimes at a small loss in quality. Mixture-of-experts models use only part of their parameters for each token, but all of them must sit in memory.
Speculative decoding lets a small draft model propose several tokens, which the large model then checks in one pass.
The economics of a token
Prices are quoted per million tokens, and output tokens usually cost several times more than input tokens. Worked example, with made-up prices: input costs $3 and output $15 per million tokens. A 2,000-token prompt with a 500-token answer then costs $0.006 + $0.0075 = $0.0135.
Reasoning models spend extra output tokens thinking before they answer, and those tokens are billed as output. Effort settings trade quality for cost and waiting time, which is why our comparison names the effort level of each model.
Rules on outputs
Providers of generative AI must mark outputs as AI-made in a machine-readable way under AI Act Article 50(2), with systems already on the market given until December 2, 2026. California’s AI Transparency Act requires hidden provenance data and a free verification tool.
Try it yourself
In the cost calculator above, set the answer to 4,000 tokens. Which models cost at least ten times more than the cheapest one?
Check yourself3 quick questions
A user pastes a 50,000-token contract and asks for a 200-token summary. Which phase handles the contract, and how?
A provider moves a model from 8-bit to 4-bit weights. What is the usual trade-off?
Prefill handles input tokens in parallel. How does decode make output tokens?
Sources
- Independent tests of model quality, speed and price (Artificial Analysis)
- Understand and count tokens (Gemini API documentation)
- Plans and pricing (Claude)
- Extended thinking: thinking tokens count as billed output (Claude documentation)
- AI model release tracker: small models for on-device use (Silicon AI News)
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon and others, 2023, arXiv)
- Fast Inference from Transformers via Speculative Decoding (Leviathan and others, 2022, arXiv)
- Mixtral of Experts (Jiang and others, 2024, arXiv)
- AI Act, Article 50 (EU AI Act Service Desk)
This explainer was generated by AI systems under the editorial rules of Silicon AI News and checked against the sources it lists. The live parts come from our checked stories, trackers, model comparison and rules checker.