After this explainer you canDescribe the two stages of training, say where training data comes from and why it is disputed, and name what laws ask makers to publish.

The short answerDevelopers show a model huge amounts of data and let it adjust itself until its predictions improve. Then they fine-tune it with examples, people’s ratings and rewards for correct answers, so it follows instructions.

In simple words

  • Pre-training: the model learns from huge amounts of text, images or code.
  • Fine-tuning: examples, ratings and rewards teach it to follow instructions and refuse harmful requests.
  • Laws use the amount of training computing as a rough sign of how capable a model is.
  • Where the training data comes from is a growing legal fight.
From raw data to a released modelIllustration + live data
  1. 1DataWeb pages, books, code, licensed archives
  2. 2Pre-trainingGuess the next token, trillions of times, for weeks on thousands of chipserrors fall
  3. 3Fine-tuningGood examples, people’s ratings, and rewards for checkable answers
  4. 4ReleaseTested, then offered in an app, through an API or as a download

Training compute, in operations · compute ≈ 6 × parameters × training tokens (dense models)

  • EU, 1025 Above this, a general-purpose model is presumed to have “systemic risk”, the EU’s term for the most powerful models.In force
  • California, Connecticut, New York, 1026 Above this, a model counts as a frontier model, one of the most powerful (explainer 5).CA in forceCT in forceNY from Jan 2027
  • Llama 3 405B, 3.8 × 1025 Meta’s largest Llama 3 model, for scale. Above 1026, both lines apply.

The lines come from the laws in our AI rules checker. Each step on the scale is ten times more computing.

The curve shows the idea, not a real run: the model’s errors, called the loss, fall as training goes on.

Words to know

Training data
The text, pictures and code a model learns from.
Pre-training
The first, long stage: the model practises guessing the next token.
Fine-tuning
The second stage: people teach the model to be helpful and safe.

Step 1: pre-training

In pre-training, the model reads a vast collection of text, code or images. It keeps guessing the next or missing piece, checks the real one, and changes its parameters a little to do better next time.

This takes weeks or months on thousands of chips, and it has long been the most expensive step. The amount of computing is counted in operations: simple calculations, such as adding or multiplying two numbers. Epoch AI tracks this number for well-known models.

Step 2: fine-tuning

A pre-trained model knows a lot but is not yet a helpful assistant. Developers train it further on examples of good answers, and on people’s ratings of its answers.

This step is meant to teach it to follow instructions, refuse harmful requests and say when it is unsure. It does not always succeed: OpenAI found that GPT-4 became worse at judging its own certainty after this step.

Newer reasoning models also practise on problems whose answers a computer can check, such as maths and code. They are rewarded when they get them right, as DeepSeek described for its R1 model.

Where the data comes from

Training data includes public web pages, books, code, licensed archives and data the developer creates. Publishers and artists have sued over the use of their work, and some now sell licences instead.

Training a model on another model’s answers is called distillation. Some AI companies’ terms, such as Anthropic’s, forbid using their services to train a rival model. Labs accuse each other of distillation, and it is one of the stories we follow.

What the law asks

In the EU, makers of general-purpose models must publish a summary of their training content and respect publishers’ opt-outs. California asks developers to post a summary of their training data.

Laws also use training compute to find the most powerful models. The EU line is 10²⁵ operations, which means 1 followed by 25 zeros. California, New York and Connecticut use 10²⁶, ten times more.

Try it yourself

Give a chat app a clear instruction, such as “Answer in exactly three bullet points.” Following instructions like this is what fine-tuning taught it.

Check yourself3 quick questions
  1. Which step teaches a model most of what it knows?

  2. Company A trains its small model on thousands of answers from Company B’s model. What is this called?

  3. An AI company releases a general-purpose model in the EU. What must it publish about training?

Sources

This explainer was generated by AI systems under the editorial rules of Silicon AI News and checked against the sources it lists. The live parts come from our checked stories, trackers, model comparison and rules checker.