After this explainer you canDescribe the two stages of training, say where training data comes from and why it is disputed, and name what laws ask makers to publish.

The short answerA model learns by reading huge amounts of text and practising guessing the next token, a word or part of a word. Then people teach it to be helpful and safe.

In simple words

  • First it practises on huge amounts of text, pictures or code.
  • Then people show it good answers and rate its replies.
  • Training runs on thousands of computer chips and costs a lot of money.
  • Some writers and publishers say their work was used without permission.
From raw data to a released modelIllustration + live data
  1. 1DataWeb pages, books, code, licensed archives
  2. 2Pre-trainingGuess the next token, trillions of times, for weeks on thousands of chipserrors fall
  3. 3Fine-tuningGood examples, people’s ratings, and rewards for checkable answers
  4. 4ReleaseTested, then offered in an app, through an API or as a download

Training compute, in operations · compute ≈ 6 × parameters × training tokens (dense models)

  • EU, 1025 Above this, a general-purpose model is presumed to have “systemic risk”, the EU’s term for the most powerful models.In force
  • California, Connecticut, New York, 1026 Above this, a model counts as a frontier model, one of the most powerful (explainer 5).CA in forceCT in forceNY from Jan 2027
  • Llama 3 405B, 3.8 × 1025 Meta’s largest Llama 3 model, for scale. Above 1026, both lines apply.

The lines come from the laws in our AI rules checker. Each step on the scale is ten times more computing.

The curve shows the idea, not a real run: the model’s errors, called the loss, fall as training goes on.

Words to know

Training data
The text, pictures and code a model learns from.
Pre-training
The first, long stage: the model practises guessing the next token.
Fine-tuning
The second stage: people teach the model to be helpful and safe.

Step 1: practice (pre-training)

During training, the model guesses the next token a huge number of times and checks itself against the real text. Each wrong guess changes the model a little, so it does better next time.

This first stage is called pre-training. It takes weeks or months on thousands of powerful computer chips.

Step 2: learning to help (fine-tuning)

The second stage is called fine-tuning. People show the model examples of good answers and rate its replies.

This teaches it to follow instructions and to say no to harmful requests. It does not always work, so a model can still make mistakes or say harmful things.

Whose data?

The data comes from the web, books, code and other sources. Some newspapers and artists have gone to court over this, and some AI companies now pay for data.

New laws also ask AI makers to publish a summary of the data they used.

Try it yourself

Give a chat app a clear instruction, such as “Answer in exactly three bullet points.” Following instructions like this is what fine-tuning taught it.

Check yourself3 quick questions
  1. During training, what happens after the model makes a wrong guess?

  2. A model has finished its first stage of training. It knows a lot, but it often ignores instructions. What comes next?

  3. A photographer learns that their photos were used to train a model without permission. What does the explainer say about cases like this?

Sources

This explainer was generated by AI systems under the editorial rules of Silicon AI News and checked against the sources it lists. The live parts come from our checked stories, trackers, model comparison and rules checker.