How do small models learn from big ones?

A teacher and a student

Big models are strong, but slow and expensive. Small models are fast and cheap, and some run on a phone. Distillation tries to get the best of both: a small “student” model is trained to imitate a big “teacher” model.

The idea is older than chatbots. Researchers did it in 2006, and in 2015 Geoffrey Hinton and two colleagues gave it the name “distillation”.

More than the right answer

A normal training text only shows which word came next. The teacher gives more: how likely it finds every possible word. These percentages are called soft labels. In the diagram, “coffee” is the word in the text, but the teacher says “tea” was nearly as likely.

Hinton’s team gave an example with pictures. A picture model may give a photo of a BMW a small chance of being a garbage truck, and almost none of being a carrot. Near misses like these teach the student that cars look more like trucks than vegetables.

Where you meet distilled models

Makers use distillation on their own models. Google says Gemini 1.5 Flash was “trained by 1.5 Pro”. The first Gemini Nano models, made for Android phones, were distilled from bigger Gemini models.

A phone model can work offline and keep your words on the device. But a student is usually somewhat weaker than its teacher. Small models fit narrow, frequent or private tasks; for hard questions, keep a big model, and test on your own cases.

Learning from another company’s model can break that company’s rules. Anthropic’s and Google’s terms, for example, forbid using their models to build competing ones.

Another way to shrink a model

A second way to make a model smaller is quantisation. The same model keeps all its numbers but stores each one less exactly, in fewer bits. It is like saving a photo at lower quality, and no second model is trained.

Makers often use both: the first Gemini Nano models were distilled and also stored at 4 bits for phones.

Soft labels

Hinton, Vinyals and Dean (2015) trained a small model on a big model’s probabilities for every answer, which they called soft targets. These show which wrong answers are near misses, which a plain label cannot.

In their tests, a small network for handwritten digits made 146 errors alone and 74 with soft targets.

Two ways to distil

Makers with the teacher’s probabilities can train on them directly. Google’s Gemma 2 2B and 9B learned this way instead of by plain next-token prediction. Meta cut Llama 3.2 1B and 3B down from the 8B model, then trained them on the 8B and 70B models’ outputs.

Others train on answers the teacher wrote. DeepSeek fine-tuned six Qwen2.5 and Llama models, from 1.5 to 70 billion parameters, on about 800,000 samples made with R1. R1’s MIT licence explicitly allows this.

How close do students get?

DistilBERT, in 2019, was 40% smaller and 60% faster than BERT, an early Google language model, and kept 97% of its language understanding in the authors’ tests. On the AIME 2024 maths contest, DeepSeek’s distilled 32-billion model scored 72.6%, against 79.8% for R1 itself.

Learning from the teacher can beat training the student alone: the same 32-billion base scored 47.0% when DeepSeek trained it with reinforcement learning directly.

Small models fit narrow, high-volume, fast or private tasks. Keep a large model for hard reasoning, and test the small one on your own cases.

Distillation, quantisation and the rules

Quantisation is a different trick: the same model stores its numbers with fewer bits. 8-bit weights halve the memory of 16-bit ones.

Using another company’s model as a teacher can breach its terms: Anthropic’s and Google’s forbid building competing models. In February 2026, Anthropic accused DeepSeek, Moonshot AI and MiniMax of over 16 million exchanges through about 24,000 fake accounts.

Anthropic still calls distillation itself a legitimate and widely used method. The problem it describes is using a rival’s model against its terms.

Probabilities and temperature

Buciluă, Caruana and Niculescu-Mizil (2006) compressed a large ensemble into one small network by training it on the ensemble’s predictions. Hinton, Vinyals and Dean (2015) generalised this as distillation, with a temperature that softens the teacher’s probabilities during training.

Soft targets carry a lot of information. With every “3” removed from its training set, their distilled digit model still recognised 98.6% of test 3s after one bias fix. A distilled speech model nearly matched a 10-model ensemble, at 60.8% against 61.1% frame accuracy.

Sequence-level and reasoning distillation

Without the teacher’s probabilities, a student can be fine-tuned on text the teacher wrote. Distilling step-by-step (2023) also used the teacher’s written reasons: a 770-million T5 beat few-shot 540-billion PaLM on one benchmark, using 80% of the data.

DeepSeek’s R1-Distill models, from 1.5 to 70 billion parameters, were fine-tuned on about 800,000 R1 samples without reinforcement learning. The s1 study fine-tuned Qwen2.5-32B-Instruct on 1,000 traces from Gemini 2.0 Flash Thinking in 26 minutes on 16 H100 chips.

Students usually trail their teachers, as R1’s distilled models do, so test a small model on your own cases before it replaces a large one.

Makers distilling their own models

The first Gemini Nano models (2023), at 1.8 and 3.25 billion parameters, were distilled from larger Gemini models and stored at 4 bits for devices. Gemma 2 and Gemma 3 use distillation in pre-training, and Qwen3’s small models draw on its flagship models.

Meta pruned Llama 3.2 1B and 3B from Llama 3.1 8B, then trained them on the 8B and 70B models’ outputs. Llama 4 Maverick was distilled from Llama 4 Behemoth, a teacher Meta had not released.

Whether GPT-6 Luna, Claude Haiku 5.5 or Gemini Flash-Lite are distilled is not published.

Not quantisation

Quantisation lowers numerical precision instead of training a new model. LLM.int8() (2022) halved memory with 8-bit weights, with no loss reported up to 175 billion parameters; GPTQ ran a 175-billion model on one GPU at 3 to 4 bits.

The two combine: the first Gemini Nano models were both distilled and stored at 4 bits. Apple trained its on-device model of about 3 billion parameters to work at 2 bits per weight.

Terms, copyright and accusations

Anthropic’s commercial terms bar using its services to train competing AI models, and Google’s Gemini API terms bar building competing models or extracting weights. These are contracts with the user.

Copyright is a separate question: the US Copyright Office says AI output is protected only where a human determined enough of its expression. Even unprotected output can still be covered by the terms under which it was obtained.

In February 2026, Anthropic accused DeepSeek, Moonshot AI and MiniMax of distilling through about 24,000 fake accounts. In September 2026, as our news reported, US agencies accused six China-based firms of “malicious distillation”, and China rejected the claims. These are accusations, not court findings.