Teaching a small model to copy a big one

orig. “Distilling the Knowledge in a Neural Network” · Geoffrey Hinton, Oriol Vinyals, Jeff Dean

Efficient AI Intermediate 3 min read Written, reviewed by Marginalia Editorial

In the margin

Heads up

Big models are accurate but slow. This paper shows how to pour most of that skill into a small, fast model.

What's going on

The trick is to train a small student model to copy the outputs of a large teacher model, not just the right answers but how confident the teacher is across all the options. Those soft signals carry extra hints that help the student learn more than it could from the labels alone. The result is a smaller model that runs faster and cheaper while keeping much of the accuracy. This is called distillation.

Why it matters

Distillation is a big reason capable AI can run on phones and in browsers. As models get larger, shrinking them down without losing much skill becomes more valuable. It is one of the core tools for making AI practical and affordable.

Source

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, Google

View on arXiv ↗ Read the PDF ↗

Still fuzzy on a sentence?

Paste it and we'll explain it even more simply.

Test your understanding

Member notes Sign in ↗

ME Marginalia Editorial TEAM

The editorial team picked this paper because it is worth your time, and an AI drafted this note. We check each one and rewrite it when the draft misses the mark.