What Is Model Distillation (and Why Small Models Are Getting Good)

Admin


The most surprising trend in AI over the past year is not that the biggest models got bigger — it is that small models got dramatically better. Lightweight models now handle coding subtasks, document parsing, tool calls, and on-device assistance that used to require a frontier model. Much of that progress comes from a technique called model distillation: training a small "student" model to imitate a large "teacher" model, so the student inherits much of the teacher's skill at a fraction of the size and cost. This guide explains how distillation works, why it is effective, and what it means for anyone choosing or building with AI models.

How Distillation Actually Works

Training a model the ordinary way means showing it the right answers: this sentence follows that one, this label goes with that image. Distillation adds a second signal — the teacher's opinion about every possible answer. When a large teacher model looks at a question, it does not just produce one answer; it assigns probabilities across many plausible ones. That distribution carries information: it shows which answers are nearly right, which are plausible but wrong, and which are nonsense. A student trained on those "soft" probabilities learns far more from each example than a student trained on a single hard answer, because it learns the teacher's judgement, not just its conclusions.

The classic demonstration is DistilBERT, published in 2019. The researchers distilled BERT — then a large language model — into a model about 40% smaller that, according to their paper, retained roughly 97% of BERT's language understanding performance while running about 60% faster. The recipe has barely changed since: take a capable teacher, generate a large set of its outputs on real tasks, and train the student to match them. Modern versions add refinements — training on the teacher's reasoning steps rather than just its final answers, filtering out the teacher's mistakes, and mixing distilled data with ordinary training data — but the core idea is the same: knowledge can be compressed, not just created.

There is a catch, and it matters. A student can never exceed its teacher on the things it was distilled from, and it inherits the teacher's blind spots along with its strengths. If the teacher is confidently wrong about a topic, the student learns to be confidently wrong too. Distillation also works best when the student is trained on the specific kinds of tasks it will actually perform. A distilled generalist is usually weaker than its teacher; a distilled specialist, trained on one domain, can match or even feel better than the teacher within that domain — because it has seen far more examples of exactly the work it needs to do.

Why Small Models Are Closing the Gap

Distillation alone does not explain the small-model surge. It works now because three things arrived together. First, the teachers got much better: a student distilled from a 2026 frontier model inherits far more capability than one distilled from a 2023 model. Second, training data improved — carefully curated, deduplicated, and filtered datasets mean small models waste less of their limited capacity on noise. Third, architectures designed for efficiency, such as mixture-of-experts designs that activate only a fraction of their parameters per token, let a compact model punch above its size.

The result is an economic shift as much as a technical one. A small model costs a fraction of a frontier model to run — sometimes a tenth or less per million tokens — and it can run on a phone, a laptop, or a single server instead of a data centre. That makes it the right tool for high-volume, repetitive work: routing requests, extracting fields from documents, calling tools, drafting first versions. The frontier model still wins on genuinely hard reasoning, but most daily AI work is not genuinely hard reasoning. It is the same task repeated thousands of times, where cost and speed matter more than squeezing out the last few percent of accuracy.

Why it matters

If you are choosing a model for a product, the question is no longer "which model is smartest?" but "which model is smart enough for this task, at this price, at this speed?" Distillation is what makes that question interesting: it has turned model selection into a trade-off instead of a race to the biggest option. For builders, it means the expensive frontier model becomes a teacher — used to generate training data and handle the hardest cases — while a fleet of small distilled models does the everyday work cheaply and quickly. For users, it means capable AI arriving on devices, offline, without a data-centre round trip. The frontier will keep advancing, but the most useful AI of the next few years may be the small models that learned from it.

A VCTRIA evergreen guide.

Post a Comment