# What is Knowledge Distillation?

Also called model distillation, distillation.

Knowledge distillation is a training technique in which a smaller student model learns to reproduce the behavior of a larger teacher model. The student trains on the teacher's outputs, which may be full probability distributions, generated text, or intermediate representations, rather than only on ground-truth labels. The goal is a compact model that retains much of the teacher's capability.

Teacher outputs carry more information than hard labels. A distribution that assigns some probability to several plausible continuations tells the student which alternatives are near misses, information a single correct answer omits. This richer signal, sometimes called dark knowledge, is why distilled students often outperform equally sized models trained on the original labels alone.

Practical variants differ in what is matched. Response distillation trains on teacher-generated text and is straightforward when only sampled outputs are available. Logit distillation matches the full output distribution and requires deeper access. Feature distillation aligns internal representations and can transfer more structure, but requires compatible architectures between teacher and student.

Distillation is a primary route to small deployable models. A compact student can serve high-volume, latency-sensitive traffic at a fraction of the teacher's compute, which matters when usage is metered per token. It pairs naturally with quantization and pruning in a compression pipeline, though each stage compounds the quality loss.

Limits and caveats apply. Students inherit teacher errors and biases along with capability, and typically fall further behind on tasks demanding long reasoning chains or rare knowledge. Distilling from a third party's model may also breach that provider's terms of service, so provenance of the training signal is a legal question as much as a technical one.

## Key points

- Small student learns to imitate a larger teacher model
- Teacher distributions carry more signal than hard labels
- Response, logit, and feature distillation differ in access needed
- Common route to fast, cheap deployable models
- Students inherit teacher errors and may raise licensing questions

## In practice

A company runs a large model for a classification-heavy internal workflow and finds the per-token cost dominated by volume rather than difficulty. They collect two hundred thousand teacher responses on real historical inputs, train a much smaller student on them, and evaluate both against a held-out labeled set. The student matches the teacher within a couple of points on their benchmark and serves the workflow at a small fraction of the compute.

## Related terms

- [Small Language Model](/en/glossary/small-language-model)
- [Quantization](/en/glossary/quantization)
- [Fine-tuning](/en/glossary/fine-tuning)
- [Inference](/en/glossary/inference)
- [Benchmark](/en/glossary/benchmark)

[Back to the AI Glossary](/en/glossary)