Distillation
Distillation is a training technique where a small 'student' model learns to reproduce the answers of a large, accurate 'teacher' model, so you can deploy the small one and keep most of the accuracy at a fraction of the running cost.
No jargon, no math
In plain words, what is it?
You want the quality of a huge model and the bill of a small one. Distillation gets you both. Take a large teacher model and a smaller student. Feed both the same questions, often millions of unlabeled ones. The teacher's answers, including the probabilities behind them, become the training targets for the student. The student is not just memorizing right answers: it absorbs the teacher's sense of what is almost right, which is why a student can beat a model of the same size trained on raw data alone. You repeat until the student matches the teacher closely enough on your test set, then you ship the student and switch the teacher off. The failure mode: if the teacher is wrong about something, the student learns the error faithfully. Mistakes copy.
- 01
Practice. A batch of practice questions goes in, and the student model answers each one.
- 02
Targets. The teacher model's answers, with their probabilities, become the training targets.
- 03
Compare. A check scores how close the student's answers are to the teacher's.
- 04
Ship. Close enough: the small student is deployed and the teacher is retired.
- 05
Retry. Not close enough: the student trains again against the same targets.
Terms that live nearby.
Large Language Model
A large language model (LLM) is an AI system trained on massive collections of text to predict the next token, which gives it a general ability to write, summarise, translate and reason without being programmed for each task.
ReadFine-tuning
Fine-tuning is additional training that teaches a general model your specific patterns: you show it hundreds or thousands of examples of your tasks, your tone and your formats, and it adjusts its internal weights until that behaviour becomes its default.
ReadInference Optimization
Inference optimization is the engineering discipline of making a trained model cheaper and faster to run: measure where time and compute go on each request, then apply levers like caching, quantization, batching and prompt trimming to cut both.
Read§ 04
Want this running in your stack?
We design and ship AI systems with permissions, logs and human approval. It starts with a 30-minute scoping call.