Distillation vs Fine-Tuning: When to Teach a Small Model Instead of Adapting a Big One

Fine-tuning and distillation get mentioned in the same breath often enough that they blur together. They are not variants of the same thing. They answer different questions, and knowing which question you're actually asking saves a lot of wasted training.

Fine-tuning asks: how do I make this model better at my task?

Distillation asks: how do I get this behaviour out of a smaller model?

What distillation actually transfers

The basic setup involves two models. A larger, more capable one — the teacher — and a smaller one — the student. You run inputs through the teacher, capture what it produces, and train the student to reproduce it.

The part that makes this more than "generate synthetic data" is what you capture. Beyond the final answer, the teacher's output distribution carries information about the answers it nearly gave and how close the call was. A teacher that assigns 70% to one label and 25% to another is expressing something a bare label does not. Training the student against that softer signal transfers more than training it on the winning answer alone.

In practice, a lot of what gets called distillation with language models is the simpler version: generate a large set of teacher outputs, then fine-tune the student on them. That works, and it's what most teams actually do. The distinction is worth knowing because the stricter form can transfer more, and because "distillation" gets used loosely for both.

Why anyone bothers

The motivation is almost always serving economics.

A large model that performs well on your task may be expensive enough to run that the task isn't worth automating. Distillation is the attempt to keep most of the capability while moving to a model small enough to serve cheaply — potentially several times smaller, with the corresponding drop in memory and cost.

The compounding effect is what makes it attractive: a smaller model needs less memory, which may drop you to cheaper hardware, and it generates faster, which improves throughput. Those benefits stack in a way that a marginal quality difference often doesn't outweigh.

The honest trade

Distillation transfers behaviour on the distribution you distilled over. That sentence contains the entire risk.

If you generate teacher outputs across inputs that genuinely represent your production traffic, the student can get remarkably close on that traffic. If your distillation set is narrower than reality, the student will look excellent in evaluation and fail on the cases you didn't think to include — and it will fail without signalling, because a small model doesn't know it's out of its depth.

A large model has broad general capability you may be relying on without realising. A distilled student has whatever you taught it. For a narrow, well-characterised task that's fine and often preferable. For an open-ended one where inputs vary in ways you can't enumerate, it's a real risk.

This is also why distillation rewards effort spent on coverage rather than volume. A hundred thousand teacher outputs over a narrow slice of inputs is worse than a much smaller set that spans the actual variety of your traffic.

How to choose

Fine-tune when the base model is roughly the right size and the gap is task knowledge — your formatting, your domain vocabulary, your house style, your classification scheme. You are adding specificity to a model you can already afford to run. Distil when a bigger model already does the job well and the problem is that running it is too expensive or too slow. You are trying to move capability down a size class, not add new capability. Do both when the natural path is: use a capable model to establish the behaviour, distil into something smaller, then fine-tune the student on real labelled data if you have it. These are sequential steps, not alternatives. Do neither when prompting the smaller model already works. This is more common than it sounds, and it costs one afternoon to find out. Test it before committing to a training pipeline.

What to watch for

Inherited mistakes. The student learns the teacher's errors and biases along with its competence, and it learns them confidently. Distillation is not a filtering step; if the teacher is wrong in a systematic way, you are now serving that wrongness more cheaply. Evaluation against the wrong reference. Compare the student to the teacher on the same held-out set, not to published benchmark numbers. The question is whether the student reproduces the teacher's behaviour on your traffic, and only your evaluation answers that. Long-output degradation. Small models tend to hold up better on short, structured outputs than on long generations, where errors compound. If your task produces long responses, evaluate those specifically rather than trusting an aggregate. Licensing. Whether you may use a model's outputs to train another model is a term-of-use question and it varies. Check before building a pipeline on top of it. Open-weight models with permissive licences avoid this class of problem entirely, which is one of the practical reasons to prefer them.

Why this is only a real option with open weights

Both techniques assume you can actually keep and run the result. If the model you produce lives behind someone else's API and can be deprecated, repriced or restricted, you have optimised a cost you don't control.

That is the underlying reason both fine-tuning and distillation are natural fits for open-weight models: the artifact you end up with is genuinely yours. InfoPlatform.ai is built on that premise — you upload data, pick an open-weight base, and get weights you can export and serve wherever you like. Once you own the weights, decisions like "should this run on a smaller model" are yours to make and yours to benefit from.

FAQ

What's the difference between distillation and fine-tuning?

Fine-tuning adapts a model to your task, usually keeping the same model size. Distillation transfers a larger model's behaviour into a smaller one so you can serve it more cheaply. Fine-tuning adds capability; distillation moves existing capability down a size class. They're complementary rather than competing.

Is distillation just training on synthetic data?

The loose version effectively is — generate teacher outputs and fine-tune the student on them, which is what most teams do. The stricter form also trains against the teacher's output distribution, which carries information about answers it nearly gave and how confident the call was. That softer signal can transfer more than the final answers alone.

When should I distil instead of fine-tune?

When a larger model already performs well on your task and the problem is that running it costs too much or responds too slowly. If the model is the right size but doesn't know your domain, formatting or classification scheme, fine-tuning is the correct tool. And if prompting a smaller model already works, do neither — it's worth an afternoon to check.

What's the main risk with a distilled model?

It learns behaviour on the distribution you distilled over, so a distillation set narrower than your real traffic produces a student that evaluates well and fails silently on inputs you didn't include. Coverage of your input variety matters more than raw volume of teacher outputs.

Does the student inherit the teacher's mistakes?

Yes, including its biases, and it inherits them confidently. Distillation copies behaviour rather than filtering it. Evaluate the student against the teacher on your own held-out set rather than against published benchmarks, and pay particular attention to long outputs, where small models degrade first.