Quantization After Fine-Tuning: Cutting the Serving Bill Without Losing the Model You Trained

Most fine-tuning advice stops at the moment training finishes. But training is a one-time cost, and serving is the bill that arrives every month for as long as the model is in production. For a model that handles real traffic, serving usually costs more over its life than training it ever did.

Quantization is the main lever on that bill, and it's underused by teams who fine-tune successfully and then serve the result at full precision because nobody told them there was a choice.

What quantization actually is

A model's weights are numbers, and those numbers are stored at some precision. Training typically works in 16-bit floating point, meaning every one of the model's billions of parameters occupies two bytes.

Quantization stores them at lower precision instead: 8-bit integers, or 4-bit. The arithmetic is the obvious part — going from 16-bit to 8-bit roughly halves the memory the weights occupy, and going to 4-bit roughly quarters it.

That sounds like a storage optimization. It isn't, really. It's a cost optimization, because of what GPU memory does to your deployment options.

Why memory is the thing that costs money

Serving an LLM requires the weights to sit in GPU memory, and GPU memory comes in fixed steps tied to specific hardware. You don't rent memory by the gigabyte; you rent a card that has a certain amount.

So the practical effect of quantization isn't a smooth cost curve, it's a set of thresholds. A model that doesn't fit on one card needs two, with all the complexity that brings. A model that just fits on a smaller, cheaper card costs a fraction of one that needs the tier above. Halving the memory footprint frequently moves you down a hardware tier, and that's where the savings actually come from.

There's a second effect worth knowing: at small batch sizes, generation is often limited by how fast weights can be read from memory rather than by raw compute. Smaller weights mean less to read per token, which can improve throughput as well as cost. This is not guaranteed and depends heavily on your serving stack, but it's a real secondary benefit rather than a marketing claim.

What you trade

Quantization is lossy. You are approximating the numbers, and approximation can change outputs.

The honest summary of where the trade sits:

8-bit is usually close to free. For most tasks the quality difference between 16-bit and 8-bit is small enough to be hard to detect without careful measurement. If you want a default, this is it. 4-bit is a real trade, sometimes worth it. Modern 4-bit methods are far better than naive rounding, but degradation becomes measurable more often, and it doesn't distribute evenly. Long outputs, multi-step reasoning, and precise formatting tend to suffer before short classification-style answers do. Below 4-bit is specialist territory. Possible, but not somewhere to go without a strong reason and thorough evaluation.

The part that matters more than the numbers: degradation is task-specific. A quantization that's invisible on one workload can be obvious on another. Which means general benchmarks tell you less than you'd like, and your own evaluation tells you almost everything.

Where quantization fits with LoRA

If you fine-tuned with LoRA, you have a small adapter and a large frozen base. That structure interacts with quantization in a way worth understanding.

The usual approach is to merge the adapter into the base model first, then quantize the merged result. This gives you a single artifact to serve and avoids surprises from quantizing a base and adapter separately.

It's also possible to train against an already-quantized base, which is what QLoRA does, and it's how fine-tuning large models on modest hardware became feasible. Note that this is a training-time memory optimization; it doesn't by itself decide how you serve the result, and those two decisions can be made independently.

How to tell whether it cost you anything

This is the step that gets skipped, and skipping it is how teams end up with a quietly worse model in production and no idea when it happened.

The test is not a general benchmark score. It's your task, on your data:

1. Freeze an evaluation set before you quantize. A few hundred representative examples with known-good outputs. If you don't have one, build it — you need it for far more than this.
2. Run the full-precision model against it and record the results. This is your baseline. Without it you have nothing to compare to and you'll end up judging by vibes.
3. Run the quantized model against the same set and compare on whatever actually matters for your task: exact match, format validity, a rubric score, human review of a sample.
4. Look at the failures specifically, not just the aggregate. An average that barely moved can hide a specific mode that broke — a format that no longer parses, a long response that now trails off. Aggregates are good at hiding exactly the failures you'd care about most.
5. Check latency and memory too, so you know what you actually bought.

If 4-bit shows real degradation, step back to 8-bit before concluding quantization doesn't work for you. The middle option is frequently the right one and it's often skipped in the jump from "full precision" to "smallest possible".

When not to bother

Quantization is not free effort, and it isn't always worth it.

If your traffic is low and intermittent, serving cost may be small enough that engineering time is better spent elsewhere. If your task is unusually sensitive to precise output formatting, the evaluation burden may outweigh the saving. And if your model already fits comfortably on the hardware you're using with room to spare, halving the footprint may not move you down a tier at all — in which case you've done work for no change in the bill.

The question worth asking first is simple: does a smaller footprint actually change which hardware I need? If the answer is no, the saving is theoretical.

The short version

Treat serving cost as a first-class part of a fine-tuning project rather than an afterthought. Quantize by default, start at 8-bit, evaluate on your own task with a frozen set and a recorded baseline, and only go to 4-bit if the numbers justify it and your evaluation says quality held.

Owning your model is the point of fine-tuning an open-weight base, and part of owning it is controlling what it costs to run. InfoPlatform.ai hands you weights you can export and serve wherever you like, which means these decisions are genuinely yours to make — including running the same model more cheaply than the platform that trained it.

FAQ

What is quantization in machine learning?

Quantization stores a model's weights at lower numeric precision — 8-bit or 4-bit integers instead of 16-bit floating point. Going from 16-bit to 8-bit roughly halves the memory the weights occupy and 4-bit roughly quarters it. The practical benefit is that a smaller footprint often lets you serve the model on cheaper hardware.

Does quantization make a fine-tuned model worse?

It can, and the effect is task-specific. 8-bit is usually close to lossless for most workloads. 4-bit is a real trade that shows up more often, and it lands unevenly — long outputs, multi-step reasoning and strict formatting tend to degrade before short classification answers do. General benchmarks won't tell you; you need to evaluate on your own task.

Should I quantize before or after merging a LoRA adapter?

The usual approach is to merge the adapter into the base model first and quantize the merged result, so you have a single artifact to serve. Training against an already-quantized base (QLoRA) is a separate, training-time memory optimization and doesn't dictate how you serve the finished model.

How do I test whether quantization hurt my model?

Freeze an evaluation set of representative examples before you quantize, run the full-precision model against it to record a baseline, then run the quantized model on the same set. Compare on what matters for your task and inspect individual failures rather than only the aggregate — an average that barely moves can hide one badly broken output mode.

Is quantization always worth doing?

No. If your traffic is low, if your task is highly sensitive to exact formatting, or if the smaller footprint doesn't move you to cheaper hardware, the saving may not justify the effort. The first question to ask is whether halving the memory actually changes which GPU you need — if it doesn't, the benefit is theoretical.