What is Parameter-Efficient Fine-Tuning?
Also called PEFT.
Parameter-efficient fine-tuning is a family of adaptation methods that adjust a small number of parameters while leaving most of a pre-trained model's weights frozen. It achieves much of the benefit of full fine-tuning at a fraction of the memory and storage cost. Common approaches include low-rank adapters, prefix tuning, and bias-only updates.
Full fine-tuning updates every weight, which requires storing gradients and optimizer state for the entire model and produces a complete new copy per task. For large models this is expensive in memory and unwieldy in operations, since each variant must be stored and served separately. Parameter-efficient methods reduce trainable parameters by orders of magnitude while keeping the frozen base shared.
The methods differ in where they intervene. Adapter approaches insert small trainable modules between existing layers. Low-rank methods add a compact learned correction to selected weight matrices. Prompt and prefix tuning learn continuous vectors prepended to the input rather than touching weights at all. Selective methods train only a chosen subset, such as bias terms or normalization parameters.
The operational payoff is multi-tenancy. Because the base model is untouched, many task-specific adapters can be stored cheaply and swapped or even composed at serving time against a single loaded base. This turns per-customer or per-domain specialization from a heavyweight infrastructure decision into a manageable one.
Limitations are real. When a target task requires capability the base model genuinely lacks, a small correction cannot supply it, and full fine-tuning or further pre-training performs better. Results are also sensitive to which layers are adapted and to the capacity allocated, so the reported gap with full fine-tuning varies considerably by task and configuration.
Key points
- Trains a small fraction of parameters, freezes the rest
- Includes adapters, low-rank updates, and prefix tuning
- Many task adapters can share one loaded base model
- Far lower memory and storage cost than full fine-tuning
- Cannot add capability the base model wholly lacks
In practice
A vendor serves eleven industry-specific variants of one assistant. Full fine-tuning would mean eleven complete model copies in memory. Instead they train eleven small adapters against a single frozen base, each a tiny fraction of the base size. At request time the router loads the base once and applies the adapter matching the customer's industry, so all eleven variants run on the hardware footprint of roughly one model.