What is Transfer Learning?
Transfer learning is the practice of reusing a model trained on one task or dataset as the starting point for a different, usually narrower task. Rather than learning from random initialization, the new task inherits representations already learned from broad data. It is the general principle that makes the pre-train then adapt pipeline possible.
The underlying observation is that low-level and mid-level representations are broadly reusable. A model that learned edges, textures, syntax, or discourse structure on general data has already solved much of the perceptual or linguistic groundwork any related task requires. Only the task-specific mapping on top needs to be learned, which requires far less data and compute.
Several adaptation strategies exist along a spectrum. Feature extraction freezes the original weights and trains only a small head on top. Full fine-tuning updates every weight. Parameter-efficient methods sit between, updating a small set of added or selected parameters. The right choice depends on how much task data exists and how far the target domain sits from the source.
Transfer is not always beneficial. When the source and target distributions differ enough, inherited representations can mislead, a phenomenon called negative transfer. Aggressive fine-tuning on a narrow dataset can also erase general capability, sometimes described as catastrophic forgetting, leaving a model that performs well on the target task and poorly on everything else.
In current language work, transfer learning is so standard that it is often invisible. Pre-training is the transfer source, and instruction tuning, preference training, and domain fine-tuning are successive transfer steps. Prompting can be viewed as a form of adaptation without weight changes, exploiting transferred capability entirely at inference time.
Key points
- Reuses representations learned on one task for another
- Cuts the data and compute a new task needs
- Strategies range from frozen features to full fine-tuning
- Mismatched domains can cause negative transfer
- Narrow fine-tuning risks eroding general capability
In practice
A team needs a classifier that sorts maintenance reports into six equipment categories and has only nine hundred labeled examples. Training a language model from scratch on that data is hopeless. Instead they start from a general pre-trained model, freeze most layers, and train a small classification head. The general model already represents technical vocabulary and sentence structure, so the nine hundred examples only need to teach the category boundaries.