# What is Low-Rank Adaptation?

Also called LoRA.

Low-rank adaptation is a parameter-efficient fine-tuning method that freezes a model's original weights and learns a small pair of matrices whose product forms a correction to selected weight matrices. Because the correction is constrained to a low rank, it holds far fewer trainable values than the weights it modifies. It is the most widely used adapter technique for large models.

The method rests on the hypothesis that the weight change needed to adapt a model to a new task has low intrinsic rank, meaning it can be expressed as the product of two thin matrices. Instead of learning a full update matrix, training learns those two factors. The rank is a hyperparameter that sets adapter capacity and trainable parameter count.

During training only the factor matrices receive gradients, so optimizer memory shrinks dramatically and adaptation becomes feasible on modest hardware. During inference the correction can either be applied alongside the frozen weights or merged into them, producing a standalone model with no additional runtime cost but losing the ability to swap adapters.

Choosing which matrices to adapt matters. Common practice targets attention projections, sometimes extended to the feed-forward layers. A scaling factor controls how strongly the correction is applied. Too low a rank underfits the task; too high a rank erodes the efficiency advantage and increases overfitting risk on small datasets.

Variants address specific pressures. Quantized approaches train adapters on top of a compressed base to cut memory further, at some cost to precision. Multiple adapters can sometimes be combined at serving time, though composition is not guaranteed to behave additively and interactions between adapters trained separately can degrade quality.

## Key points

- Learns two thin matrices whose product corrects frozen weights
- Rank hyperparameter sets adapter capacity and size
- Adapters can be merged into weights or swapped at runtime
- Usually applied to attention projections first
- Quantized variants cut memory further at some precision cost

## In practice

A team adapts a general assistant to write in their documentation style using about four thousand curated examples. They train a low-rank adapter at a modest rank against the frozen base, updating well under one percent of the model's parameters. The resulting adapter file is small enough to version in their artifact store alongside code, and they keep the base and adapter separate so a style revision means shipping a new adapter, not a new model.

## Related terms

- [Parameter-Efficient Fine-Tuning](/en/glossary/parameter-efficient-fine-tuning)
- [Fine-tuning](/en/glossary/fine-tuning)
- [Parameters](/en/glossary/parameters)
- [Quantization](/en/glossary/quantization)
- [Transfer Learning](/en/glossary/transfer-learning)

[Back to the AI Glossary](/en/glossary)