What is Differential Privacy?
Also called DP.
Differential privacy is a mathematical definition of privacy guaranteeing that the output of an analysis is nearly unchanged whether or not any single individual's record was included. It is achieved by adding calibrated random noise to computations, and the strength of the guarantee is expressed by a privacy budget parameter, conventionally written as epsilon, where smaller values mean stronger privacy.
The guarantee is about the algorithm, not the dataset. Because an individual's presence barely changes the result, an observer with arbitrary side information still cannot confidently infer whether that person was in the data. This is a stronger property than the heuristic de-identification approaches it competes with, since it does not depend on assumptions about what auxiliary data an attacker holds.
Two deployment models dominate. In the central model a trusted curator holds raw data and adds noise to released statistics or to model gradients during training. In the local model noise is added on the device before data is ever collected, removing the need to trust a curator but requiring far more data to reach comparable accuracy. Statistical agencies and large platforms have deployed both.
The cost is accuracy, and the accounting is cumulative. Every query against a dataset consumes part of the privacy budget, and once it is exhausted further releases erode the guarantee. Composition theorems track this consumption. Choosing epsilon is a policy decision rather than a technical one, and there is no consensus threshold that counts as sufficiently private across contexts.
For machine learning, differentially private training bounds how much any single training example can influence the resulting model, which directly limits memorization and extraction of training records. The tradeoff is measurable degradation in model quality, larger compute requirements, and difficulty setting parameters for the very large models where memorization concerns are strongest.
Key points
- Guarantees one record barely changes the output
- Noise is calibrated to a privacy budget, usually called epsilon
- Central and local models trade trust against accuracy
- Repeated queries consume the budget cumulatively
- Private training limits memorization at a cost in accuracy
In practice
A product team wants to publish how often each feature is used, without letting anyone infer whether a specific customer used a sensitive feature. Rather than releasing exact counts, the pipeline adds noise drawn from a calibrated distribution to each count and records the budget spent. Published figures remain accurate enough to rank features and track trends, while no single customer's participation can be inferred from the release.