What is Alerting?
Also called alerts, paging.
Alerting is the practice of notifying a responsible person or system when monitored signals indicate a condition that requires attention. An alert defines a condition, a severity, a destination, and ideally a documented response. The central design problem is precision: alerts that fire without a required response train recipients to ignore them, which is how genuine incidents get missed.
A useful alert satisfies three tests. It is actionable, meaning a specific response exists. It is urgent, meaning the response cannot reasonably wait. And it is unambiguous, meaning the recipient can tell from the notification what is affected and where to start. A condition that fails any test belongs on a dashboard or in a report rather than in a notification channel.
Alert fatigue is the dominant failure mode. When most notifications require no action, recipients stop reading them, and the one that mattered arrives in a stream that is already being ignored. The countermeasures are removing alerts that have not led to action, adding time windows so brief blips do not fire, grouping related alerts into one notification, and routing informational conditions somewhere other than the on call path.
Severity should map to a routing decision rather than to a feeling. A common structure has one tier that pages a person immediately at any hour, a second that reaches a team during working hours, and a third that only lands in a channel or digest. Once tiers determine routing, arguments about whether something is critical become concrete questions about whether it justifies waking someone.
Alerts on absence deserve separate mention because they are frequently missing. A dead man switch inverts the usual logic: a heartbeat is expected on a schedule, and the alert fires when the heartbeat stops. This catches the case where the monitoring path itself has broken, which is precisely the situation where every other alert has fallen silent for the wrong reason.
Key points
- Alert only on conditions that are actionable and urgent.
- Fatigue from noisy alerts is the main cause of missed incidents.
- Severity tiers should map to concrete routing decisions.
- Group related alerts and add windows to suppress brief blips.
- Dead man switches catch a monitoring path that has gone silent.
In practice
A platform pages on call only when the checkout error rate exceeds two percent for five consecutive minutes, since that condition has an owner and a runbook. Slow background report generation posts to a team channel instead, because it can wait until morning. A separate heartbeat check expects a signal every ten minutes from the metrics pipeline, and pages when two are missed, so a broken monitoring path cannot masquerade as a quiet night.