Alert Fatigue: Cutting NOC Page Volume Before the Pager Burns Out
Published 20 August 2026 · NetMind Engineering
The NOC manager did not buy a paging pipeline. The team grew one — a Datadog here, a Splunk index there, a Prometheus rule per service owner, a Synthetic check on every customer-facing endpoint, a Slack webhook that turned into three more Slacks, a CMDB that nobody will defend in a post-incident review. Every page that hit the rotation was a real signal at the moment it was wired. None of them, alone, were the problem. The problem is the sum, and the sum has quietly outrun the responder's triage bandwidth. Alert fatigue is the moment that sum passes the line, and the program that addresses it is rarely about a single alert. It is about a layer that sits between the sources and the human-facing surfaces, owns the canonical incident state, and stops feeding twenty near-identical pages into a rotation that can absorb one.
Alert volume growth in the modern NOC
The shape of alert volume growth is the same in every NOC I have walked into. It does not arrive as a single catastrophic spike. It arrives as a slow curve that keeps drifting upward — a quarter point at a time — until the rotation quietly carries twice the pages it carried a year ago, and no one remembers the moment that doubled the load. The causes are visible in retrospect and invisible in the moment: a microservices decomposition that fans out one upstream fault into ten downstream symptoms, a Kubernetes rollout that adds a per-pod alarm per service, a multi-tenant edge that doubles the cardinality of every threshold rule, a third observability vendor whose rules overlap with the first two.
Each cause is small. None of them are the metric. The metric is the queue. The right way to read it is on a per-shift, per-responder basis — how many pages does a single on-call engineer absorb on a Saturday compared to a year ago, and what fraction of those pages does the engineer still trust on first read. The wrong way to read it is on a per-quarter aggregate, where growth in page count is a trailing metric and the rotation has already absorbed the harm.
The second tell is structural. The fan-out effect — one upstream fault replicating as twenty or thirty near-identical alarms — is the shape alert volume growth takes in a modern NOC. One customer-facing endpoint goes unhealthy at the CDN edge. The probe fails. The synthetic fires. The latency SLO trips. The customer-experience metric drops. The downstream service's own error budget burns down. A PagerDuty incident opens. Each signal is correct. None of them carry enough context on their own to name the upstream fault. The rotation receives the page count the fan-out produces, not the page count the fault produced. That is the metric that grows the queue, and the metric that does not respond to more rules.
The on-call cost: burnout, MTTA drift, and queue trust
The on-call cost of alert fatigue is not measured in pages. It is measured in the behaviour the rotation develops to survive the queue. The senior engineer opts out of the rotation before the post-incident review does. The P1 lands inside a flood of P3s and gets triaged last, because every page in the queue is a P3 and the brain treats them all as one. The post-incident review slide stops being a root-cause discussion and starts being a queue-trust discussion. The team builds a heuristic that was never in any runbook: treat every page as noise, escalate only what the downstream metric independently confirms, and accept the silent miss as a cost of keeping pace with the page flood.
MTTA — mean time to acknowledge — is the metric that drifts first. Acknowledge time has a floor at the time it takes the operator to read, classify, and decide. When page count outpaces triage bandwidth, the floor rises. The team that was triaging every page in under a minute within a quarter sees the same minute stretch into five, then into fifteen, then into a separately-tracked backlog. The drift shows up in dashboards only after the rotation has absorbed the harm. That is why the right metric for a team that has not yet measured fatigue is per-shift page count, not per-shift MTTA — the latter is a lagging indicator of a problem the former is already producing.
Queue trust is the third cost, and the one that compounds the others. The on-call that does not trust the pager stops trusting the upstream signal. The post-incident review stops being a way to learn from incidents and starts being a way to argue about which acknowledged-too-late fault the rotation should have triaged first. A rotation that has lost queue trust has lost the ability to learn from the queue at all — the operator treats every page as a transaction, the queue stops teaching, and the next outage is triaged against a queue the team no longer reads. The metric a team tracks to detect this state is the actionable-page ratio — the fraction of acknowledged pages that turn out to need a human action — and the team that catches the ratio dropping below a meaningful band is the team that still has time to fix the upstream.
Deduplication vs suppression — what actually cuts noise
The two responses to a noisy queue are dedup and suppression, and they are not the same. Dedup — correlation upstream of the human-facing surface — collapses twenty near-identical alarms that share an upstream fault into a single correlated incident so the on-call receives one page naming the root cause, not twenty pages naming symptoms. Suppression silences by threshold or rule — the on-call receives fewer pages because a rule is hiding some of the pages that would have arrived. Both approaches reduce page volume. Only one of them preserves the signal the on-call needs to triage the fault. The team that picks suppression as the default is the team that ends up with a quiet queue and a slow MTTA, because the fault is still happening and the rotation is still missing it — quieter, not faster.
Dedup is the harder of the two to land because it requires a layer upstream of the human-facing surfaces that can correlate on the fault, not on the rule. That layer does not exist in most toolchains by default — Datadog does not know that a PagerDuty incident is correlated with a Slack incident channel is correlated with a Synthetic probe is correlated with a downstream SLO — so the team that wants dedup has to introduce it. Suppression is the easier of the two to ship because it does not require a new layer, only a new rule, but each rule reduces page volume at the cost of signal and never improves MTTA. The pattern in NOCs that have managed to cut noise cleanly has been: build the correlation layer first, then tune the suppression rules underneath it, never the other way around.
Suppression is the wrong default before dedup. A quiet queue is not a faster queue; it is a queue where the fault is still happening and the rotation is missing it. The team that learns to correlate before it learns to suppress is the team that gets to keep the signal the on-call needs.
The third cost — what most teams call "missed-severity patterns" — is the one that exposes the difference most clearly. When a P1 fault lands alongside a flood of P3 symptoms, a team that has tuned for suppression will have silenced the P3 noise and left the P1 to sink into the noise floor. A team that has tuned for dedup will have collapsed the symptoms into a correlated incident and surfaced the P1 as the named root cause so the on-call triages one fault instead of one alert at a time. The page volume in both states is the same; the MTTA on the P1 is not. That is the test that separates the two responses, and the test most teams run only after a missed-severity incident to a post-incident review that should not have been a surprise in the first place.
Measurable noise-reduction outcomes: page volume, MTTA, MTTR, shift experience
The argument for the dedup layer is strongest when it is measurable. The metrics that hold up across every NOC I have watched land a correlation layer are: page volume per shift, MTTA delta, MTTR delta, and the on-call's experience of the shift as a fraction of the whole. Page volume per shift is the metric that moves first and most, because the correlated incident replaces twenty pages with one and the operator's triage load shifts by an order of magnitude on the same fault. MTTA is the metric that moves second, because the rotation that has fewer pages to triage reaches the actionable page faster and is no longer triaging on the queue boundary.
MTTR — mean time to resolve — moves third, because the on-call that arrives at a correlated incident with the root-cause already named spends the resolution time on the fix rather than on the diagnosis. The drop is modest and persistent, and it is the metric the team reports to leadership because it is the one leadership already tracks. The actionable-page ratio — the fraction of acknowledged pages that turn out to need a human action — moves in the opposite direction and is the metric that tells the team whether the correlation layer is preserving signal rather than silencing it. A drop in page volume that does not come with a stable or rising actionable ratio is a suppression outcome, not a dedup outcome, and the team should treat it as such.
The metric most NOC managers track informally and rarely report is the on-call load split — the difference between routine triage time and the "hard call" time that needs an experienced operator on shift. The right correlation layer grows the second category and shrinks the first, and the metric the team should watch is whether the time-on-shift shifts toward the kind of work the rotation joined the rotation to do. The framing to take into the post-quarter review is not "we cut page volume by X percent" but "we grew the actionable ratio and MTTA dropped alongside" — the "right number of pages" framing, not the "fewer pages" framing. The team that frames it that way is the team whose leadership understands what a correlation layer is buying.
Where to start — the correlation layer that wins on day one
The correlation layer that wins on day one is the one the on-call rotation already trusts the most. For most NOCs that is the paging and incident surface — the PagerDuty account, the Slack incident channel, the ServiceNow queue — because that is where the team currently looks to know what is happening on the network. A layer that lands upstream of those surfaces preserves the surfaces the operators have muscle memory on while absorbing the routing, correlation, and policy work the surfaces were not designed to do. The team that lands the layer behind the trusted surface — not in front of it — is the team whose operators keep the muscle memory and learn to trust the layer as a feed, not as a replacement.
The teams that get the most from a correlation layer treat it as a first-class artifact. They version the contract at the API route, not in the payload, so a schema drift in Datadog or Splunk does not silently drop the fields the on-call was using to triage. They tune the routing policy until the actionable-page ratio settles into a band the rotation trusts. They treat overrides — the on-call re-classifying a page the layer surfaced as the wrong severity — as the learning signal the layer needs to sharpen, not as noise to be averaged over. The teams that get the least treat the correlation layer as a black box that fires pages into a slightly different sequence; the same trap that caught the original point-to-point bridges, just a layer further upstream.
The path from a noisy queue to a correlation layer is not a rip-and-replace of any single tool. It is a layering — one source at a time, starting with the surfaces the on-call rotation already trusts, ending with a toolchain the team understands as a single system instead of a federation of independent alarms. The teams that make the move are the ones that stop treating the page volume as the system and start treating it as a tax on the system. Alert fatigue, addressed by a correlation layer that lands upstream of the trusted surfaces, is the end of that tax — and the first quarter the team stops absorbing silence as a baseline in the post-incident review, the rotation will be the one to call it out.