Why Accuracy Is the Wrong Metric for AI Incident Triage

Consider a real-world on-call scenario: at 2 a.m., a database latency alert wakes the on-call engineer. A few minutes later, several downstream services start paging their owners. An AI triage system reviews the alerts, matches them to earlier incidents, and points to the checkout service. It assigns medium severity and sends the page to the application team. 

The choice makes sense. Checkout incidents have produced similar symptoms before. This time, though, the source is a replication inconsistency in another region. While the application team checks recent deployments, retries and failovers spread the impact. 

The system did not invent a random answer. It chose the familiar story and sent responders in the wrong direction. That is the failure mode I am concerned about. 

AI-assisted triage is often presented as the answer to alert fatigue. These systems combine historical incidents with live telemetry, ownership data, and deployment records. A language model then estimates severity, suggests an owner, and recommends where to start. In a busy on-call rotation, that sounds genuinely useful. 

I arrived at this view after spending years on call for large-scale distributed systems serving hundreds of millions of users. Later, while rebuilding an incident-response program and running simulations for engineers, I saw the same problem from another angle. Alert fatigue is rarely just a matter of volume; more often, alerts are incomplete, redundant, or disconnected from user impact. AI can condense that mess, but it cannot recover context the organization never captured. 

The Signal Fidelity Problem 

Large production environments generate noise from several directions at once. A dependency failure can trigger alarms in every service that calls it. Partial failures make metrics cross the same threshold again and again. Brief network disruptions may resolve before anyone opens a dashboard, and a deployment can move half a dozen indicators without causing any real customer impact. 

Experienced engineers learn how to read around this noise. They know which service tends to show the first reliable symptom, which page is a chronic false positive, and which dashboard looks alarming even on a normal day. Much of that knowledge is informal. It lives in old incident threads, in team habits, and in the heads of people who have seen the system fail before. 

An AI assistant still works from whatever the organization has recorded: alerts, metrics, traces, runbooks, change histories, and ownership maps. When those sources disagree, the ambiguity remains. A polished summary can make it harder to see. 

Bad alerting cannot be repaired at the presentation layer. An alert may have no clear relationship to user impact, no reliable owner, and the same symptoms as ten unrelated failures. Rewriting it in fluent English does not make it a better signal. 

Where the Numbers Become Misleading 

Some incidents really are good candidates for automated triage. Capacity exhaustion, a known bad deployment, an expired certificate, or a familiar dependency outage often resembles something in the historical record. Correlation and routing can save time there. 

The incidents I worry about are rarely alien in every respect. They mix familiar pieces in a new way: a partial regional failure combined with stale routing state, new code touching old data, or two safety mechanisms that work separately and fail when they activate together. 

Depending on the system, a confidence score may show how strongly the model prefers one classification based on the patterns it has learned. It does not tell the responder whether an important signal is missing, telemetry is stale, or the current incident only superficially resembles old examples. 

Those errors are not equally expensive. Correctly routing a routine alert may save ten minutes. Downgrading an early sign of data corruption can lose far more, especially when the first classification anchors responders on the wrong explanation. An average accuracy score hides that distinction. 

Suppose a triage system routes 95 percent of alerts correctly. That sounds impressive until we look at the remaining five percent. Sending a low-priority ticket to the wrong team is annoying, but suppressing an early warning of a cross-region incident is a different category of failure. Treating both as one incorrect prediction produces a metric that is easy to report and hard to trust. 

Evaluation has to reflect both the cost of an error and the conditions around it. I would look for clusters: incomplete telemetry, several services failing together, weak ownership data, or a thin historical record. A mistake in one of those cases deserves more attention than another correct classification of a known certificate problem. 

Mean time to resolution is also too broad to explain whether the assistant helped. I would track narrower measures: time to the correct owner, time to the first evidence-backed hypothesis, time to the first safe action, wrong-team routing, and unsupported recommendations. One particularly useful measure is how often the assistant sends responders down a path they would not otherwise have pursued. 

The Skill AI Makes More Important 

One lesson came from an incident-response program I led at a company building a public-facing service. We ran simulated production failures in which engineers had to investigate several competing explanations. Some exercises included misleading alerts on purpose. The aim was to teach responders to notice when the evidence had stopped supporting their first theory and change direction. 

Across the broader program, which also included updated runbooks and a simpler incident process, time to mitigation improved by roughly 1.5x. Just as important, engineers became more comfortable challenging the initial page instead of treating it as an accurate description of the failure. 

AI makes that skill more important. A fluent summary can become a powerful anchor, particularly at night or during a fast-moving outage. Engineers need practice tracing its claims back to the evidence, noticing what it omitted, and discarding its explanation when the system’s behavior no longer fits. 

What the Assistant Should Deliver 

The assignment I would give a triage assistant is narrow: prepare an evidence package that helps the responder choose the next check. Declaring the root cause is outside that boundary. The package should answer a few practical questions: 

• What changed shortly before the incident began?
• Which users, regions, and services appear to be affected?
• Why does the suggested owning team look like the best match?
• What supports the leading explanation, and what contradicts it?
• What remains unknown, and what is the safest next check?

The source of each fact matters. Deployment history, service ownership, timestamps, SLO calculations, and configuration changes are better retrieved by deterministic systems. The language model can organize and explain the result. It should not invent a missing deployment, infer ownership from a service name, or quietly smooth over inconsistent timestamps. 

Competing explanations should remain visible. During an incident, a responder needs to follow each claim back to the metric, log line, change record, or runbook behind it. An explanation that can be checked quickly is more useful than a confidence score on its own. 

Run It in Shadow Mode First 

Historical replay is a sensible starting point. Hide the documented root cause and give the assistant only the evidence that responders had at the time. Then compare its output with the real timeline: whether it found the correct owner, surfaced the decisive signal, and suggested a safe first action even when its main hypothesis was wrong. 

The tests should include incomplete and misleading evidence, not only cleanly documented incidents. Remove a dashboard that responders relied on, add a stale metric, or combine symptoms from two known failure modes. Include incidents with weak postmortems as well. Historical data naturally overrepresents failures the organization managed to understand and document. 

If the offline results are promising, the next step is shadow mode. On-call engineers can see the output, but the system cannot suppress, downgrade, or reroute a page. Record when responders use the recommendation, correct it, or ignore it. The revealing cases are often those where the answer sounds persuasive but cannot be traced to solid evidence. 

Some conditions warrant human review regardless of the model’s stated confidence: possible data corruption, cross-region impact, an unfamiliar combination of symptoms, or large gaps in telemetry. A low-confidence answer is an obvious reason to pause, but it is not the only one. High potential impact should be enough. 

Keep the Diagnosis With the Engineer 

AI can remove a meaningful amount of mechanical work from incident response. Gathering related alerts, retrieving recent changes, and finding the right runbook all take time. Having that context ready is valuable when the on-call engineer is tired, and the system is producing more information than one person can absorb. 

Keeping that boundary also makes failures easier to inspect. When the assistant is wrong, responders should be able to separate facts retrieved from monitoring systems from interpretations supplied by the model. 

For now, I would keep the boundary simple. Conventional systems collect the facts, while AI organizes them and points out patterns. The on-call engineer still owns the decision. A useful assistant does not need to sound like the most confident person in the incident channel; it needs to help the responder ask the right question sooner.