An AI Mission for Engineering: Automate Incident Triage

The first twenty minutes of a 3am page are almost never spent diagnosing anything. They are spent answering one question — is this mine — and that question can be very nearly answered before anyone's phone rings. The expensive failure in incident response is not a slow diagnosis; it is a page that woke the wrong person.

The alert fires at 3:04 in the morning, and it says what alerts say: error rate on a service has crossed a threshold and stayed there for two evaluation windows. Somewhere a phone lights up on a nightstand, and an engineer who was asleep four minutes ago is now sitting up in the dark with a laptop, waiting for a VPN to connect and a dashboard to load. What happens next is the part that nobody writes runbooks for. The engineer is not yet trying to work out what broke; they are trying to work out whether the thing that broke belongs to them. They check whether anything shipped to their service in the last few hours, then whether the errors originate in their code or are merely passed through from something they call, then whether a dependency they do not own is quietly degraded, then whether the failures map to a region, a client, or a single unlucky shard. Only when that boundary has been drawn does the actual investigation begin, and by then a good chunk of the night is gone.

That opening stretch is the real substance of incident triage, and it is badly misunderstood by almost everyone who builds tooling for it. We talk about triage as though it were early diagnosis, a compressed version of the analysis a postmortem will later do properly, when in truth it is a routing decision made under sleep deprivation with partial information, and the expensive way for it to go wrong is not that the diagnosis takes an extra hour. It is that the page lands on the wrong team — because the cost of that mistake is a second handoff at a worse hour than the first, on top of a person who lost an entire night for something that was never theirs to fix.

The alert knows where it hurts, not where it started

The reason ownership is hard at 3am is structural rather than a failure of anybody's diligence. On-call rotations are drawn around ownership of code: this team owns this service, so this team carries its pager. Alerts, meanwhile, fire on observable surfaces — the endpoint whose latency rose, the queue whose depth grew, the job whose success rate fell — and in any system with more than a handful of moving parts, the surface where a symptom becomes visible is routinely nowhere near the change that caused it. A timeout budget consumed three hops downstream shows up as errors at the edge. A configuration value rolled out to a shared platform component manifests as one team's failures and not another's, for reasons that will make perfect sense in daylight and none at all in the dark. A client team's retry policy change turns a blip into saturation somewhere they have never looked. The alert is a truthful report about where it hurts, and pain location has never been a reliable guide to injury location.

So the engineer who has been woken becomes, for those first twenty minutes, a human join across systems that hold no memory of each other. The deploy history lives in one place, the configuration changes in another, the dependency health in a third, the ownership mapping in a registry that may not have survived the last reorg, and the memory that this error signature appeared once before in a colleague's head. None of these systems knows the others exist, and none was consulted before the page was sent, so the page arrives carrying a threshold breach and nothing else — and a person who was unconscious moments ago is asked to assemble the context from scratch, at the hour when human beings are worst at exactly that kind of careful cross-referencing.

This is not a marginal activity in an engineering organization, either, which is worth saying plainly given how often operational work is treated as the tax rather than the job. IDC's analysis of how developers actually spend their time found that coding accounted for as little as sixteen percent of it, with the overwhelming majority going to operational and background work — the carrying of systems rather than the writing of them. On-call sits squarely inside that majority, and the hours it consumes are not only the ones spent awake but the days afterward, worked at reduced capacity by someone whose sleep was taken from them.

A misrouted page costs two nights, not one

Once you accept that triage is fundamentally a routing problem, the cost structure becomes clear and slightly alarming. When a page reaches the right team, even a slow investigation is productive: every minute the engineer spends builds toward the fix, and the context they accumulate is context that will still be useful an hour later. When a page reaches the wrong team, none of that is true. The minutes spent establishing that the problem lives elsewhere produce nothing but the negative result, and then the handoff begins — a second person woken, at a later hour, rebuilding an understanding of the incident from a thread written by someone exhausted and mildly resentful. The organization has now spent two nights of human sleep to move a ticket one seat sideways, and the incident has been running the whole time.

The temptation, having noticed this, is to make the routing itself smarter — to build something that decides who ought to be woken, or whether the situation warrants waking anyone at all. That temptation should be resisted firmly, and the reason is asymmetry. A page that turns out to be unnecessary costs a night, which is a real cost no one should wave away, but a page suppressed, downgraded, or delayed by a system that reasoned its way to the wrong conclusion costs the incident itself, and the blast radius of that is unbounded. Whatever gets built into the path between an alert and a human must not sit in judgment of whether the human is needed; the correct default is to wake someone promptly and to keep severity in human hands. Compress the context, never the alarm.

It is worth being skeptical here, because this is exactly the territory where the current wave of tooling oversells itself. Gartner has predicted that over forty percent of agentic AI projects will be canceled by the end of 2027, naming among its reasons the practice it calls "agent washing" — existing tools relabeled without any change in what they can actually do. An alert router with a language model bolted to the front is still an alert router; it still picks a queue from a fixed table and hands the same context-free page to whoever it lands on. Adding a probabilistic guess about ownership to a system that then acts on that guess alone is not an improvement in triage. It is the misroute problem with a confident voice.

Answer the ownership question before the phone rings

What is genuinely automatable is the assembly, and the window for it opens the instant the alert condition is met — seconds before any human could plausibly respond. In that window, nothing requires judgment, and everything requires access: pulling every deploy and configuration change that touched the alerting service and its immediate dependencies in the relevant time range, checking whether any of those dependencies are themselves showing correlated deviation, comparing the error signature against the ones this system has produced before and what resolved them, identifying which code paths the failing requests traverse and who most recently changed them, and establishing the shape of the blast radius. Every one of those is a read against a system that already holds the answer, and together they constitute most of what the engineer would otherwise spend their first twenty minutes doing by hand.

This is the natural shape of an AI Mission rather than a script, because the sequence is not fixed. Which questions matter depends on what the earlier answers turn out to be, and that adaptive quality is what separates a reasoning system coordinating specialist agents across observability, deployment, source control, and Enterprise Knowledge — reaching each through the Model Context Protocol — from the runbook automation that has existed for a decade and never helped much at 3am. It is the same logic driving what a growing number of engineering organizations describe as the move toward an autonomous enterprise, applied to the narrowest and most humane possible target: not replacing the on-call engineer's judgment, but ensuring they are not asked to exercise it from a standing start.

The output of that work should be a dossier attached to the page, not a verdict rendered in place of one. It should say what changed, what is degraded, what correlates, and which teams' systems appear implicated, with a link behind every claim so the engineer can verify rather than trust. It should also be explicit about what it could not determine, because assembled context that hides its gaps is worse than none at all, steering a tired person confidently in a direction nobody actually checked. Human-in-the-Loop belongs at every point where the system would otherwise act: it suggests candidate owners and shows its evidence, and the surface owner still gets paged, still holds severity, and still decides who else needs waking.

The mental model worth carrying out of this is that pages have a temperature. A cold page carries a threshold and a timestamp and obliges the person receiving it to reconstruct an entire situation from nothing, at the hour they are least equipped to do it. A warm page arrives with the ownership question already mostly answered and the evidence laid out beside it, so the engineer's first act is to check a conclusion rather than to build one. Most organizations measure their incident response by time to resolution, which quietly averages away the difference. The number that would actually change behavior is how much of the ownership question was already answered at the moment the phone rang — because every percentage point of that is a handoff that did not happen, and somewhere, a night that somebody got to keep.