AI Incident Response: Agents in the On-Call Rotation

Everything about incident response as practiced today is an accommodation of human limits. The rotation exists because humans sleep. The escalation policy exists because humans miss pages. The severity matrix exists because human attention must be rationed. The runbook exists because human memory fails at 3 am. The incident commander role exists because humans cannot debug and communicate simultaneously. Strip out those five accommodations and you have stripped out most of what an SRE textbook calls incident management.

Which raises a question engineering leaders are going to spend the next few years answering whether they want to or not: what does incident response look like when some of the responders are not human? Not tooling, not a chatbot bolted onto the paging system, but agents that hold a place in the rotation's workflow, take the first pass at triage, run the communication layer, execute bounded remediations, and escalate to humans against explicit thresholds.

This is already how a visible minority of teams operate, and the pattern is spreading for the dullest possible reason: the economics of the pager are brutal and everyone knows it. On-call burnout is a top-three attrition driver in every engineering survey that bothers to ask. Mean time to resolution has been stubbornly flat for a decade despite an observability industry worth billions, because the bottleneck was never seeing the problem, it was the serialized human labor of orienting, communicating, and remembering. Do the arithmetic on your own rotation: a team of six carrying a pager pays roughly a fifth of its senior capacity in interrupted nights, recovery days, and the defensive scheduling everyone does around their on-call week, and it pays that whether or not anything breaks. That is the standing tax the current model charges before a single incident is counted.

I want to make the case for mixed human-and-agent incident response the honest way: by taking the objections seriously, because the objections are good, and the teams that ignored them have already produced the cautionary tales.

Objection one: you cannot page accountability

The strongest objection first. When a human responder makes a bad call at 3 am, there is a person who owns it, learns from it, and answers for it. An agent that misdiagnoses a failure or runs the wrong remediation owns nothing. Incident response is precisely the domain where accountability gaps kill, and "the AI did it" is the beginning of an organizational rot that ends with nobody responsible for anything.

Correct, all of it. And it is an argument about design, not about feasibility, because the teams doing this well solved it with a rule so simple it barely deserves the word architecture: every agent in the incident workflow has a named human owner, and every action an agent may take is sorted, in advance and in writing, into one of two lists. The reversible-and-bounded list, restart the stateless service, roll back to the last known-good config, scale the pool within preset limits, post the approved-template status update, the agent may execute, logged and announced in the channel as it happens. Everything else, and the everything-else list is deliberately long, the agent may only prepare: assemble the diagnosis, draft the remediation, lay out the evidence, and put it in front of the human with the pager.

The accountability question then has the same answer it has always had. The human on call owns the incident. What changed is what the human is accountable for: judgment, not stenography. One VP of engineering put the before-and-after crisply: "My responders used to be accountable for noticing, remembering, typing, and deciding. Three of those four were always a waste of an engineer. Now they're accountable for deciding, and they decide better, because they're not doing the other three."

Insist on the named owner. Insist on the two lists. Walk away from any setup, vendor or homegrown, that cannot show you both.

Objection two: the agent will be confidently wrong at the worst moment

Also correct, and anyone who tells you otherwise is selling something. Agents misread evidence. They pattern-match to the wrong prior incident. At 3 am, a confidently wrong briefing is arguably worse than no briefing, because a sleep-deprived human anchors on it.

Three things blunt this in practice. First, the two-lists design above means a wrong diagnosis costs minutes, not availability: the agent's wrongness is confined to preparation, and preparation is checkable. The teams that got burned are, without exception, teams that let agents act outside the reversible list. Second, the briefing format matters enormously, and mature teams enforce it: claims must arrive with their evidence attached, the graph, the diff, the log lines, so the human is reviewing an argument rather than trusting a verdict. An agent that says "probably the 01:47 deploy, here is the diff and here is the error correlation" fails safe; an agent that says "it's the deploy, rolling back" does not. Third, and least appreciated: wrongness is measurable and improvable in a way human 3-am wrongness never was. Every agent triage gets scored against the eventual postmortem finding, in the open, and the trend line either earns expanded scope or contracts it. Nobody has ever run that loop on a human responder without a lawsuit.

The honest comparison is not agent versus perfect responder. It is agent-plus-rested-human versus exhausted-human-alone, and that comparison stopped being close a while ago.

The honest comparison is not agent versus perfect responder. It is agent-plus-rested-human versus exhausted-human-alone, and that comparison stopped being close a while ago.

Objection three: junior engineers learn from the pager

This one is usually delivered as a gotcha and deserves better, because it is true and it is the objection with the longest fuse. A generation of senior engineers got their systems intuition from nights spent tracing failures nobody would help them with. If agents absorb triage and first response, where does the next generation's scar tissue come from?

Where it should have come from all along: deliberately, instead of traumatically. The pager was always a terrible teacher, a random curriculum administered at the worst hour to whoever was unlucky, with no feedback loop except survival. Teams taking the skills problem seriously are doing things the old model never did: juniors run incident replays against the recorded timelines agents now keep of every real incident, the aviation-simulator model our industry always claimed to admire and never built because reconstructing incidents by hand was too expensive. Juniors take review shifts where their job is to audit and challenge agent triage calls from the past week, which trains exactly the judgment the new world runs on. And the rotation still exists, still includes them, still pages them for the incidents beyond the agents' scope, which are, by construction, the interesting ones.

What disappears is the hazing, the fourth consecutive 2 am page for the same known issue with the unfixed root cause. If your engineer development pipeline depends on that, the pipeline was already broken and the pager was hiding it.

Objection four: our incidents are special

Every team believes its failure modes are too novel, too entangled, too context-heavy for anything but senior human intuition. Every team is right about roughly a tenth of its incidents and wrong about the rest. Pull your own incident history and sort it honestly: the recurring known issue, the noisy alert, the capacity blip, the failed deploy with the obvious rollback, the certificate that expired because chasing renewals is nobody's job. That is the bulk of the pager's volume, it is beneath your engineers, and it is exactly the tranche agents already handle well, including the unglamorous connective work around the genuinely hard cases: assembling the briefing, running comms, keeping the timeline, cross-referencing the postmortem archive, chasing the follow-up items afterward so the same incident stops recurring.

The tenth that is truly novel still gets your best humans, arriving oriented instead of blind, with the organization's entire incident memory pre-searched and the stakeholders already fed. Novelty was never the argument against this. Novelty is the argument for reserving human attention for it.

Objection five: humans will stop checking

Automation complacency is the aviation industry's gift to this conversation, and it is a real gift: decades of evidence that humans supervising reliable automation gradually stop supervising, and that the failure, when it comes, finds an operator whose vigilance atrophied precisely because the system earned it. If the triage agent is right forty times in a row, the human approves the forty-first briefing without reading it, and the forty-first is the wrong one.

The teams that take this seriously borrow aviation's answers rather than relearning them. Approval interfaces are designed to require engagement: the human confirms the evidence itself rather than rubber-stamping the conclusion. Agents are configured to express calibrated uncertainty instead of uniform confidence, because a system that says "80 percent match, but the memory pattern doesn't fit" trains skepticism where a system that always sounds sure trains sleep. Review shifts, the same ones that develop juniors, double as a vigilance mechanism: someone is always grading last week's agent calls, in public, so drift gets caught as a trend rather than an outage. And a few teams run the practice with the strongest pedigree of all: periodic injected drills where the agent's briefing is deliberately wrong and the rotation's job is to catch it. Complacency is not a reason to keep humans doing everything. It is a design requirement, with fifty years of prior art, for letting them stop.

Where teams actually start

For all the strategy above, the adoption path that works is almost embarrassingly narrow. Nobody credible starts with autonomous remediation. Teams start with the communication layer, the briefing on page, the stakeholder answers, the status drafts, the kept timeline, because it is the layer where a wrong agent costs a correction instead of an outage, and where the on-call rotation feels the relief within one bad night. Trust accrues, the two lists get their first entries, scope expands against evidence. Six months of that, and the question inside the team quietly inverts from "should we let it do more" to "why are humans still doing this part." Which is the only sequence in which the ending below arrives at all.

The decision, restated

Here is what this choice actually is, for the leader who owns reliability and the engineers attached to it. You are not deciding whether to trust AI with your incidents. You are deciding how much longer to keep running incident response as a human-limits accommodation layer: paying senior engineers to be switchboards, letting institutional memory live in the heads of whoever hasn't quit yet, and treating sleep deprivation as a professional development strategy.

Somewhere in your organization, this quarter, an engineer you cannot afford to lose is deciding whether the pager is a reason to leave. The rotation that keeps her is one where the 2 am page arrives with the diagnosis attached, the comms handled, the history searched, and only one question left, the question she is actually good at: is this right, and what do we do?

Build that rotation before your competitor's job posting describes it.