Anatomy of a sev-2, told through the channel

The uncomfortable truth about incident response is that the technical work is usually the minority of the work. Below is a reconstruction of two incidents, same class of failure, payment webhook processing degraded, at two different companies. Both were resolved by smart engineers in about the same amount of hands-on-keyboard time. Everything else diverged. The channel messages tell the story better than any argument could.

Company A, 02:11 to 04:40

02:11 Pager fires. Marcus, on call, acknowledges.

02:19 Marcus, in the incident channel, to nobody: "Anyone know if the webhook consumer changes shipped today?" It is 2 am. Nobody knows anything. He starts digging through the deploy log himself.

02:31 First stakeholder arrives, a support lead: "Customers reporting failed payment confirmations, what do I tell them?" Marcus, mid-investigation, stops to type a paragraph.

02:44 Marcus finds the suspect deploy, needs the service owner. Pages her. She was not on call. She wakes up her partner, her dog, and then her laptop.

02:58 An engineering manager, awake for reasons of his own: "What's the customer impact? Do we need to post a status page update?" Marcus stops again. Estimates impact by hand-running a query he has to write first.

03:15 Status page updated, twenty minutes after the manager asked, sixty-four minutes after customers started failing.

03:20 to 04:05 The actual fix: the service owner identifies a config regression, rolls it back, verifies recovery. Forty-five minutes of real engineering.

04:12 Support lead again: "Is it safe to tell customers it's resolved?" Nobody is sure who decides that. Three people discuss it.

04:40 Incident closed. Elapsed: two and a half hours. Engineering time: under an hour. The rest went to questions, status, paging archaeology, and impact math, performed by the same two people who were trying to fix the thing. Marcus's summary at standup, later, exhausted: "I spent more time narrating the incident than fixing it."

Company B, 02:11 to 03:05

02:11 Pager fires. Elena acknowledges. By the time she has her terminal open, the incident channel already contains a briefing, posted by the team's triage agent: error rate graph, first bad timestamp, the two deploys that landed within the preceding hour with diffs linked, the service owner's name, and a note that the failure signature matches incident 2024-117 from last spring, postmortem attached.

02:13 Elena, reading the old postmortem: same config subsystem. She pages the service owner through the channel; the agent handles the page and includes the briefing, so the owner wakes up oriented instead of confused.

02:16 The support lead arrives with the same question support leads always have. She does not ask Elena. She asks the channel, and the agent answers with current impact, drawn from the query it maintains for exactly this: roughly 4 percent of payment confirmations delayed, none lost, retries succeeding. The support lead has her customer language in two minutes and Elena never stopped typing.

02:20 The agent posts a drafted status page update for approval. Elena reads it, cuts one adjective, approves. Public within eleven minutes of the page, not sixty-four.

02:24 to 03:00 The fix. Same class of work as Company A: identify regression, roll back, verify. The service owner does it with the prior postmortem open, which saves her the twenty minutes of rediscovery Marcus paid. Meanwhile the agent updates stakeholders every fifteen minutes without being asked, answers an executive's "how bad is this" in the channel at 02:41, and quietly notices that recovery in the EU region is lagging the rollback, which it flags to Elena before she has looked at that dashboard. That flag matters: EU needed one extra cache flush, and finding it at 02:50 instead of at 09:00 tomorrow is the difference between one incident and two.

03:05 Incident closed. The agent posts the timeline it kept, every event timestamped, for the postmortem nobody will now have to reconstruct from memory and screenshot archaeology.

The delta is not the engineers

The instinct is to conclude Company B has better engineers. They don't; if anything Marcus worked the harder shift. The delta is that Company A runs incidents the way the whole industry has for fifteen years, with the responder as a human switchboard: every question, status update, page, impact estimate, and timeline entry routes through the person who is simultaneously supposed to be thinking about the actual problem. We built elaborate on-call rotations, severity matrices, and incident commander training, and then wired every channel of communication directly into the skull of the one person who most needs to concentrate.

Company B did not remove humans from the loop. Count them: Elena ran the incident, a human owner made every judgment call, a human approved every external message, humans wrote the fix. What Company B removed was the switchboard. The agent answered what was answerable, drafted what needed human sign-off, chased who needed chasing, watched what nobody was watching, and remembered what the organization had already learned once and would otherwise have paid to learn again.

Interruption research has been telling us for two decades that a knowledge worker needs twenty-plus minutes to recover deep focus after a context switch. We nod along, then design incident response as a context-switch generator aimed at our most sleep-deprived people, and wonder why the same outage takes one company an hour and another company half a night.

Marcus, by the way, is fine. Tired, but fine. His company is fine too, for now. Their next sev-2 arrives on schedule, statistically speaking, in about three weeks, and it will find them exactly as they were.