Flight data recorders exist for one reason: after something goes wrong, people want evidence instead of folklore. Software teams face the same need when a service fails, and a loosely used term, IncidentOps, has grown up around how they respond and what they learn. This explainer keeps the jargon low and the evidence high, using aviation technology as the reference frame for readers at a pilot-training site.
What the term actually covers
According to the Ops Atlas page on IncidentOps, the word is an informal nickname, not a standard discipline. Underneath it sits ordinary incident management: spotting that something has broken, naming it as an incident, putting someone in charge, telling the people affected, and reviewing the event afterwards so the next one goes better. The Atlas adds a boundary worth remembering. Planned outages are not incidents, and neither is every single alert that fires.
Think of an avionics shop. A scheduled box swap during a maintenance check is routine work. A cockpit warning that appears in flight is a different category, and the crew’s reaction to it follows a script. The software version has the same split, and getting it right stops teams from wasting emergency machinery on routine jobs.
Plain-language walk through the lifecycle
1. Notice
Something trips a monitor, or a customer complains. Notice sits at the border of this discipline; the Atlas hands the design of alerts and telemetry to a neighbouring field and keeps its own attention on what happens once a human is paged.
2. Declare
Someone announces that an incident is under way. The Atlas favours written criteria so nobody has to argue about it at three in the morning. A cheap early declaration that turns out to be minor is far better than a late one, because starting late tends to eat hours.
3. Organise
One person coordinates and does not touch the fix. A small group of subject experts is the only set of people allowed to change production. Another person writes the outward updates. The entry compares this with a flight deck where the pilot flying, the pilot monitoring and the radio each have a defined lane.
4. Stabilise
The first goal is relief. Generic moves such as reverting a release, shifting load, switching off a new capability or failing over need a fault location rather than a proven root cause. Analysis can wait until the aircraft is under control.
5. Communicate
Updates follow a rhythm, and a hosted status page uses a handful of simple states: investigating, identified, monitoring and resolved. Silence is treated as the worst outcome.
6. Learn
A review names an owner before the call even closes. The Atlas cites one vendor that expects reviews within three calendar days for its most serious level and five business days for the next. Actions should be few, each with an owner and a date.
Why the numbers deserve suspicion
Most explainers skip measurement, yet the Atlas spends serious space on it. The headline figure many teams chase is mean time to resolve. The entry notes that the abbreviation has at least four accepted meanings, which Atlassian lists as repair, recovery, respond and resolve. A chart labelled with the abbreviation alone tells you nothing about which interval was timed or which incidents counted.
The deeper problem is statistical. Incident lengths are not bell-shaped; most are brief, while a handful of rare marathons stretch the right-hand side of the chart. A Google researcher simulated what happens when a team really does shorten incidents by ten percent. In three separate company datasets, the measured average still came out worse than before in about 38, 40 and 20 percent of the runs. Run the opposite test, with no real change at all, and there was a roughly one-in-five chance of seeing a spurious half-hour gain. The Atlas quotes the researcher’s blunt advice: when judging trends, the default stance should be to reject such averages.
For a pilot, the analogy is block time. Averaging the duration of every flight a school logs will not reveal whether instruction improved, because one diversion or weather hold distorts the mean. A fairer view shows the spread and says plainly which flights are included.
What to report instead
- Distributions of time-to-detect, acknowledge, mitigate and resolve, with the phase stated.
- The impact against a reliability target, counted in minutes of budget used.
- Pager load, as a trailing average so a bad week is visible but not decisive.
- The share of follow-up actions closed by their deadline.
Evidence about people, not just systems
The Atlas also collects softer evidence. A 2026 community survey of 1,363 self-selected respondents found about three in ten naming alert fatigue as a problem, and roughly one in six calling cross-team coordination painful. Because respondents chose to take part, the Atlas treats it as a signal rather than a measurement of the whole industry, and that caution is worth copying.
There is a sobering workload figure too: a large search company is quoted as budgeting around six hours of effort per incident on average, which scales to something like seventy-two person-hours per quarter for a team handling a dozen of them. Nothing about that number is dramatic, but it shows incidents are a real tax on the roster.
Design rules that come from failure cases
Two architectural rules appear in the entry. The first says that the route used to page people, and the page used to tell customers what is happening, must not depend on the system they are reporting about. The illustration is a 2017 storage outage in which the cloud provider’s own health dashboard was unavailable for hours because it relied on the failing service, so the provider fell back on social posts and banners.
The second rule is about paperwork: capture the timeline a single time while the response runs, then let that one record feed the review, the customer summary and any regulatory notice instead of retyping it three times. Retyped versions drift apart, and the differences cause arguments later.
If you like seeing how infrastructure learning is framed for working engineers, a guide on building infrastructure engineering skills from this same blog covers the training side. And the evidence-minded habit of questioning averages shows up outside engineering too, for example in an article about spotting small money leaks that looks at where the totals really come from.
How to check your own understanding
Try these five questions on any operation, aviation or otherwise:
- Is there a written line that tells you when an event becomes an incident?
- Does the coordinator stay out of the repair work?
- Can you point to the page where customers get updates, and is it hosted apart from the thing that might fail?
- Does your reporting show spread, not just a single mean?
- Do reviews end with owned, dated actions rather than general intentions?
If you answered no to several, you have found the work. If you answered yes to all five, you are probably nearer the middle rungs of the Atlas maturity ladder than the top, and that is a respectable place to be.
Where aviation readers can apply this
You do not need to run servers to use any of it. A flying club coordinating aircraft availability, a ground-school booking system or a parts-ordering process will eventually fail in some way. Agreeing in advance who coordinates, who contacts students and who fixes the fault turns a messy afternoon into a manageable one, and a short written review keeps the same surprise from repeating.
Takeaway
Strip away the vocabulary and the subject is plain: decide quickly that something is wrong, give people clear jobs, restore service before chasing causes, speak to those affected on a schedule, and learn without blame. The evidence-led twist is to distrust single-number scorecards. A cockpit voice recorder never reduces a flight to one average, and software teams should not either.

