In my NOC years, roughly 80% of the time lost in an incident was not spent fixing anything. It was spent finding out what was broken.
That single number is the most useful thing I know about mean time to resolution, and it is the one that vendor material almost never leads with — because the part it addresses is the smaller half.
🔑 The short version
- MTTR is not one number. It is four intervals, and tools only shorten two of them.
- The largest interval is usually finding the problem, not repairing it.
- The interval nobody measures is the handoff — the minutes spent deciding whose problem this is.
- A platform that correlates signals attacks the second interval well. It cannot touch the fourth.
- If your MTTR is not improving after buying a tool, the bottleneck was almost certainly organizational.
Break the number apart before you try to move it
“Reduce MTTR” is not an actionable goal, because MTTR is a sum of four very different things that respond to completely different remedies.
| Interval | What happens | What actually shortens it |
|---|---|---|
| Detect | From the failure starting to somebody knowing | Better alerting — and fewer alerts, not more |
| Diagnose | From knowing to understanding where | Correlated signals. This is where tooling earns its price |
| Repair | Applying the fix | Automation, runbooks, permissions |
| Coordinate | Deciding whose problem this is | Nothing you can buy |
Most organizations measure the total and optimize the third. The first two are usually bigger, and the fourth is frequently the biggest of all — and it is invisible, because nobody logs “we spent forty minutes establishing that this was not the network team’s fault.”
Detection: the counterintuitive part
Everyone assumes better detection means more alerts. In practice it means fewer.
A team receiving two hundred notifications a day is not detecting faster — it is detecting more slowly, because the real one arrives among a hundred and ninety-nine that did not matter, and the human brain stops reading. I have watched this happen in operations rooms with excellent tooling and exhausted people.
The two changes that genuinely move this interval:
- Alert on the user journey, not the component. “Payments are failing” wakes the right person. “CPU at 91%” wakes somebody who then has to work out whether anyone cares.
- Break alerts down by dimension. A localized degradation disappears inside a national average — the same reason a whole region can be down while availability looks excellent. Alerting per region or per service is what stops that.
Diagnosis: where a platform earns its money
This is the interval where a modern observability platform genuinely helps, and it is worth being precise about why.
The old workflow was: see a spike on a metric dashboard, open a second tool to search logs around that timestamp, guess which service to look at, open a third tool for traces. Every jump is a context switch, and every context switch is where minutes disappear.
Having the three signals in one place, linked by a shared identifier, removes those jumps. That is the actual product being sold, underneath the marketing — not more data, but fewer transitions between windows at two in the morning.
Platforms in this category also apply machine learning to spot anomalies without someone predefining a threshold, and to group related alerts into a single incident. Both are real capabilities. I would treat them the way I treat any vendor claim: useful, and worth testing against your own noisy production data rather than a demo — because the demo environment never has your five years of accumulated exceptions.
On correlating signals and incident tooling: Datadog tracing documentation and its anomaly detection documentation. On defining what to measure before optimizing it: Google SRE, service level objectives.
Repair: the interval that is really about permissions
Automation gets all the attention here, and it does help: automatic rollbacks, self-healing runbooks, one-click actions from the alert.
But in every organization I have worked in, the repair interval was governed by something much less glamorous: who is allowed to act at three in the morning.
If the person who understands the problem needs to wake somebody else for approval, or file a ticket with a provider, or wait for a change window, no automation platform touches that. The delay is in the authority model, not the tooling.
A metric you answer for but cannot influence is not a metric. It is theater.
I learned that reporting monthly on a voice quality number whose controlling policies belonged to the connectivity provider. We could look. We could not touch. And the fault, when it came, came from the side that showed us nothing.
The interval nobody measures
Coordination. The meeting. The six teams and six green dashboards, each honestly reporting that the problem is not theirs.
Nobody records this time, because it does not feel like downtime — it feels like working. But it is frequently the longest single stretch of an incident, and it has one structural cause: each team can only see its own piece, so nobody can see the space between pieces.
The instrument for that is distributed tracing — not because it is faster, but because it produces one picture that all six teams read the same way. It converts an argument into a diagram.
And the non-technical version, which matters just as much: decide who runs an incident before you have one. Not who fixes it — who runs it. Most of the coordination interval is the absence of that decision.
What I would ask before buying anything to fix MTTR
| The question | Why it matters |
|---|---|
| Do we know our four intervals separately? | If not, you cannot tell whether a tool helped. This costs nothing to start |
| How many alerts does a person receive per shift? | Above a few dozen, buying better detection makes things worse |
| How many windows does an engineer open during an incident? | This is the number a correlated platform actually reduces |
| Who is authorized to act at 3am, without asking? | Governs the repair interval more than any automation |
| Who runs an incident, as opposed to fixing it? | If the answer is “it depends,” you have found your longest interval |
| Can we test with our real alert noise? | Anomaly detection is easy on clean data. Yours is not clean |
Frequently asked questions
What is MTTR and why is it hard to reduce?
Mean time to resolution is the average time from a failure starting to service being restored, and it is hard to move because it is not one thing. It is four intervals — detecting, diagnosing, repairing and coordinating — that respond to entirely different remedies. Most teams measure only the total and optimize repair, which is often the smallest of the four.
Which part of an incident actually takes the longest?
Usually finding the problem rather than fixing it — in my NOC years roughly 80% of the lost time went to locating the fault. The other large and invisible interval is coordination: the minutes spent establishing whose problem it is, which nobody logs because it feels like working rather than downtime.
Does buying an observability platform reduce MTTR?
It reduces the diagnosis interval, genuinely, by removing the context switches between separate tools for metrics, logs and traces. It does not touch the coordination interval, and it only helps repair if your people already have authority to act. If MTTR does not improve after buying a tool, the bottleneck was almost certainly organizational rather than technical.
Do more alerts mean faster detection?
No, generally the opposite. A team receiving two hundred notifications per day detects more slowly, because the one that matters arrives among a hundred and ninety-nine that do not and people stop reading. The changes that help are alerting on the user journey instead of on components, and breaking alerts down by region or service so a localized failure does not vanish inside a healthy-looking average.
How do I start measuring MTTR properly?
Record four timestamps per incident: when it started, when someone knew, when the cause was located, and when service was restored. That gives you the four intervals separately and costs nothing but discipline. Without that breakdown you cannot tell whether a purchase helped, and you will keep optimizing whichever interval is easiest to see.
What I take from this
The reason MTTR resists improvement is that it is treated as a technical metric, and at least half of it is not.
Detection and diagnosis respond well to tooling, and the tooling available now is genuinely better than what I had. Repair responds to permissions. Coordination responds to having decided, in advance, who is in charge. Two of the four are organizational, and no purchase order reaches them.
Which is why the most valuable thing you can do this month costs nothing: start recording the four timestamps. Within a handful of incidents you will know which interval is actually eating your night — and, more often than anyone expects, it will not be the one you were about to buy a solution for. ⏱️
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.