There is a specific kind of two in the morning that only happens in distributed systems.
Something is slow. Not broken — slow. Six teams are on the call. Every one of them has a dashboard, every dashboard is green, and every person on that call is entirely honest when they say the problem is not on their side.
They are all right. The problem lives in the space between them, and nobody owns the space between.
I spent years in that call, in a different decade, with a telephone instead of a terminal. Distributed tracing is the thing that ends it.
🔑 The short version
- A distributed trace follows one request across every service it touches, and shows where the time went.
- It replaces the meeting about whose fault it is with a picture everybody reads the same way.
- Its real superpower is proving the problem is not yours — with evidence, to a third party.
- OpenTelemetry graduated at the CNCF in May 2026, so this stopped being a bet.
- The mistake that costs money is not technical: it is attaching unbounded values to your telemetry.
What a trace actually is
A metric tells you the system is slow. A log tells you one thing that happened. Neither tells you where, and in a system made of a dozen services, where is the entire question.
A trace assigns one identifier to a request when it arrives and carries that identifier through every hop it makes. Each hop records how long it took. At the end you have one picture: this request spent 12 milliseconds here, 30,000 there, and then gave up.
That is it. The concept is almost insultingly simple, and it changes incident calls completely — because it converts an argument between teams into a diagram nobody can dispute.
Anatomy of one checkout
Concrete beats abstract, so: a customer presses “pay.” That single request crosses three services.
| Hop | Time | What you learn |
|---|---|---|
| Checkout service | 8 ms | Fine. Not the problem, and now you can prove it |
| Inventory | 12 ms | Also fine. Two teams just left the call |
| Payment gateway | 30,000 ms | There it is. And it is a third party |
Thirty seconds in one hop, then a timeout. Without the trace, six teams spend an hour proving innocence. With it, the conversation starts at the right place in under a minute — and the two teams who were never involved go back to sleep.
The part nobody tells you: proving it isn’t yours
This is the use I care about most, and it is not usually how tracing gets sold.
In managed services, a large share of your incidents come from somewhere you do not control: a carrier, a payment provider, an identity service, a partner’s API. And the hard part is never suspecting it. The hard part is proving it to someone who is not on your payroll and does not want it to be their fault.
I lived the extreme version of this for years: I was accountable every month for a voice quality number whose controlling policies belonged to the connectivity provider. We could look; we could not touch. And when the fault came from their side, that side showed us nothing.
A metric you answer for but cannot influence is not a metric. It is theater.
A trace is what turns that from an argument into a document. You are no longer saying “we think it’s you.” You are showing exactly where the time went, with timestamps. That changes the meeting, and eventually it changes the contract.
It stopped being a bet
Graduation is the foundation’s highest maturity level — the same one Kubernetes holds. For procurement that matters more than it sounds: adopting the standard is no longer choosing a promising project, it is choosing the default.
Worth watching in the same breath: profiling is being folded in as a fourth signal, and work is underway to capture telemetry from the operating system without touching application code. Both point the same direction — less instrumentation written by hand, more of it arriving by default.
Graduation and maturity: CNCF announcement and InfoQ. Profiling as a fourth signal and instrumentation without touching code: profiles in alpha and the OBI goals.
How to instrument without losing your mind
The failure mode here is ambition. Teams decide to instrument everything, it takes a quarter, and the value arrives too late to defend the effort.
- Start at the edge. The service where requests enter. That one hop already tells you which journeys are slow.
- Then follow one journey end to end — the one that generates the most arguments. Not the easiest: the most contested.
- Propagate the identifier into your logs from day one. This is the step everyone skips and everyone regrets. Without that shared thread, jumping from a log line to its trace is impossible and you are back to manual archaeology.
- Automatic instrumentation first, hand-written later. The libraries cover most of it. Custom spans are for the parts that are genuinely yours.
The mistake I see every time
It is never a missing feature. It is cardinality, and it arrives disguised as a good idea.
Somebody attaches the customer ID to a metric during an incident, because in that moment it is genuinely useful. Fifty endpoints becomes fifty times ten thousand customers, and the invoice grows by multiplication rather than addition. In code review the line looks identical whether it adds fifty series or half a million.
The rule that has never failed me: if a value can be unbounded, it does not belong on a metric. Customer IDs, order numbers, session tokens, full URLs. Put them on the trace or the log, where you pay per event, not per combination.
Frequently asked questions
What is a distributed trace?
It is the record of one request as it crosses every service in your system. A single identifier is attached when the request arrives and travels with it through each hop, and every hop records how long it took. The result is one picture showing exactly where the time went — which is the question metrics and logs cannot answer on their own in a distributed system.
How does tracing help when the problem is a third party?
It turns a suspicion into evidence. In managed services many incidents originate outside your control — a carrier, a payment provider, an identity service — and the hard part is not suspecting it, it is proving it to someone who does not work for you and does not want it to be their fault. A trace shows precisely where the time was spent, with timestamps, so the conversation stops being an argument and starts being a document.
Where should I start instrumenting?
At the edge — the service where requests enter — because that single hop already reveals which journeys are slow. Then follow one full journey end to end, choosing the one that causes the most arguments rather than the one that is easiest. Use automatic instrumentation first and write custom spans only for what is genuinely specific to you. And propagate the trace identifier into your logs from day one: without that shared thread you cannot jump from a log line to its trace.
Is OpenTelemetry mature enough for production tracing?
Yes. It graduated at the CNCF in May 2026, the foundation’s highest maturity level and the same status held by Kubernetes, and it is the second most active project there. Two things are worth watching: continuous profiling is being folded in as a fourth signal, and work is progressing on capturing telemetry from the operating system without touching application code.
Why does tracing sometimes make my bill explode?
Usually not because of the traces themselves, but because of cardinality — the number of unique label combinations, which grows by multiplication rather than addition. Attaching a customer ID to a metric turns fifty endpoints into five hundred thousand series if you have ten thousand customers. The rule: if a value can be unbounded, keep it off metrics and put it on the trace or the log, where you pay per event instead of per combination.
What I take from this
Go back to the call with six teams and six green dashboards.
Nobody in that room was lying and no dashboard was wrong. Each one was accurately describing its own piece. The failure was that no instrument existed for the thing that was actually broken — the path between the pieces — so six accurate views added up to no understanding at all.
That is what a trace is: not more data, but the first instrument that describes the space between systems instead of the systems themselves.
If you take one action into this week: make sure your logs carry the trace identifier. It costs almost nothing, nobody does it until they have needed it once, and it is the difference between jumping to the answer and starting the archaeology again. 🧵
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.