Long before anyone called it observability, I was the monitoring system for a national voice network. Every morning at eight I picked up a Cisco phone, dialed the main number of each switchboard — about ten of them — and wrote “ok” in a Word document.
That document had exactly one signal in it. And it taught me, the hard way, what each of the three signals actually does — because I had one, and I kept discovering what the other two would have told me.
🔑 The short version
- Metrics tell you that something is wrong. Cheap, aggregated, no detail.
- Logs tell you what happened. Full detail, one event at a time, expensive at volume.
- Traces tell you where it happened across services. The one most teams add last and miss most.
- The thing that blows up your bill is cardinality — unique label combinations — and almost nobody sees it coming.
- None of the three is sufficient alone, and the gap between any two of them is where incidents live.
What each one actually is
Skip the textbook definitions. Here is what each signal does for the person on call at two in the morning.
A metric is a number over time. Requests per second, error rate, latency at the 95th percentile. It is small, it is cheap to keep for years, and it answers exactly one question well: is this normal compared to before? What it cannot tell you is which request, which customer, or why.
A log is a recorded event with its details. Timestamp, message, and whatever context somebody thought to include. It answers what exactly happened — and it is the most expensive of the three at volume, because you keep the detail whether you ever read it or not. Which, mostly, you don’t.
A trace follows one request through every service it touches. Each hop is a span with its own duration. It answers where the time went, which in a distributed system is usually the actual question and almost never obvious.
My morning phone call was a metric, and a bad one: one data point per day, binary, with no detail attached. When a call failed I had no log to read and no trace to follow. I had to become both, by hand, starting from zero.
The same incident, told three ways
This is the part that makes the difference concrete. One failure, three signals:
| Signal | What it says | What you do with it |
|---|---|---|
| Metric | “Checkout errors went from 0.2% to 7% at 14:03” | You know there is an incident, and when it started. You still know nothing about the cause |
| Log | “Payment gateway timeout after 30s — order 88213” | Now you know the failure mode. But is it one gateway, one region, or everything? |
| Trace | “Checkout → inventory (12ms) → payment (30,000ms) → timeout” | Now you know exactly which hop is burning the time, and you can stop guessing |
Notice the sequence: the metric wakes you up, the log explains, the trace locates. Teams that only have the first two spend their incidents arguing about which team owns the problem. That argument is what the trace replaces.
The one that quietly costs you money
Here is the concept that separates people who have paid an observability invoice from people who have only read about metrics: cardinality.
Cardinality is the number of unique combinations of labels attached to a metric. And it multiplies, which is the part that catches everyone.
Say you track request latency, and you label it by endpoint. Fifty endpoints, fifty series. Fine. Now somebody adds the customer ID as a label, because it seemed useful during an incident.
Nobody made a bad decision. Somebody added a label that was genuinely helpful once. Cardinality is the only cost in this field that grows by multiplication rather than addition, and it is invisible in code review — the line looks identical whether it adds fifty series or half a million.
The rule I use is blunt and it has never failed me: if a label can take an unbounded number of values, it does not belong on a metric. Customer IDs, order numbers, session tokens, full URLs. That information belongs in a log or a trace, where you pay per event instead of per combination.
Why no single signal is enough
Each one has a blind spot that another one covers, and the blind spots are where outages live.
| If you only have… | What you will miss |
|---|---|
| Metrics | Anything affecting a small group. A national average stays green while a whole region is down |
| Logs | Trends. A latency creeping up over three weeks is invisible one line at a time |
| Traces | Everything not inside a request: batch jobs, cron, queues, and anything asynchronous |
The first row is the one I have watched hurt real organizations. On that voice network, small sites simply disappeared inside the national aggregate — twelve people with no service, and a dashboard reporting excellent availability. The metric wasn’t wrong. It was answering a question nobody had asked.
Where to start, in order
You do not need all three on day one, and trying is how teams end up with an expensive mess nobody trusts.
- Start with metrics on the user journey, not on the infrastructure. Not CPU — “did the payment go through.” Cheap, and it tells you when something is wrong.
- Add traces next, not logs. This is the counterintuitive one. Most teams add logs because they already have them lying around, but traces are what shorten an incident, because they answer where.
- Then bring order to the logs you already produce. Not more of them — fewer, structured, with a retention that somebody actually decided.
- Standardize the format from the first day. OpenTelemetry, so today’s choice of backend is not permanent. Changing tools later should mean changing a destination, not re-instrumenting everything you own.
Definitions and the three-signal model: OpenTelemetry on signals and the project at the CNCF. On how the three relate and where each one costs: Peter Bourgon, metrics, tracing and logging. On choosing what to monitor at all: Google SRE, monitoring distributed systems.
Frequently asked questions
What is the difference between metrics, logs and traces?
A metric is a number over time and tells you that something changed — cheap to store, no detail. A log is a recorded event with its context and tells you what happened — full detail, expensive at volume. A trace follows one request across every service it touches and tells you where the time went. In an incident they arrive in that order: the metric wakes you, the log explains, the trace locates.
What is cardinality and why does it raise my bill?
Cardinality is the number of unique label combinations attached to a metric, and it grows by multiplication rather than addition. Fifty endpoints is fifty series; add customer ID as a label with ten thousand customers and you have five hundred thousand. The practical rule: if a label can take an unbounded number of values — customer IDs, order numbers, session tokens, full URLs — it does not belong on a metric. Put it in a log or a trace, where you pay per event instead of per combination.
Which signal should I implement first?
Metrics on the user journey rather than on infrastructure — not CPU, but whether the payment went through. Then traces, which is the counterintuitive part: most teams add logs next because they already have them, but traces are what shorten an incident, since they answer where the time is going. Bring order to logs third, and standardize the format from day one so changing backends later means changing a destination rather than re-instrumenting everything.
Can I get by with just logs?
You can operate, but you will miss trends and you will overpay. Logs are excellent at explaining a single event and poor at showing a slow degradation — a latency creeping up over three weeks is invisible one line at a time. They are also the most expensive signal at volume, because you retain the detail whether or not anyone ever reads it. Metrics catch the trend cheaply; logs explain the case.
Why do averages hide outages?
Because an aggregate is designed to smooth away outliers, and a small group of affected users is an outlier. A national availability metric can sit comfortably in the green while an entire region has no service, since that region is a small fraction of the total. The fix is not a better average — it is measuring the distribution, or breaking the metric down by the dimension where failure actually clusters, such as site, region or customer segment.
What I take from this
Back to the phone and the Word document.
What that routine had going for it was that somebody checked before the users did. What it lacked was not effort — it was that a single signal, taken once a day, cannot answer more than one question. I knew a switchboard was unreachable. I did not know what had failed, or where, and I had to reconstruct both from nothing while people waited.
Twenty years later the industry sells the three signals as a technology stack. They are not a stack. They are three different questions — is this normal, what happened, and where — and the tooling exists because answering all three by hand does not scale past about ten phone calls.
So if you are deciding where to start: pick the question that hurts most in your last three incidents, and buy the signal that answers it. Buying all three at once, before you know which question you keep failing to answer, is how you end up with an expensive dashboard nobody opens. 📞
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.