Here is the sentence that ends most observability budget meetings badly: “but our traffic didn’t grow.”
It didn’t. That was never what you were paying for.
I work in product for observability services in Latin America, which means I sit on the uncomfortable side of these conversations — the one where a customer has already signed, the invoice arrived, and somebody has to explain a number nobody forecast. And the explanation is almost always the same. You are not billed for how many people used your system. You are billed for how much your system talked about itself.
Add AI agents to your product and your system becomes extremely talkative.
🔑 The short version
- Observability cost tracks telemetry volume, not traffic. AI generates far more telemetry per interaction.
- A single agent loop can emit dozens of times more spans than a conventional API call.
- Large accounts report year-over-year increases of 30–50%, and 40–200% once LLM monitoring is switched on.
- Three levers actually move the number: tail-based sampling, data tiers, and standardizing on OpenTelemetry.
- The strongest lever is the one nobody lists: portability is negotiating power, and your vendor knows whether you have it.
The unit you pay for changed, and nobody sent a memo
Observability is largely priced on what you ingest and index. That model was designed for a world where a web request produced a handful of spans.
An agent does not answer in one shot. It loops: it thinks, calls a tool, reads the result, thinks again, calls another tool. Every turn of that loop emits its own span — with its prompt, its response, its token counts and its latency. You are not doing more things. Each thing is now leaving ten times the trace behind it.
Multiply that by concurrent users and the ingestion graph goes vertical. Nothing broke. The system is working exactly as designed, and that is the part people find hardest to accept.
Treat those numbers as third-party estimates rather than hard data — but the order of magnitude is real, and it comes from an argument worth repeating: auditing an agent requires capturing every intermediate step, not just the final answer. The traceability you need for governance is the same traceability that inflates your bill. You cannot have one without the other, and no vendor is going to open with that.
What the increases actually look like
Again: these are figures reported by other outlets, useful for sizing the problem, not for quoting as your own benchmark. What matters is the shape. The increase is not linear and it is not proportional to anything your business tracks. That is precisely why it surprises finance, and why “our traffic didn’t grow” keeps getting said in rooms where it doesn’t help.
Spans per interaction and the case for capturing intermediate steps: The New Stack on agentic AI observability. Year-over-year increases: byteiota. Mid-market annual spend: oneuptime.
Sampling, in money rather than theory
Sampling is boring until you put it in currency, so here is the arithmetic. It is illustrative — not a benchmark, not a promise.
Take a service emitting 50 million spans a day. Most of them are traces of requests that went fine, look identical to each other, and will never be opened by a human being.
Now keep 100% of error traces and 10% of normal ones. Assume 2% of your traces contain errors:
From 50 million to under 6 million, without losing a single error trace. If your backend bills on ingestion, that reduction lands almost directly on the invoice.
The distinction that makes this work is when the decision happens. Head-based sampling decides at the start of a trace, before anyone knows whether it will fail. Tail-based sampling decides at the end, once the outcome is visible. That is the whole reason tail-based can keep “every error” and “a fraction of the noise” at the same time — and it is the question to ask a vendor, phrased exactly like that.
The second lever: stop paying premium rates for data nobody queries
Not all telemetry deserves the same treatment, and most contracts price it as if it did.
| What it is | How often it’s queried | What it should cost |
|---|---|---|
| Errors and slow traces | During every incident | Full rate. This is what you’re buying |
| Successful request traces | Almost never, and sampled when they are | Cheap tier, or not retained at all |
| Logs older than 30 days | Audits and post-mortems | Cold storage. Slow retrieval is fine here |
| Verbose debug output | The week it was turned on | Should have an expiry date, and rarely does |
The last row is the one I find most often, and it is never anyone’s fault. Somebody raised the logging level during a bad night, the incident closed, and the setting stayed. Two years later it is a line item.
The lever nobody puts in the deck
Here is the one I care about most, because it is not technical at all.
Everything above reduces what you send. This one changes what happens when you ask for a better price.
If your instrumentation is proprietary — vendor agents, vendor SDKs, vendor formats — then leaving is a rewrite of every service you own. Your vendor has modeled that cost, probably more carefully than you have. You are not negotiating. You are asking.
If your instrumentation is OpenTelemetry, moving means changing where the data is sent. That is a genuinely different conversation, and it happens before you open your mouth, because the vendor already knows which of the two you are.
I am not claiming a standard makes migration trivial. Dashboards, alerts and years of accumulated queries do not port themselves, and anyone who tells you otherwise is selling something. What I am claiming is narrower and more useful: standardized telemetry moves you from “we cannot leave” to “leaving would be a project.” Every percentage point you have ever won on a renewal came from that distinction.
What I’d ask before signing
| The question | Why it matters |
|---|---|
| What exactly is the billable unit? | Ingested gigabytes, indexed spans, hosts, custom metrics — the answer changes which lever works |
| Does the price change when we turn on LLM monitoring? | This is where the 40–200% jumps come from. Ask before, not after |
| Can we sample before you bill us, or after? | If sampling happens on your side, you are paying to send data you then discard |
| Can retention differ by data type? | If there is one retention slider, your noisiest data sets the price for everything |
| What happens to our historical data if we leave? | Ask now, in writing. Asking during a renewal is asking too late |
| Do you ingest OpenTelemetry natively, or through a translation layer? | “Compatible” and “native” are very different, and the gap shows up as lost attributes |
Frequently asked questions
Why does AI make observability more expensive?
Because observability is priced on telemetry volume, and AI produces far more of it per interaction. A conventional API call emits a handful of spans; an agent loops through reasoning steps and tool calls, and each turn emits its own span with prompts, responses, tokens and latency. Estimates put a conventional call at 2–3 spans and an agent loop at 40–75. Your traffic can stay flat while your ingestion multiplies.
What is the difference between head-based and tail-based sampling?
Head-based sampling decides whether to keep a trace at the moment it starts, before the outcome is known. Tail-based waits until the trace finishes and decides with the result in hand. That is why tail-based can retain 100% of error traces while keeping only a fraction of successful ones. The trade-off is that it must buffer traces in memory while it waits, which is infrastructure you operate — compare that cost against the ingestion you avoid.
How much can sampling actually reduce a bill?
In an illustrative case — 50 million spans a day, 2% of them errors, keeping all errors and 10% of normal traces — volume drops to under 6 million spans a day, roughly 88% less, with no error trace lost. If your contract bills on ingestion, that reduction translates almost directly. Your own ratio depends on your error rate and how much duplicated success traffic you emit.
Does OpenTelemetry actually lower costs?
Not by itself. What it changes is your position: with proprietary instrumentation, leaving means rewriting every service, and your vendor has priced that. With standardized telemetry, leaving means redirecting where data is sent. Dashboards, alerts and accumulated queries still do not port themselves — but you move from “we cannot leave” to “leaving would be a project,” and that is where renewal discounts come from.
Is this only a Datadog problem?
No. Any platform priced on ingestion and indexing has the same dynamic, including self-hosted stacks, where the cost simply moves to storage, compute and the people maintaining it. The vendor name changes what the invoice looks like, not what drives it. The question to ask is always the same one: what is the billable unit, and what makes it grow?
What I take from this
The uncomfortable part of this topic is that there is no villain in it.
The vendor did not raise prices. Your team did not do anything careless. You instrumented your AI features properly — which is what everyone, including me, told you to do — and proper instrumentation of an agent means recording every step it takes. The bill is the honest consequence of doing the work correctly.
So the decision is not how to spend less on observability. It is which questions you are willing to pay to be able to answer, and which ones you can afford to answer more slowly, or not at all. That is a product decision, not an engineering one, and it is the reason this conversation should start before the invoice arrives rather than after.
If you take one question into your next vendor meeting, make it the first one on that list: what exactly is the billable unit? Surprisingly often, the room does not have a shared answer. 📊
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.