The promise sounds like something you should be suspicious of: see what your applications are doing without touching them. No instrumentation, no code change, no deployment.
It is real. And the interesting part, for me, is not the technology — it is who in your organization is allowed to turn it on.
🔑 The short version
- It observes from underneath the applications, so nobody has to modify code.
- It is not dangerous by design: the programs are checked before they run and cannot bring the machine down.
- The cost is small but not zero, and “small” stops being small at cluster scale.
- It sees connections, latency and syscalls. It does not see your business logic.
- The real obstacle is not technical. It is that it needs privileges nobody wants to hand out.
What it is, in operational terms
Forget the acronym. Practically: it lets small, safe programs run in the lowest layer of the operating system and report what is happening there — which processes talk to which, how long a connection took, what the machine is doing when it appears to be doing nothing.
The consequence that matters: you get visibility into applications nobody instrumented, including the ones whose source code left with an engineer four years ago.
Anyone who has inherited a system nobody wants to touch understands immediately why that is valuable. I have worked with plenty of them, and “we cannot instrument that, it is too fragile” is a sentence I have heard in every organization I have been in.
Why it is not dangerous by design
The first reaction from any competent operations person is correct: running code in the lowest layer of the operating system sounds like a spectacular way to take down a machine.
The design anticipates it. Before any of those programs runs, it is checked — the system refuses anything that could loop forever, access memory it should not, or fail to terminate. A program that does not pass simply never runs.
So the honest risk profile is not “it might crash the kernel”. It is more mundane and more likely: overhead, and the wrong people having a very powerful lens.
What it costs, with numbers
Every vendor quotes the first column. The honest framing is the multiplication: a very small cost per event, multiplied by an enormous number of events, is no longer very small.
On a cluster where accelerators work in lockstep and the slowest node sets the pace for everyone, a few percent of overhead on the wrong machine is not a rounding error — it is the straggler that holds up the whole training run.
And now the part I care about: who can grant that
Here is where this stops being a technology conversation.
To use this you need privileged access to the operating system of every machine you want to watch. Not application permissions — system-level privilege, on production hosts.
In every regulated organization I have worked with, that permission is guarded by a team that is not the one asking for it, and for good reasons. The request “give us privileged access to all production hosts so we can see better” is not a request that gets approved in a meeting.
So the pattern is predictable: the pilot works beautifully in a lab, the value is obvious, and the project stalls for four months in a security review nobody scheduled.
Visibility you are not permitted to enable is not visibility. It is a proposal.
Which is the same lesson I learned reporting on a quality metric whose controlling policies belonged to a third party: seeing and being allowed are different problems, and the second one is rarely on the project plan.
What it sees, and what it does not
| Sees | Does not see |
|---|---|
| Which process talks to which, and how long it took | Why your code made that decision |
| Network behaviour, connections, retransmissions | The meaning of a business transaction |
| System calls, file and resource access | The internal state of your application |
| Everything, including software you cannot modify | Anything that requires knowing your domain |
That last row is the honest summary. It is superb at “what is happening on this machine” and useless at “why did this customer’s payment fail”. It complements instrumentation; it does not replace it, and anyone selling it as a replacement is overselling.
Where it fits in an AI cluster
Specifically here, its best use is not watching applications — it is watching the fabric.
In a GPU cluster the recurring question is whether a slow training run is slow because of the code or because of the network between accelerators. Without visibility into the interconnect, those two look identical, and teams spend weeks optimizing the wrong one.
Observing from underneath answers that without asking the machine-learning team to instrument anything — which matters, because they generally have neither the time nor the inclination to add observability to a training script.
What to ask before signing
| The question | Why it matters |
|---|---|
| Who grants privileged access, and have we asked them? | The real blocker. Ask in week one, not month four |
| What is the measured overhead on a workload like ours? | Per event it is small. Multiplied it may not be |
| What happens to overhead under peak load? | The cost grows when you can least afford it |
| Does this replace instrumentation or complement it? | If the answer is “replace”, they are overselling |
| Can we see the interconnect, not just the hosts? | In a GPU cluster that is the question worth paying for |
What the technology is and how the safety check works: what is eBPF. State of adoption: eBPF in 2026. Overhead in production systems: performance monitoring with eBPF.
Frequently asked questions
What does eBPF do for observability?
It lets small, verified programs run in the lowest layer of the operating system and report what is happening there — which processes talk to which, how long connections took, what the machine is doing when it appears idle. The practical consequence is visibility into applications nobody instrumented, including software whose source code left with an engineer years ago.
Is it dangerous to run code in the kernel?
Less than it sounds, because it is checked before it runs: anything that could loop forever, access memory it should not, or fail to terminate is refused outright. The honest risks are more mundane — overhead, and giving a very powerful lens to people who should not have it. The crash scenario is the one the design specifically prevents.
How much overhead does it add?
Small per event, which is the number vendors quote, and the multiplication is what people forget: a tiny cost times an enormous number of events is no longer tiny. On a GPU cluster it matters more than elsewhere, because accelerators work in lockstep and the slowest node sets the pace for all of them. Ask for overhead measured on a workload like yours, and specifically under peak load.
What is the real obstacle to adopting it?
Not the technology — the permission. It requires system-level privilege on production hosts, and in regulated organizations that access is guarded by a team that is not the one asking for it. The predictable pattern is a pilot that works beautifully in a lab and then stalls for months in a security review nobody scheduled. Ask who grants that access in week one.
Does it replace instrumenting my code?
No, and anyone selling it that way is overselling. It is excellent at what is happening on this machine and useless at why this customer’s payment failed. It sees connections, latency and system calls, not business logic or your application’s internal state. It complements instrumentation, particularly for software you cannot modify.
What I take from this
The technology here is genuinely elegant, and it is not the interesting part.
What makes this worth a Product conversation is that it exposes, very cleanly, a gap that exists in every organization: the team that needs the visibility and the team that can authorize it are different teams, and nobody puts that on the project plan.
I have watched excellent tools die in that gap. Not rejected — just never scheduled, because the technical evaluation finished and the permission conversation never started.
So if you are evaluating this: have the security conversation before the technical one. It costs a meeting, and it is the difference between a four-month pilot and a four-month wait. 🔍
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.