Every network I built for twenty years was optimized for the same thing: moving a lot of data, on average, reliably.
The network inside an AI cluster is optimized for something else entirely, and until you internalise the difference, none of the vendor conversations make sense. It does not care about the average. It cares that no single exchange is ever late, because one late exchange stops every accelerator in the building.
🔑 The short version
- Accelerators work in lockstep: compute, exchange, compute. The slowest exchange sets the pace for everyone.
- RDMA lets one machine read another’s memory without interrupting its processor. That is the whole trick.
- Two roads implement it, InfiniBand (a dedicated network) and RoCE (RDMA over Ethernet).
- The honest difference is not speed. It is who can operate it in eighteen months.
- The number that belongs in every proposal and never appears is tail latency, not average bandwidth.
Why accelerators need a different network
Go back to the kitchen. The accelerators are cooks doing the same repetitive task in parallel. But they are not independent, because a model too big for one of them is split across many, so each cook constantly needs what the cook beside them just produced.
They compute. They stop. They exchange. They compute again. Thousands of times.
Which means the interesting property of that network is not how much it can carry. It is how long the worst handoff takes. If one exchange out of a thousand is slow, every accelerator waits for it, and you are paying for all of them to wait.
This is the mental adjustment. In the networks I grew up with, a slow packet was an annoyance absorbed by buffers and retries. Here, a slow packet is idle silicon at industrial scale.
RDMA: passing the ingredient without bothering the chef
Normally, when one machine sends data to another, the receiving machine’s processor has to get involved: interrupt what it was doing, copy the data, hand it to the right process. That interruption is cheap once and ruinous a million times.
RDMA removes it. One machine writes directly into another machine’s memory, and the second machine’s processor is never disturbed.
In kitchen terms, instead of tapping the cook on the shoulder every time you bring an ingredient, you place it directly in their bowl and they find it there when they reach for it. No interruption, no context switch, no waiting for attention.
That is the entire concept, and everything else in this article is about which road you take to get it.
The two roads, and how they actually differ
| InfiniBand | RoCE (RDMA over Ethernet) | |
|---|---|---|
| What it is | A dedicated network built for this, decades old in high-performance computing | The same idea running over the Ethernet you already know |
| Congestion control | Built in from the start. It was designed for this problem | Bolted on, and it has to be configured correctly |
| Who operates it | A specialised skill. Scarce and expensive to retain | Your existing network team, with training |
| Supplier choice | Effectively concentrated | Several vendors compete |
| Where it fails | Rarely, but when nobody in-house understands it | When the congestion settings were never tuned |
Both work. Genuinely. Which is exactly why choosing on a benchmark is the wrong method. The benchmark will not tell you which one your team can still operate after the consultant leaves.
And here is where I had already been
None of this is as new as it is sold, and I say that as someone who lived the earlier version.
Years ago I worked on a national voice network where the entire discipline was about the worst case, not the average. A call that arrives late is not a call that arrives, because quality collapses, people hang up, and no amount of aggregate bandwidth compensates. We were already engineering for tail latency; we just did not use that phrase.
And the second thing that transfers directly. I spent years responsible for a quality number whose controlling policies belonged to the connectivity provider. We could look. We could not touch.
A metric you answer for but cannot influence is not a metric. It is theater.
That matters here more than it looks. If you rent this infrastructure from a provider, which most organizations in Latin America will, ask what you will be able to see about the interconnect, and what you will be permitted to change. Those are different questions and both belong in the contract.
The number that should be in every proposal
Proposals lead with peak bandwidth, because peak bandwidth is a large impressive number. It is also close to irrelevant here.
What determines whether a training run finishes in nine hours or fourteen is the latency of the slowest exchanges under real load, not the fastest and not the average. Ask for the 99th percentile, under congestion, with a workload that resembles yours.
If the answer is a peak figure and a smile, you have learned something useful about the proposal.
Comparison of the two approaches and how congestion is handled: RoCE versus InfiniBand and RoCEv2 explained. Congestion notification in these fabrics: understanding DCQCN. Practical guide to the options: GPU networking guide.
How to choose without repeating somebody else’s opinion
- Start from who operates it. If nobody in your organization has run a specialised fabric and you cannot hire for it locally, that answers most of the question.
- Ask for tail latency under congestion, with your workload shape. Not peak, not average.
- Ask who tunes congestion control, and when. On the Ethernet road this is the difference between working and appearing to work.
- Test at scale, not with a demo rack. One of everything always works, which is what a demo is for.
- If you are renting it, what can you see, and what can you change? Visibility without authority is the trap I already fell into once.
What to ask before signing
| The question | Why it matters |
|---|---|
| What is the p99 latency under congestion? | The number that decides your training time. Peak bandwidth does not |
| Who tunes congestion control, and who does it in year two? | The most common cause of a fabric that quietly degrades |
| Can we hire that skill locally? | InfiniBand expertise is scarce in this region and expensive to keep |
| Can we test with our own workload at real scale? | A demo rack proves nothing about lockstep behaviour |
| If we rent: what interconnect telemetry do we get? | Without it, a slow fabric looks exactly like slow code |
Frequently asked questions
What is RDMA and why does an AI cluster need it?
RDMA lets one machine write directly into another machine’s memory without interrupting its processor. In an AI cluster the accelerators work in lockstep, computing, exchanging results and computing again, thousands of times, so the cost of interrupting a processor on every exchange becomes ruinous. Removing that interruption is what makes training across many accelerators viable.
What is the difference between InfiniBand and RoCE?
Both deliver RDMA. InfiniBand is a dedicated network designed for this from the start, with congestion control built in, operated by a specialised and scarce skill set. RoCE runs the same idea over Ethernet your team already understands, but its congestion control has to be configured correctly. Both work; the real difference is who can operate the result in eighteen months.
Why is average bandwidth the wrong metric here?
Because accelerators work in lockstep and the slowest exchange sets the pace for all of them. A network can have excellent average throughput and still stall a cluster if its worst-case exchanges are slow. The figure to demand is tail latency, the 99th percentile under congestion with a workload resembling yours, not peak bandwidth, which is the number proposals lead with because it is large and impressive.
What usually goes wrong with RoCE?
Not the technology. It is the configuration. Congestion control left at defaults works beautifully in a pilot with a handful of nodes and degrades under real load. It is the classic shape of a proof of concept that was never designed to fail. Ask specifically who will tune it, and what happens the day that person changes job.
What should I ask if I rent this infrastructure instead of building it?
Two things that are different questions. What telemetry about the interconnect you will be able to see, and what you will be permitted to change. Without visibility into the fabric, a slow network looks exactly like slow code and you will spend weeks optimising the wrong thing. Without authority to change anything, you are accountable for a number you cannot move, which is not a metric but theater.
What I take from this
The part I find genuinely interesting is how little of this is new.
Engineering for the worst case rather than the average, obsessing over the late arrival instead of the throughput, discovering that the bottleneck is the handoff and not the work. That is voice networking, and it is thirty years old. The hardware changed completely; the discipline did not.
What did change is the cost of getting it wrong. A late packet used to mean a call that sounded bad. Now it means several hundred thousand dollars of silicon standing still, politely, waiting.
So if you take one question into the next proposal meeting, ask for the 99th percentile under congestion. Watch what happens. The reaction tells you more than the number would. 🔌
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.