When somebody says “the cluster network,” they are describing something that does not exist.
There is no cluster network. There are six, each with a different job, a different failure mode and, usually, a different owner. And most of the confusion in these projects comes from treating them as one thing with one diagram.
🔑 The short version
- It is not one network. It is six, and only two of them are what people mean when they say “the network”.
- The first split that matters: inside the rack versus between racks. Different technology, different rules.
- Accelerators are wired into rails — each one gets its own lane out, so they never queue behind each other.
- The dangerous property is not complexity. It is that a change in the middle reaches everything at once.
- What you actually need is not the diagram. It is the map of who owns which network.
First: it is not one network, it is six
Here they are, in the order they will cause you trouble:
| The network | What it carries | What happens if it fails |
|---|---|---|
| Between accelerators, inside the rack | Intermediate results, constantly | Everything slows down. This is the one that decides training time |
| Between racks | The same, once you outgrow one rack | Same, at a larger scale and with more ways to be late |
| Storage | Training data feeding the accelerators | Expensive silicon sits idle waiting to be fed |
| Management | Orchestration, telemetry, control | You lose visibility. Everything keeps running, blind |
| Out-of-band | Access to the hardware when it is unwell | You cannot reach a machine precisely when you need to |
| Corporate / external | Users, APIs, the outside world | The part your existing team already knows how to run |
Notice the fifth row, because it is the one that gets cut from budgets and it is the one I would fight for. Out-of-band access is what lets you reach a machine that has stopped answering. Skipping it is cheap until the night you need it, and then it costs a trip to the data center.
Inside the rack and outside the rack are two different worlds
This is the distinction that makes the rest make sense.
Inside the rack, accelerators are connected by a dedicated, extremely fast fabric designed for one thing: letting them exchange data as if they shared memory. It is short, it is proprietary, and it is why a rack-scale system is sold as a single object rather than as parts.
Outside the rack, you are back in a world you recognise — switches, cables, topologies — but tuned for a workload that punishes lateness rather than rewarding throughput.
The consequence for planning: growing inside a rack and growing beyond it are completely different purchases. The first is nearly free in complexity; the second brings a whole network to design, operate and staff. Where that boundary sits should be an explicit decision, not something you discover.
The rails: one lane each, on purpose
Here is the piece that looks odd on a diagram until you know why it exists.
Each accelerator gets its own lane out of the machine — its own path to the switching layer, rather than sharing one big pipe with its neighbours. That arrangement is called a rail.
The reason is the same one that governs everything here: accelerators work in lockstep, so anything that makes one wait makes all of them wait. Sharing a pipe means they queue behind each other, and a queue is just latency that arrives unevenly.
It is the opposite instinct to the networks I grew up building, where aggregating traffic into a fat trunk was the efficient answer. Here, aggregation is the thing you are trying to avoid.
How far this actually scales
The honest framing for a product decision: the first rack is a purchase. The second is a project. Everything about operating this changes when the traffic has to leave the box — the skills, the monitoring, the failure modes, the vendor conversations.
Which is why the most useful question at the start is not how much compute you need. It is are we going to outgrow one rack, and when? If the answer is no, a great deal of this article stops applying to you.
The fabric inside the rack: NVIDIA on scale-up networking. Logical and physical layouts of a rack-scale system: reference architecture, logical and physical topologies. Fabric options in general: AI fabric topologies.
The day one change in the middle reached everything
Years ago, on a corporate network, a protocol was in use that propagates configuration from a central point to every switch automatically. It exists to save work, and it does.
Somebody made a change in the middle. It replicated everywhere, exactly as designed. And it was wrong.
Not one switch misconfigured — all of them, simultaneously, correctly propagated. The mechanism worked perfectly and that was the problem.
I bring it up here because AI fabrics are built on the same premise: centralized, homogeneous, configured from one place. The property that makes them manageable is the same one that makes a mistake arrive everywhere at once.
So the question I would put to any vendor proposing this: what is the blast radius of one wrong change, and how do we roll it back? The answer is rarely on the slide, and it is the difference between a bad afternoon and a bad week.
The map is worth more than the diagram
Every proposal comes with a topology diagram. Beautiful, symmetric, colour-coded.
What none of them include is the thing you will actually need at two in the morning: who owns each of those six networks.
In practice they end up split — the rack fabric belongs to the vendor or the integrator, storage to one team, management to another, out-of-band to whoever remembers it exists, and corporate to the network team. When something is slow, the first hour goes to establishing which of the six is at fault and who to call.
I know that hour well. I spent years accountable for a quality number whose controlling policies lived with a third party — we could look, we could not touch, and when the fault came from their side, that side showed us nothing.
A metric you answer for but cannot influence is not a metric. It is theater.
So before the diagram, ask for the map. Six networks, six owners, six phone numbers. That page is worth more than the topology, and nobody will produce it unless you ask.
What to ask before signing
| The question | Why it matters |
|---|---|
| Are we staying inside one rack, or growing beyond it? | The first is a purchase, the second is a project. Decide it, do not discover it |
| Who owns each of the six networks? | This is the page that shortens incidents. It never comes unrequested |
| Is out-of-band access included? | The first thing cut from a budget, and the one you need at 3am |
| What is the blast radius of one wrong change? | Centralized configuration propagates mistakes as efficiently as fixes |
| What telemetry do we get from the fabric? | Without it, a slow interconnect looks exactly like slow code |
Frequently asked questions
How many networks does an AI cluster have?
Six, with different jobs and usually different owners: the fabric between accelerators inside the rack, the network between racks, storage, management, out-of-band access, and the corporate or external network. Only the first two are what people normally mean by “the cluster network”, and treating all six as one thing is where most of the confusion in these projects starts.
What is the difference between inside the rack and outside the rack?
Inside, accelerators are joined by a dedicated, very short, very fast fabric that lets them exchange data almost as if they shared memory — which is why a rack-scale system is sold as one object rather than as parts. Outside, you are back to switches, cables and topologies, but tuned to punish lateness rather than to maximise throughput. Growing within a rack is nearly free in complexity; growing beyond it is an entire network to design and staff.
What is a rail in GPU networking?
It is the arrangement where each accelerator gets its own lane out of the machine instead of sharing one large pipe with its neighbours. It exists because accelerators work in lockstep: anything that makes one wait makes all of them wait, and sharing a pipe means queueing behind each other. It is the opposite of the aggregation instinct that serves you well in conventional networks.
Why is out-of-band access worth insisting on?
Because it is how you reach a machine that has stopped answering. It is typically the first line cut from a budget, since everything works without it right up until the night it does not — and then the alternative is physically travelling to the data center. It is cheap insurance against the exact scenario it covers.
What should I ask about the topology before signing?
Whether you are staying inside one rack or growing beyond it, since those are completely different commitments. Who owns each of the six networks, which is the page that shortens incidents and never arrives unrequested. Whether out-of-band access is included. What the blast radius of one wrong change is, because centralized configuration propagates mistakes as efficiently as it propagates fixes. And what telemetry you get from the fabric itself.
What I take from this
The word “topology” makes this sound like a drawing problem, and it is not.
The drawing is the easy part — vendors produce beautiful ones. What decides how this goes is who answers when one of the six is slow, and whether the person who can see the problem is also permitted to fix it. That is an organizational question wearing a technical costume, and it is the same one I have been running into for twenty years with completely different hardware.
The technology changed. The thing that breaks projects did not.
So if you take one page out of the next proposal meeting, make it the boring one: six networks, six owners, six phone numbers. Nobody will hand it to you, and it will save you more hours than any diagram. 🗺️
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.