The data centers I grew up in were cold, loud and reassuringly boring.
Racks of servers drawing a few kilowatts each. Air conditioning that mostly worked. Raised floors with cables underneath, and a physical world that failed in ways you could learn: a power supply, a fan, a link, sometimes rats chewing something they should not have.
Everything I learned in those rooms still applies. It is just no longer enough — and the gap is not about software. It is that the electricity bill and the temperature became the design constraints.
🔑 The short version
- A CPU is a chef who can do anything. A GPU is thousands of cooks who all chop, at the same time.
- Training a model is mostly the same simple operation, repeated, independently — which is exactly what thousands of cooks are for.
- One GPU on its own is nearly useless at this scale. What matters is how they talk to each other.
- The binding constraint is no longer floor space. It is kilowatts per rack and getting the heat out.
- Before any quotation, the real question is how much power a rack in your site can take today.
What a GPU is, said plainly
A CPU has a few very capable cores. It is good at doing different things one after another: run this, then decide, then run that. It is the chef — trained for everything, and there are only a handful of them.
A GPU has thousands of simple cores that all perform the same operation at the same time. They only chop. But there are thousands, and they never disagree about what to do next.
So why does AI land on the side of the cooks? Because training a model is, underneath, multiplying matrices. Millions of times. Simple operations, all identical, and crucially independent of one another — no cook has to wait for the one beside them. It is chopping onions at industrial scale, and there is no chef alive who can compete.
That is the whole idea. Everything else is engineering around it.
Why one GPU on its own does almost nothing
Here is where the intuition from ordinary data centers breaks.
A model too large to fit in one accelerator has to be split across many. Which means those accelerators have to exchange intermediate results constantly — not occasionally, constantly, in lockstep. They compute, they synchronize, they compute again.
The consequence is uncomfortable and it governs everything: the slowest GPU sets the pace for all of them. Sixty-three perfect accelerators and one running hot means sixty-four accelerators running at the slow one’s speed, while your average utilization graph looks excellent.
So the interesting engineering is not the chip. It is the network between chips, and it is not the network in your office: it is not optimized for moving a lot of data on average, but for making sure no single exchange is ever late. The mean does not matter. The worst case does, because the worst case stops everyone.
The five things that genuinely change
| What changes | Data center you know | AI facility |
|---|---|---|
| Power per rack | A handful of kW | Tens of kW, and rising fast |
| Cooling | Air, usually enough | Air stops being enough. Liquid enters the room |
| What you buy | Servers, one at a time | Racks as a unit. It is not sold in pieces |
| The bottleneck | Storage or the application | The interconnect between accelerators |
| Failure mode | Loud. Something dies | Quiet. Everything slows down and nothing alerts |
The last row is the one that concerns me professionally, because it defeats every monitoring habit built over twenty years. We learned to watch for things stopping. This hardware mostly does not stop — it throttles to protect itself, your job takes fourteen hours instead of nine, and nothing turns red.
What to ask if you are evaluating one of these
| The question | Why it decides things |
|---|---|
| How many kW per rack does the site support today? | Not after an upgrade. Today. This is usually the whole answer |
| How does the heat leave the room? | If the answer is only air, ask again at the density being proposed |
| Are we training, or serving a trained model? | Completely different requirements. Serving is far more forgiving |
| What happens when one accelerator degrades? | If nobody can answer, nobody is watching the failure mode that matters |
| Who operates it after handover? | During the pilot, the vendor’s best engineer. Afterwards, your team |
For Latin America there is a sixth, and it is usually the real one: are we building this, or renting it from a sovereign cloud or a regional colocation? Most organizations here will arrive at this hardware through somebody else’s facility, and that is a sound decision rather than a lesser one.
Rack density trends and cooling limits: Uptime Institute on rising rack density and its field report on cooling. What a rack-scale system demands from a facility: readiness analysis for rack-scale AI systems. Scale of a large training run: Hugging Face on Llama 3.1.
Frequently asked questions
What is the difference between a CPU and a GPU?
A CPU has a few very capable cores and is good at doing different tasks one after another. A GPU has thousands of simple cores that perform the same operation simultaneously. The CPU is the chef who can cook anything; the GPU is thousands of cooks who only chop. Neither is better — the CPU wins on varied, sequential work and the GPU wins on repetitive, parallel work such as training a model.
Why does AI need GPUs specifically?
Because training a model is, underneath, multiplying matrices millions of times. Those operations are simple, identical to each other, and above all independent — none has to wait for another to finish. That is precisely the shape of work thousands of parallel cores are built for, and it is where they beat a general-purpose processor by orders of magnitude.
Can I put GPUs in my existing data center?
For experimentation and small-scale inference, often yes. For training large models, rarely without construction work — and the limit is not floor space, it is the electrical power available and the ability to remove the heat. Before any quotation, the question to settle is how many kilowatts per rack the site supports today, and how that heat leaves the room.
Why doesn’t a single GPU solve the problem?
Because a model too large to fit in one accelerator must be split across many, and those accelerators then exchange intermediate results constantly, in lockstep. That makes the interconnect between them the real bottleneck, and it means the slowest accelerator sets the pace for every other one. Sixty-three healthy GPUs and one throttling produces a cluster running at the slow one’s speed.
How does this hardware fail?
Quietly, which is the problem. Conventional infrastructure tends to fail loudly — a process dies, a disk fills, something turns red. Accelerators under thermal stress keep working and reduce their own clock speed to protect themselves. Nothing crashes, no alert fires, and the job simply takes longer. Every monitoring habit built around things stopping misses it completely.
What I take from this
I keep coming back to those cold, boring rooms.
Nothing I learned there became wrong. Networks are still the foundation, physical failures still happen, and the discipline of knowing what you do not know still saves projects. What changed is which variable decides the outcome. For twenty years it was capacity — servers, storage, bandwidth. Now it is electricity and heat, which used to be somebody else’s department entirely.
And the professional consequence is the part I would underline: the failure mode moved from loud to silent, and our instincts were all trained on loud.
So if you are being asked to evaluate one of these projects, start where it actually gets decided: how many kilowatts a rack can take, and where the heat goes. That conversation costs a meeting and settles more than a month of technical comparison. ⚡
May the systems be with you. ✦
Featured image generated with artificial intelligence.
Ethel Méndez
Senior Product Manager in B2B observability and networking, with 20 years of field work, NOC and managed services across Latin America. I write what I learned running real systems, not what I read in a course.