GPU Cluster Networking Requirements

Free Technical Audit

Expert Review

Back in early 2024, I helped a robotics company build a 32-GPU cluster. We spec’d the compute right — H100s, plenty of memory, fast storage. Network? We threw in some cheap 100GbE switches and called it a day. First training run of a 7B-parameter language model: 35% utilization. Nodes were starving for data. That cluster was useless until we ripped out the entire network and replaced it with InfiniBand NDR400. Utilization went to 92%. The lesson stuck: GPU cluster networking requirements are not an afterthought — they are the architecture.

A GPU cluster is a collection of interconnected servers (nodes) each with multiple GPUs, working together on a single problem — typically training large AI models or running simulation workloads (GPU Cluster Explained: Architecture, Nodes and Use Cases). But the GPUs themselves are useless if they can’t talk to each other fast enough. Networking is the circulatory system. If it’s clogged, nothing works.

In this guide, I’ll walk you through the real networking decisions you face when building a GPU cluster: bandwidth, latency, topology, protocol choice, and cost trade-offs. I’ll show you what we tested, what broke, and what actually scales. You’ll leave knowing exactly how to evaluate a cluster’s network — and how not to waste six figures on mistakes I’ve already made.

Why Networking is the Bottleneck (Not Compute)

Most people think the bottleneck is GPU compute. “Just buy more H100s,” they say. That’s wrong. In large-scale distributed training, the bottleneck is almost always how fast gradients can move between GPUs.

Take a model like Llama 3 70B. During training, each GPU computes its own portion of gradients. Then all GPUs must all-reduce those gradients — a collective communication step. A single all-reduce of 0.5 GB of gradients across 64 GPUs on a slow network can take 2 seconds. At 10,000 steps, that’s over 5 hours wasted on communication alone. With a properly designed network, that same operation takes 20 milliseconds. The difference between “I can train this model in 3 days” and “I’ll be dead before it finishes.”

The math is brutal: communication overhead scales linearly with model size and quadratically with the number of GPUs (in a naive ring all-reduce). So as models grow — and they are — networking becomes the dominant cost, both monetary and time.

The Three Dimensions: Bandwidth, Latency, and Congestion

When you evaluate a cluster network, forget marketing specs. Focus on three things:

Bandwidth — the raw bit rate per link. Measured in Gbps. Modern minimum: 200 Gbps per GPU. Better: 400 Gbps. If you’re using H100 GPUs with NVLink 4.0 (900 GB/s intra-node), you need inter-node bandwidth that doesn’t become a wall. For reference, an H100’s PCIe Gen5 x16 provides 128 GB/s — and many clusters use that as the inter-node path via a network adapter. You want at least 200 Gbps per GPU for decent scaling.

Latency — the time to send a message from GPU A to GPU B. In a cluster, this includes serialization, wire time, switch hops, and deserialization. Target: under 10 microseconds for a small message inside the rack. Above 20 µs and things start to choke on small gradient chunks.

Congestion — the real killer. Even with high bandwidth, if two flows collide on a switch link, you get head-of-line blocking. Bad. This is where topology and flow control matter more than raw speed.

I’ve seen clusters with 400 Gbps links that ran at 50% of peak because they used a poor topology with oversubscription. The networking equipment was “high-end,” but the design was dumb.

Topology Choices: Fat-Tree vs. Dragonfly vs. Torus

You have three main topologies for GPU clusters today. Here’s my take after building with each.

Fat-Tree (Clos Network)

The industry standard. It’s a multi-layer tree of switches where each node connects to a leaf switch, leaf switches connect to spine switches, and all paths are equal cost (ECMP). At SIVARO, we use a two-layer fat-tree for up to 512 GPUs. It’s simple, well-understood, and easy to debug.

Pros:

Uniform bandwidth between any two GPUs (non-blocking if you provision correctly)
Simple routing (equal-cost multipath)
Easy to scale by adding spine switches

Cons:

Requires 2x the switch ports for a non-blocking design. For N GPUs, you need N/2 leaf ports * 2 for uplinks. That’s a lot of optics and cables.
Cable management becomes a nightmare at scale.

For most companies building clusters under 1,000 GPUs, fat-tree is the right choice. Don’t overthink it.

Dragonfly

A more exotic topology designed for extreme scale (10,000+ GPUs). Groups of switches form “groups” that interconnect with high-radix links. It minimizes diameter: only two hops between any two nodes in the entire cluster.

Pros:

Lower latency at scale (3–4 hops vs. 5–6 in fat-tree for large clusters)
Fewer total cables — you use high-radix links between groups

Cons:

Hard to debug. Routing is non-trivial; you need adaptive routing algorithms to avoid congestion.
If a link between groups fails, performance degrades unpredictably.

We tested a Dragonfly topology for a 2,048-GPU cluster in 2025. Performance was 10% better on all-reduce benchmarks compared to fat-tree, but we spent two months tuning adaptive routing. Fat-tree would’ve been up and running in two weeks.

My advice: Unless you’re building a supercomputer, skip Dragonfly.

Torus

Used in some HPC systems (like the original Tensor Processing Unit pods). Each node connects to its neighbors in a grid.

Pros:

Low cost — you only use 4–6 ports per node
Deterministic latency between adjacent nodes

Cons:

Bandwidth drops as messages travel far across the torus. Long-haul communications eat many hops.
Non-uniform: some pairs of GPUs are neighbors, others are far. Makes training large models inefficient.

Torus is dead for deep learning. Don’t.

Final call: Fat-tree for almost everyone. Dragonfly only if you’re building a national lab.

Real-World Bandwidth Numbers: What You Actually Need

Let’s get concrete. Here’s a table I use when planning clusters:

Number of GPUs	Bandwidth per GPU (inter-node)	Recommended
4–8	25 Gbps	Ethernet (okay for small models)
16–64	200 Gbps	InfiniBand NDR200 or HDR100
128–512	400 Gbps	InfiniBand NDR400 or Ethernet with RoCEv2
1024+	800 Gbps (per GPU pair)	InfiniBand XDR800 or NVLink across nodes with NVSwitch

These are per-GPU bandwidth numbers. For a node with 8 GPUs, you need a NIC that can handle 8x that — typically a 400 Gbps or 800 Gbps NIC aggregated.

I once spec’d a 128-GPU cluster with 100 Gbps per GPU. All-reduce benchmark showed 120 GB/s aggregate bandwidth — against a theoretical 1.2 TB/s. Waste of money. We upgraded to 400 Gbps and hit 950 GB/s.

How do you measure? Use NCCL tests. Example:

bash
# Install NCCL tests
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make

# Run all-reduce benchmark on 2 nodes with 8 GPUs each
mpirun -np 16 -hostfile hostfile ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 1

Output will show you the bus bandwidth in GB/s. Anything below 80% of wire speed means your network is broken.

The NVSwitch and NVLink Ecosystem (for NVIDIA)

If you’re using NVIDIA GPUs, NVLink changes the game. NVLink is a high-speed GPU-to-GPU interconnect inside a node and, with NVSwitch, across nodes.

In an HGX H100 baseboard (8 GPUs), each GPU has 18 NVLink 4.0 links. Total intra-node bandwidth: 900 GB/s. That’s enormous. But if you need to connect multiple such nodes, you have two options:

NVSwitch inside each node (standard) — you can pair nodes via NVLink bridges or InfiniBand/Ethernet for inter-node.
NVSwitch across nodes (DGX SuperPOD style) — uses NVLink cables to connect multiple HGX bases into a single NVSwitch fabric. This gives near-uniform high bandwidth across up to 256 GPUs.

The latter is expensive. Each NVSwitch adds $150K+ in hardware. But for training models above 100B parameters, it’s necessary. At SIVARO, we built a 256-GPU cluster with NVSwitch. All-reduce latency dropped from 1 ms to 50 µs compared to InfiniBand. That saved us 3 days per training run on a 70B model.

If your budget doesn’t allow NVSwitch, use InfiniBand NDR400. It’s the next best thing.

Ethernet vs. InfiniBand: The No-BS Comparison

This debate is tired. Let me give you the straight story.

InfiniBand:

Purpose-built for HPC/AI. Low latency (sub-1 µs per hop), native RDMA, lossless fabric (credit-based flow control).
NDR400 gives 400 Gbps per port. XDR800 is coming in 2026.
Requires specialized switches and NICs — expensive. A 40-port NDR400 switch costs ~$40K.
Protocols: IB Verbs, but also supports RoCE? No, native IB is better.

Ethernet:

Commodity. You can use 400 Gbps Ethernet switches from Cisco, Arista, Mellanox (now NVIDIA).
Requires RoCEv2 (RDMA over Converged Ethernet) to get RDMA. PFC (Priority Flow Control) needed for lossless. PFC is fragile — one misconfigured switch and you get deadlocks.
Latency: 2–5 µs per hop. Worse than IB, but often good enough.
Cost: 30–40% cheaper for the same bandwidth.

Which one for GPU clusters?

If you’re building a cluster for training only and your model fits in 1–2 nodes (8–16 GPUs), Ethernet is fine. For anything larger, InfiniBand wins on performance and reliability. I’ve seen Ethernet clusters with RoCEv2 that took weeks to tune PFC thresholds. InfiniBand just works.

But here’s a contrarian take: For inference clusters, Ethernet is better. Inference is latency-sensitive but bursty. You don’t do all-reduce across 512 GPUs. You deploy models on single GPUs and route requests. Ethernet’s ecosystem and tooling (tcpdump, iperf, standard monitoring) make debugging easier.

So choose based on workload, not fanboyism.

Case Study: SIVARO's 128-GPU Cluster in 2025

We built a cluster for a customer doing medical imaging AI. Requirement: train a 20B parameter vision transformer in 48 hours.

Hardware:

16 nodes x 8 H100 GPUs each (128 GPUs total)
Networking: Each node had 1x ConnectX-7 Dual-port 400 GbE NIC (RoCEv2) connected to 2x leaf switches.
Topology: Two-tier fat-tree, 8 leaf switches, 4 spine switches. No oversubscription — every leaf-to-spine link was 400 Gbps.

Initial benchmark with NCCL all-reduce: 650 GB/s aggregate (theoretical peak with 128 GPUs * 200 Gbps = 3.2 TB/s? Wait — per GPU 400 Gbps is 50 GB/s. So 128 * 50 = 6.4 TB/s if fully utilized. But overhead cuts it. We got 650 GB/s, which is only 10%. Something was wrong.

Diagnosis: NICs were in different PCIe slots — some Gen4, some Gen5. Also, RoCEv2 PFC settings were different across switches. We standardized PCIe slots (Gen5 x16 for every NIC) and tuned PFC. Benchmarks jumped to 4.8 TB/s — 75% efficiency. Acceptable.

Lesson: The network is not just the wire. It’s the PCIe slot, the switch configuration, and the firmware tuning. What is the best option to setup on premise GPU cluster for ... has a thread where someone spent three months on PFC tuning. Don’t be that person.

GPU Cluster Benchmark Comparison: What to Measure

When you evaluate different clusters (cloud vs. on-prem, different switch vendors), you need a gpu cluster benchmark comparison that matters. Don’t run synthetic flops. Run:

NCCL all-reduce bandwidth — this is the single most important metric for training. Use nccl-tests with large message sizes (128 MB – 1 GB). Compare per-GPU bandwidth. Your goal: above 400 Gbps (50 GB/s) for modern NICs.
All-to-all bandwidth — for data-parallel training with sharded models. Use alltoall_perf. This stresses the fabric more.
End-to-end training throughput — run a real model (e.g., GPT-2 1.5B) and measure tokens per second. That’s the only number that counts.

Here’s a script we use to benchmark a new cluster:

python
# benchmark_gpu_cluster.py
# Run on all nodes with MPIs
import subprocess
import json

result = subprocess.run(
    ['mpirun', '-np', '128', '--hostfile', 'hosts.txt',
     './build/all_reduce_perf', '-b', '128M', '-e', '128M', '-f', '2', '-g', '1'],
    capture_output=True, text=True
)
print(result.stdout)
# Parse output to extract bus bandwidth
# e.g., 128M 1380 125.12 128.45

Output should parse into a table. We store results in a dashboard.

For public cloud vs. on-prem: I’ve tested Vast.ai (Vast.ai: Rent GPUs) — their networking is hit-or-miss. Some nodes are on 10 Gbps Ethernet, others on 100 Gbps. You never know. On-prem gives you control. Cloud gives you flexibility. Your call.

GPU Cluster Cost for Deep Learning: Network is 30-40%

If you’re pricing a cluster, don’t just look at GPU cards. The network infrastructure can be 30–40% of total cost. Here’s a breakdown for a 128-GPU cluster (16 nodes, 8 GPUs per node):

Component	Cost (USD)	Percentage
GPUs (128x H100 at $30K each)	$3,840,000	65%
Servers (16x dual-socket, 512 GB RAM)	$640,000	11%
Networking (switches, NICs, cables)	$1,200,000	20%
Storage (NVMe, InfiniBand attached)	$100,000	2%
Cooling, power, rack, installation	$220,000	4%
Total	$6,000,000

That $1.2M for networking is real. At 400 Gbps per GPU, you need 16x dual-port NICs ($3,000 each = $48K), 8 leaf switches ($30K each = $240K), 4 spine switches ($50K each = $200K), and 300+ QSFP56 or QSFP112 cables ($100 each = $30K). Then optics, cooling, and labor. It adds up fast.

If you want to reduce cost, oversubscribe the network. For example, use 2:1 oversubscription at the spine — each leaf has 400 Gbps uplinks for 800 Gbps of leaf bandwidth. That cuts spine ports in half. But your training might suffer. For models with low communication (e.g., vision), it’s fine. For large language models, non-blocking is essential.

I’ve seen companies blow their entire budget on GPUs and have no money left for the network. They end up with a cluster that runs at 40% utilization. That’s worse than buying fewer GPUs with a proper network. 5 Key Considerations when Building an AI & GPU Cluster puts it well: “network is the bottleneck, not compute.” Preach.

Building for Tomorrow: 2026 and Beyond

It’s July 2026. NVIDIA just announced Blackwell Ultra with 3.6 TB/s NVLink per GPU. InfiniBand XDR800 is shipping. Ethernet 800 GbE is around the corner. What does that mean for you?

First, don’t cheap out on network for future-proofing. A cluster built today should be able to handle next-gen GPUs with higher communication rates. That means:

Use at least 400 Gbps per GPU today.
Plan for 800 Gbps in 2–3 years via retiming or cable swaps.
Choose modular switches that support line cards for higher speeds.

Second, look at DPUs (Data Processing Units) like NVIDIA BlueField-3. They offload networking, storage, and security from the CPU. In our latest build, BlueField-3 improved all-reduce latency by 20% because the CPU wasn’t busy with protocol overhead.

Third, consider disaggregated networking. Rather than fixed per-node NICs, you can use shared network accelerators in the rack. It’s new, unproven, but promising for ultra-scale.

The network is not getting simpler. But you can build it right if you treat it as a first-class citizen.

Frequently Asked Questions

1. Can I use regular Ethernet for a small GPU cluster (4–8 GPUs)?
Yes. For a single node or two nodes, 25–100 Gbps Ethernet is fine. You don’t need InfiniBand until you cross 16 GPUs. Just make sure your NICs support RDMA (RoCEv2) and your switches have proper flow control.

2. What’s the minimum bandwidth per GPU for training a 70B parameter model?
Realistically, 200 Gbps per GPU with a non-blocking fat-tree. Below that, communication time dominates. With 100 Gbps per GPU, you’ll lose 30–50% of training throughput.

3. Is InfiniBand worth the extra cost?
If you’re doing large model training (100B+ parameters) or multi-node training for hours at a time, yes. The reliability and performance justify the 20–40% premium. For inference or small models, Ethernet works.

4. How do I test if my network is causing bottlenecks?
Run an NCCL all-reduce benchmark across all GPUs. Compare the achieved bandwidth to wire speed. If it’s less than 50%, you have a problem. Also run nvidia-smi topo -m to check GPU topology and NUMA affinity.

5. Can I use cloud GPU clusters instead of building my own?
Yes. Cloud providers (AWS, Azure, GCP) offer clusters with InfiniBand or EFA. They handle networking. But you pay a premium — 2–3x the cost of on-prem over 1–2 years. Vast.ai: Rent GPUs is cheaper but often uses shared networks that are slower.

6. What’s the role of NVLink vs. network?
NVLink connects GPUs inside a node (or across nodes with NVSwitch). The network connects nodes. They are complementary. NVLink is faster (often 900 GB/s intra-node) but limited to 8 GPUs per domain. The network (InfiniBand/Ethernet) is slower but scales horizontally.

7. How do I avoid congestion losses in Ethernet clusters?
Use PFC (priority flow control) carefully. Enable ECN (explicit congestion notification) and configure DCQCN (Data Center Quantized Congestion Notification). Test with stress traffic. Most failures come from misconfigured PFC causing head-of-line blocking.

8. What about GPU cluster networking requirements for inference?
Inference clusters don’t need all-reduce between nodes. Networking requirements are lower: 25–100 Gbps per GPU is enough. Focus on low latency for request routing and high throughput for model serving.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.