GPU Cluster Networking: What Nobody Tells You About Training LLMs at Scale

GPU Cluster Networking: What Nobody Tells You About Training LLMs at Scale

I spent three months in 2025 debugging a training cluster that should have worked. 1,024 H100s. Brand new InfiniBand. Everything spec'd perfectly on paper.

Training kept stalling. Loss wouldn't converge right. GPUs sat idle 40% of the time.

The problem wasn't compute. It wasn't even memory. It was the network fabric. Specifically, I'd ignored the gpu cluster networking requirements for large language models — and those requirements bite differently than any HPC workload you've run before.

Here's what I learned the hard way, so you don't have to.

Why Your Network Is The Real Bottleneck

Most people think training LLMs is GPU-limited. You buy more GPUs, you train faster.

That's wrong after ~256 GPUs. At that point, you're network-bound. The math is brutal: a single forward-backward pass on a 70B parameter model requires shuffling ~560GB of gradients across the cluster. Every. Single. Batch.

If your network can't move data faster than your GPUs can compute, your $10M cluster runs at 60% utilization. That's $4M in GPUs sitting idle.

GPU cluster rental cost for a 1,024-H100 node runs roughly $4,000-6,000/hour depending on vendor. Wasting 40% of that means burning $1,600-2,400/hour on nothing. Over a 3-month training run, that's real money.

The Three Network Killers In LLM Training

All-Reduce Is A Beast

Modern LLM training uses distributed data parallelism (DDP) or fully sharded data parallelism (FSDP). Both rely on all-reduce operations to synchronize gradients.

Here's what happens:

1. Each GPU computes gradients on its shard
2. The network aggregates gradients from all GPUs
3. The result is broadcast back to every GPU

This is a collective operation. It doesn't finish until the slowest GPU has communicated with every other. Distributed systems are only as fast as their slowest link — and with all-reduce, the tail latency of your network directly determines training speed.

Bandwidth Is Just The Start

Everyone obsesses over bandwidth. 200Gbps. 400Gbps. 800Gbps.

Bandwidth matters. But latency matters more for small message sizes, and message completion rate matters most for all-reduce.

I tested two InfiniBand fabrics side by side, both 400Gbps. First one averaged 7.3 Tflops/GPU on a 70B model. Second one hit 9.1 Tflops/GPU. Same switches, same cables, different topology. The second had 40% fewer hops between any two GPUs.

PCIe Is The Hidden Tax

Most clusters connect GPUs to the network via PCIe Gen4 or Gen5. A single H100 can saturate 400Gbps InfiniBand. But if you have 8 GPUs per node sharing 4 NICs, you need to understand the PCIe topology.

I've seen clusters where two GPUs in the same box shared a single PCIe switch's upstream bandwidth. One GPU got 400Gbps. The other got 50Gbps. Good luck converging that.

The Actual gpu cluster networking requirements for large language models

Let me be specific. Here's what I've validated across 5 production clusters since 2024.

Minimum viable network for 128-256 GPUs:

• 200Gbps InfiniBand NDR (or equivalent 400Gbps Ethernet)
• Fat-tree topology with at most 3 hops between any two GPUs
• No oversubscription at any tier

For 512-1024 GPUs:

• 400Gbps InfiniBand NDR
• Dragonfly+ topology (reduces hops from 5 to 2-3)
• Dedicated NIC per GPU (8 NICs per 8-GPU node)

For 2000+ GPUs:

• 800Gbps InfiniBand (available late 2025)
• Multi-tier Dragonfly or 3D Torus
• SHARP in-network computing (aggregates gradients in switches)

If you're wondering about the best gpu cluster configuration for deep learning, I'll say this: don't skimp on the network. A 512-GPU cluster with 3:1 oversubscribed networking will train slower than a 256-GPU cluster with 1:1 non-blocking fabric. I've seen this confirmed by engineers at Mistral and AI21 in private conversations.

How We Fixed Our 40% Idle Problem

Back to that cluster I mentioned. Here's the diagnosis process.

First, we profiled the training step time. Distributed computing systems hide their bottlenecks well. We used NVIDIA's Nsight Systems to trace communication patterns.

The data showed something infuriating: 60% of training time spent in all_reduce waits. Not compute. Not data loading. Network.

We traced it to three issues:

1. Incorrect topology mapping. The InfiniBand subnet manager had assigned routes that crossed 5 switches for some GPU pairs instead of the expected 3
2. Packet drops at scale. At 90% line rate, the switch buffers started dropping packets. Retransmissions killed throughput
3. CPU-side bottleneck. The MPI library was pinning communication threads to the same cores as data loaders, creating contention

Fixes:

• Reconfigured subnet routing to match physical topology
• Enabled flow control and reduced MTU from 4096 to 2048 (counterintuitive, but fewer retransmissions at scale)
• Pinned MPI threads to dedicated cores, away from all GPU operations

After: idle dropped from 40% to 8%. Training speed increased 2.7x. Same hardware.

Ethernet vs InfiniBand: The 2026 Reality

Here's where I'll take a contrarian position.

Most people default to InfiniBand for LLM training. It's been the standard for years. The RDMA (Remote Direct Memory Access) performance is unbeatable, and the collective operations are optimized in hardware.

But in 2026, Ethernet has caught up. Ultra Ethernet Consortium's specification is delivering RDMA over Converged Ethernet (RoCE v2) with performance within 5-10% of InfiniBand at half the cost.

I ran a comparison in March 2026:

• InfiniBand NDR400: $3,200 per port (switch + NIC)
• Ultra Ethernet 400G: $1,700 per port

On a 1,024-GPU cluster with 128 NICs, that's $192,000 savings. The training throughput difference was 3% for InfiniBand.

For 128-256 GPU clusters, I'd pick Ethernet today. Distributed system architecture is simpler to manage, and the skills are easier to find. For 512+, InfiniBand still wins on stability and collective performance.

Two Networking Patterns You Must Know

Ring All-Reduce vs Tree All-Reduce

Ring all-reduce organizes GPUs in a logical ring. Each GPU sends to its neighbor. After N steps, every GPU has the full gradient.

The bandwidth required is:

python
# For ring all-reduce
bandwidth_per_gpu = (model_size_in_bytes) / (num_gpus * compute_time)

# For a 70B model in fp16 with 512 GPUs:
# ~140GB gradients / (512 * 30ms) = ~9 GB/s per GPU
# That's ~72 Gbps — well within 200Gbps InfiniBand

Tree all-reduce uses a binary tree structure. It's faster for small clusters but doesn't scale as well. I use ring for >64 GPUs, tree for <64.

SHARP In-Network Computing

NVIDIA's SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) does gradient aggregation inside the switch. Instead of GPU→GPU→GPU communication, the switch itself computes the sum.

This cuts bandwidth requirements by ~50% for all-reduce. But it requires specific hardware (NVIDIA Quantum/InfiniBand switches with SHARP support).

In our 512-GPU cluster, SHARP reduced all-reduce latency from 85ms to 52ms on a 70B model. That's a 39% improvement in the single most expensive operation.

What Kubernetes Gets Wrong About LLM Networking

I like Kubernetes. We use it at SIVARO for inference serving. But for training at scale, Kubernetes networking abstractions break down.

The issue: Kubernetes network plugins (Calico, Cilium, etc.) add overhead. Even with DPDK and optimized CNIs, you lose 10-15% on raw bandwidth. Worse, container networking can disrupt RDMA because NIC memory regions aren't visible to the container process by default.

If you're orchestrating a distributed training cluster:

• Use Kubernetes for management, not data plane
• Deploy training pods with hostNetwork: true to bypass CNI overhead
• Use NVIDIA MIG or GPU Operator for device management, but let the MPI runtime handle network placement

Distributed systems: an introduction from Confluent explains this tension between abstraction and performance well. Sometimes you need the raw hardware access.

Practical Code: Configuring Your Network Stack

Here's the actual NCCL configuration we use for 512-GPU clusters:

bash
# /etc/nccl.conf
NCCL_IB_DISABLE=0
NCCL_NET_GDR_LEVEL=PIX
NCCL_NET_GDR_READ=1
NCCL_ALGO=Ring
NCCL_PROTO=Simple
NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1
NCCL_IB_TC=106
NCCL_IB_SL=0
NCCL_IB_TIMEOUT=22
NCCL_DEBUG=WARN

The key parameters:

• NCCL_NET_GDR_LEVEL=PIX: Enables GPU Direct RDMA at the max level
• NCCL_IB_HCA: Pin specific IB ports (avoid virtual functions)
• NCCL_IB_TIMEOUT=22: Increase timeout for large clusters (default 20 isn't enough for >256 GPUs)

And the subnet manager configuration for optimal routing:

bash
# Check routing with ibdiagnet
ibdiagnet -r -o /tmp/routing

# Verify all paths are balanced
ibdiagpath --route_check

I watch for any path with hop count >3σ from the mean. That's your bottleneck.

The Cost Reality Nobody Talks About

GPU cluster rental cost is the headline number, but network cost is the hidden multiplier.

A 1,024-H100 cluster at $5,000/hour runs for $3.6M/month. Network equipment adds 15-20%: switches, NICs, cables, transceivers. That's another $200,000-400,000 one-time.

But the real cost is utilization. At 60% utilization, your effective hourly cost per useful FLOP is 67% higher than sticker price.

I've seen teams rent 512-GPU clusters for $2,400/hour, hit 50% networking efficiency, and effectively pay $4,800/hour for real work. They'd have been better off renting a smaller cluster with better networking.

When evaluating gpu cluster networking requirements for large language models, ask providers:

• What's the bisection bandwidth?
• Is it non-blocking at full scale?
• What collective algorithms are HPL-tested?
• Get the NCCL benchmark results, not just InfiniBand marketing specs

What I'd Tell My Past Self

If I could send a message to 2024 me starting this journey:

1. Benchmark before you train. Run NCCL all-reduce benchmarks for 30 minutes before starting any training job. Plot bandwidth vs message size. If you don't see linear scaling from 1KB to 1GB, something's wrong.

2. Oversubscription kills. A 1:1 non-blocking network costs more upfront but pays for itself in 2 months of training.

3. Latency beats bandwidth. Low-latency, moderate-bandwidth switches outperform high-latency, high-bandwidth ones for all-reduce. Mellanox ConnectX-7 NICs with 1.2us latency outperform 2.4us NICs even at lower bandwidth.

4. Don't trust the defaults. Default NCCL settings are tuned for small clusters. Every parameter matters at 512+ GPUs.

Introduction to distributed systems from the academic side gets this right: distributed coordination at scale has edge cases that simple models don't capture. Network profiling is the only way.

The Network Is The Computer

Training a 175B parameter model requires moving ~350TB of data per epoch. That's not a GPU problem. It's not even a storage problem. It's a network problem.

Most teams optimize what they can measure. GPU utilization is easy to measure. Network utilization is hard. So they throw money at GPUs and wonder why throughput doesn't scale.

The best cluster I've seen wasn't the one with the most GPUs. It was a 256-GPU cluster with a perfectly tuned 400Gbps non-blocking InfiniBand fabric. It trained a 70B model faster than a competitor's 512-GPU cluster with 2:1 oversubscribed networking.

Network is not the plumbing. It's the computer.

---

FAQ: gpu cluster networking requirements for large language models

Q: Do I need InfiniBand or is Ethernet sufficient for LLM training?

A: For clusters under 256 GPUs, RoCE v2 Ethernet works fine. For 512+, InfiniBand with SHARP still offers 5-10% better throughput and more predictable performance. The gap narrows yearly — by 2027 Ultra Ethernet may match InfiniBand at scale.

Q: What's the minimum bandwidth per GPU for training a 70B model?

A: 100-200Gbps per GPU minimum. 400Gbps recommended for 512+ GPU clusters. Below 100Gbps, you'll spend more time in all-reduce than compute.

Q: How do I calculate gpu cluster rental cost for a specific training run?

A: Multiply (hourly rate) × (expected hours) ÷ (expected utilization). Utilization for well-networked clusters is 85-92%. For poorly networked ones, 50-65%. Factor that in.

Q: What's the best gpu cluster configuration for deep learning on a budget?

A: 128 H100s with 400Gbps non-blocking Ethernet beats 256 H100s with oversubscribed InfiniBand every time. Network efficiency matters more than GPU count above a threshold.

Q: Why does my training stall at scale?

A: Check NCCL timeout errors. If you see NCCL WARN NET/IB: Got completion with error, you're hitting packet drops. Increase NCCL_IB_TIMEOUT to 22, enable flow control, and check your subnet routing for long paths.

Q: How much does networking add to my total cluster cost?

A: 15-25% of total hardware cost for full non-blocking fabric. For a $5M GPU cluster, expect $750K-1.25M in networking. Cheaper clusters cut corners here — that's how you get 60% utilization.

Q: Can I mix Ethernet and InfiniBand in the same cluster?

A: Don't. Mixing fabrics adds complexity and kills RDMA performance. Pick one. If you need both, run separate pods on separate networks.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.