GPU Clusters for LLM Training: A Builder’s Guide
I spent three months in early 2025 trying to train a 7-billion-parameter model on a single 8x A100 node. It was a disaster. Not because the hardware was bad—it was top-tier. But because I treated the cluster like a big GPU instead of a distributed system. The networking was wrong, the parallelism strategy was wrong, and my failure rate was 40% for runs over 12 hours.
That’s when I stopped thinking about GPUs and started thinking about distributed systems.
A gpu cluster for llm training isn’t a pile of Nvidia cards. It’s a coordinated, failure-prone, bandwidth-constrained distributed system where every millisecond of latency between GPUs translates to hours of wasted training time. If you’re spending $10,000+ per node per month, you can’t afford to get the architecture wrong.
This guide is what I wish I’d read in 2024. No theory porn. Just the decisions I’ve made building clusters for production LLM workloads at SIVARO, the mistakes I’ve seen, and the configurations that actually work.
The Network Isn’t the Bottleneck—It’s the Bottleneck
Most people think GPU compute is the limiting factor for LLM training. They’re wrong. At least, partially wrong.
Modern GPUs compute fast. Really fast. An H100 does 2,000 TFLOPS of FP8. A B200 pushes past 4,500. But the second you need to sync gradients across 64 GPUs, your training speed is determined by how fast those 64 GPUs can talk to each other.
The gpu cluster networking requirements for large language models are brutal:
- NVIDIA NVLink: 900 GB/s per GPU (H100). Non-negotiable within a node.
- InfiniBand NDR400: 400 Gb/s per port. Minimum for inter-node communication.
- RoCE v2: 200-400 Gb/s. Cheaper but higher latency. Use only if your budget forces it.
I tested both InfiniBand and RoCE v2 on a 32-node cluster in December 2025. InfiniBand delivered 98% scaling efficiency up to 128 GPUs. RoCE v2 hit 82% at 64 GPUs and dropped to 71% at 128. The gradients just wouldn’t arrive fast enough.
The best gpu cluster configuration for deep learning I’ve deployed: 8x H100 per node, NVLink-Switch inside the node, 8x InfiniBand NDR400 per node, and a fat-tree topology. No oversubscription. That cluster cost $2.3M. It trained a 70B parameter model in 18 days instead of 43 on the RoCE setup.
Don’t skimp on networking. You’ll pay for it in wall-clock time.
Parallelism Strategies: What Actually Works
You need three forms of parallelism to train a large model efficiently. Not one. Not two. Three.
Data Parallelism (DP)
The simplest. Each GPU has a full copy of the model. You split the batch. This works until the model doesn’t fit on one GPU. For 7B models and below, DP is fine. Above that, you need more.
Tensor Parallelism (TP)
Split individual layers across GPUs. Each GPU holds a slice of each layer. Communication is intense—every attention computation requires an all-reduce. Use TP within a node only. Intra-node NVLink bandwidth is essential. I’ve seen people try TP across nodes. It’s slow.
Pipeline Parallelism (PP)
Split layers across stages. GPU 0 handles layers 1-4, GPU 1 handles 5-8, and so on. Communication is sequential, which reduces bandwidth pressure. But you get pipeline bubbles—idle GPUs waiting for the previous stage to finish.
Megatron-LM (Nvidia, 2024) combines all three. Distributed systems theory tells us that coordination overhead grows with partition count. Megatron minimizes that by making TP local, PP sequential, and DP global.
Here’s the profiling code I run before deciding on a parallelism config:
python
# quick parallelism profiler for LLM training
import torch.distributed as dist
def estimate_memory(model_size, num_gpus, tp_size, pp_size):
# assumes mixed precision training
param_memory = model_size * 2 # fp16 weights
optimizer_memory = model_size * 4 # adam states
gradient_memory = model_size * 2 # fp16 grads
per_gpu_params = param_memory / tp_size
per_gpu_opt = optimizer_memory / tp_size
per_gpu_grad = gradient_memory / tp_size
activation_memory = model_size * 12 / (tp_size * pp_size) # rough estimate
total = per_gpu_params + per_gpu_opt + per_gpu_grad + activation_memory
return total # in GB
print(estimate_memory(70, 64, 8, 4)) # 70B model, 64 GPUs, TP=8, PP=4
println(estimate_memory(70, 64, 4, 8)) # same model, different split
The first config (TP=8, PP=4) uses less memory per GPU but requires more intra-node bandwidth. The second (TP=4, PP=8) reduces communication pressure but increases pipeline bubbles.
I’ve settled on a heuristic: keep TP within a single node (max 8 GPUs), keep PP within a rack (max 4 nodes), and scale DP across racks. That’s the distributed system architecture sweet spot.
Storage: The Silent Killer
Everyone obsesses over GPUs and networking. Almost no one designs the storage system correctly. Then training runs fail because data loading is slower than compute.
Your GPU cluster needs three tiers of storage:
- Hot tier: NVMe SSDs on each node. For checkpoint writes and tokenizer cache. Local. Fast. Small (2-4 TB per node).
- Warm tier: Parallel filesystem (Lustre, GPFS, or Weka). Shared across all nodes. For training data and model checkpoints. Needs 100+ GB/s throughput.
- Cold tier: Object storage (S3, GCS). For raw datasets and archived checkpoints. High latency. Don’t train from this.
I’ve seen teams train directly from S3. Don’t. One company I advised in 2025 (not naming names) lost 3 days because their data loader hit S3 rate limits mid-training. The cluster sat idle. $180,000 in compute wasted.
Here’s the data loading pipeline we use at SIVARO:
python
import torch
from torch.utils.data import IterableDataset
class StreamingLLMDataset(IterableDataset):
def __init__(self, data_path, tokenizer, seq_length=4096):
self.data_path = data_path # local NVMe path
self.tokenizer = tokenizer
self.seq_length = seq_length
self.data = self._load_mmap_data()
def _load_mmap_data(self):
# memory-map the pre-tokenized data
return np.memmap(self.data_path, dtype=np.uint16, mode='r')
def __iter__(self):
# yield sequences in order, no shuffling (handled by sharding)
for i in range(0, len(self.data) - self.seq_length, self.seq_length):
yield {
'input_ids': torch.tensor(self.data[i:i+self.seq_length]),
'labels': torch.tensor(self.data[i+1:i+self.seq_length+1])
}
Key insight: pre-tokenize everything and memory-map the result. No on-the-fly tokenization during training. That shaves 30-40% off data loading time.
The Failure Problem
Every distributed system fails. Distributed computing isn’t about preventing failures—it’s about recovering from them faster than your training kernel decays.
LLM training runs for days or weeks. In that time, you will experience:
- GPU hangs (silent or with NCCL timeouts)
- PCIe errors
- InfiniBand link flips
- Power supply failures
- Cooling failures
I’ve logged over 200 training runs in production. Our mean time between failures is 4.7 days. That’s with redundant power, redundant networking, and daily health checks.
The fix isn’t more hardware. It’s checkpointing. But not the naive kind.
bash
# our checkpointing script - saves every 1000 steps with async upload
python train.py --model-size 70B --num-gpus 256 --checkpoint-every 1000 --async-checkpoint-to-s3 --tensorboard-logdir /shared/logs
We save local checkpoints to NVMe on node 0, then asynchronously copy to S3. That way the training loop pauses for exactly the local write (typically 3-5 seconds for a 70B model) and the S3 transfer happens in the background.
Most teams sync to shared storage synchronously. Their training loop pauses for 30-60 seconds every checkpoint. Over 18 days of training with 500 checkpoints, that’s 5-8 hours of pure idle time. Useless.
Cost Optimization: The Real Math
Let me be direct. Running a gpu cluster for llm training is expensive. An 8x H100 node costs ~$40-$50/hour on cloud providers. On-prem, you’re looking at $250,000 per node fully loaded (hardware, networking, power, cooling, rack).
The cloud vs. on-prem decision depends on one number: utilization.
If your cluster is going to run 80%+ of the time for 12+ months, buy on-prem. I run the numbers for every client. The breakeven point is usually around 7-8 months of 90% utilization. Below that, rent.
But utilization isn’t just runtime. It’s effective utilization. If your GPUs are running at 30% of peak TFLOPS because your parallelism is wrong, you’re burning money.
Here’s the formula I use:
effective_utilization = (model_flops * batch_size * seq_length * num_steps) /
(gpu_peak_flops * num_gpus * wall_clock_time)
For our best config: 68% utilization. For the worst: 12%. The spread is enormous.
One more thing: spot/preemptible instances are tempting. I tested them on AWS P5 instances in March 2026. The 60% cost savings looked great. But we averaged 2.4 preemptions per day. Every preemption lost 1-2 hours of work (even with frequent checkpointing). Net savings: 22%. Not worth the engineering hassle.
The Human Problem
You can buy the perfect cluster. You can configure InfiniBand with sub-microsecond latency. You can get 95% scaling efficiency.
None of that matters if your team doesn’t understand distributed debugging.
I’ve spent weeks teaching engineers how to read NCCL logs. How to interpret nvidia-smi output during all-reduce. How to use torch.distributed.barrier() to detect hanging ranks.
Introduction to distributed systems from 2009 is still relevant. The fundamental problems—partial failure, timing, consistency—haven’t changed. The scale has.
Train your team. Not just on PyTorch. On distributed systems fundamentals. Failure detection. Consensus protocols. Distributed systems theory at Confluent’s site has great practical resources.
Monitoring: What to Watch
Don’t monitor what’s easy. Monitor what’s predictive.
- NCCL all-reduce latency: If it spikes above 50 microseconds for intra-node or 10 microseconds for inter-node, something’s wrong.
- GPU memory bandwidth utilization: Should be >80% during training. Below 60%? Your model’s compute-bound.
- Data loading time per step: Should be under 10% of step time. If it’s 20%+, pre-tokenize more aggressively.
- Checkpoint write time: Watch the p99, not the p50. A single slow disk can stall the entire training run.
We use a custom dashboard built on Prometheus and Grafana. Nothing fancy. But it alerts our team when any of these metrics drift.
FAQ
Q: How many GPUs do I need for a 70B parameter model?
A: Minimum 32 GPUs (4x 8-GPU nodes) with 80GB memory per GPU. You’ll need tensor parallelism (TP=4 or 8) and pipeline parallelism (PP=2 or 4). 64 GPUs gives you more headroom.
Q: Is NVLink mandatory?
A: For tensor parallelism, yes. TP requires high-bandwidth intra-node communication. Without NVLink, TP becomes the bottleneck. Use NVLink-Switch (NVSwitch) for 8 GPUs per node.
Q: Can I use Ethernet instead of InfiniBand?
A: For small clusters (4-8 GPUs) and models under 7B, sure. For anything larger, no. Ethernet adds 5-10x latency and drops packets under load. InfiniBand has RDMA that bypasses the kernel.
Q: How important is CPU-to-GPU bandwidth?
A: Less important than GPU-to-GPU. Data loading is the bottleneck here. Use CPU affinity and NUMA-aware memory allocation. We pin data loading threads to specific CPU cores.
Q: Should I use mixed precision training?
A: Absolutely. FP16/FP8 training is the standard for LLMs. It halves memory usage and doubles throughput. Just watch for loss spikes—use gradient scaling.
Q: What about checkpoint formats?
A: torch.save() with pickle is fine for small models. For 70B+, we use safetensors for speed and safety. No arbitrary code execution risk.
Q: How do I handle multi-node job scheduling?
A: SLURM is the industry standard. Kubernetes with Volcano scheduler works too but has higher overhead. We use SLURM with sbatch wrappers and custom MPI launchers.
Q: What’s the biggest mistake you see?
A: Starting with too many GPUs. Teams buy 256 GPUs and wonder why training is slower than on 64. Parallelism overhead grows with scale. Start small, profile, then scale.
The Bottom Line
A gpu cluster for llm training is a distributed system wearing a GPU disguise. The hardware matters, sure. But the networking, parallelism strategy, storage system, monitoring, and failure recovery matter just as much—if not more.
I’ve built clusters that worked and clusters that burned money. The difference wasn’t the GPUs. It was whether the team understood distributed systems engineering.
Start with the network. Validate your parallelism strategy with small-scale runs. Pre-tokenize your data. Checkpoint obsessively. Monitor the right metrics. Train your engineers.
Do that, and $10M in GPUs will actually produce models. Skip it, and you’ll have a very expensive space heater.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.