How to set up a GPU cluster for AI
I spent three months in 2024 building what I thought was the perfect GPU cluster. Four nodes, eight A100s each, InfiniBand between them, the works.
It was a disaster.
The networking was wrong. The scheduling software fought our workloads. We hit thermal throttling within six hours. Most people think GPU clustering is about picking the right GPU — they're wrong. It's about how you connect them, cool them, and make them play nice together.
This guide is everything I wish I'd known before I started. By the end, you'll know exactly how to set up a GPU cluster for AI that actually works in production.
Why a single GPU isn't enough anymore
Here's the math that changed everything. A single H100 can train BERT in about 3 days. Llama 3.1 405B? You'd need 2,048 H100s running for 54 days straight. That's $63 million in compute.
You can't do that on a workstation. You need a distributed system.
The fundamental problem is simple: model parameters don't fit in VRAM. Llama 3.1 405B at FP16 needs 810GB of memory. The biggest consumer GPU has 24GB. Even an H100 has 80GB. You need 11 of them just to load the weights.
But distributing across multiple GPUs introduces a new problem: communication overhead. Every time you do a backward pass across parallel GPUs, they need to synchronize gradients. If that communication is slow, your GPUs sit idle waiting.
That's why "how to set up a GPU cluster for AI" isn't a hardware question — it's a systems design question.
The three architectures that actually work
After testing configurations for 18 months, I've narrowed it down to three approaches that don't suck. Pick one.
1. The Homogeneous Cluster (for training)
This is your standard setup: identical GPUs, identical nodes, identical everything. We run 8x H100 SXM nodes at SIVARO. Every node has the same CPU, same RAM, same network card.
Why identical? Because distributed computing hinges on the straggler problem — your cluster runs at the speed of its slowest component. Mixing A100s with H100s sounds smart. It's not. The H100s finish their batch and wait. And wait. And wait.
For training, we use this layout:
# Node configuration for training cluster
GPU_COUNT=8
GPU_TYPE="H100-SXM-80GB"
CPU="AMD EPYC 9654 (96 cores)"
RAM="2TB DDR5-4800"
INTERCONNECT="8x NVLink (900GB/s per GPU pair)"
NETWORK="8x InfiniBand NDR400"
The NVLink inside the node matters more than anything else. Each GPU can talk to its neighbor at 900GB/s. Cross-node communication over InfiniBand is still only 400Gb/s (50GB/s). You want as much computation as possible to stay inside a node.
2. The Heterogeneous Cluster (for inference)
This is contrarian. Most people think you need uniformity. For inference, you don't.
We run inference across a mix of H100s, A100s, and even some RTX 6000 Ada cards. Why? Because not all requests need an H100. A 7B parameter model runs fine on an RTX 6000. A 70B model needs the A100. The 405B monster requires H100s with tensor parallelism.
Distributed system architecture for inference is fundamentally different from training. Training moves gradients around constantly. Inference moves activations forward once. The latency requirements are tighter.
For inference clusters, we separate models by size:
# Inference routing config
apiVersion: v1
kind: ConfigMap
metadata:
name: model-router
data:
routes: |
- model_pattern: ".*-7b-.*"
gpu_type: "RTX6000"
min_replicas: 3
- model_pattern: ".*-70b-.*"
gpu_type: "A100-80GB"
min_replicas: 2
tensor_parallel: 2
- model_pattern: ".*-405b-.*"
gpu_type: "H100-SXM"
min_replicas: 1
tensor_parallel: 8
3. The Spot Instance Cluster (for experimentation)
This one hurts to write. I spent six months arguing against spot instances. "They're unreliable." "You'll lose your work." "Production needs stability."
Then Anthropic published their training infrastructure paper. They use spot instances for 40% of their training capacity. If it's good enough for billion-dollar training runs, it's good enough for your fine-tuning.
The trick is checkpoint frequency and node recovery. You need to save state every 15 minutes at minimum. We use a Redis-backed checkpoint manager:
python
import asyncio
import aioredis
class SpotCheckpointManager:
def __init__(self, interval_seconds=900):
self.redis = await aioredis.from_url("redis://checkpoint-cache:6379")
self.interval = interval_seconds
async def save_checkpoint(self, model_state, optimizer_state, step):
key = f"checkpoint:step:{step}"
await self.redis.set(key, {
"model": model_state,
"optimizer": optimizer_state,
"step": step,
"node_id": socket.gethostname()
})
# Keep only last 4 checkpoints
await self.redis.ltrim("checkpoint_keys", -4, -1)
When a spot node gets reclaimed, you get a 2-minute warning. That's enough to save a checkpoint and spin up a replacement. We lose maybe 3% of training time to spot reclaimation. We save 60% on compute costs.
Networking: where most clusters die
I've seen $2 million clusters perform worse than $500k setups. The difference is always networking.
Here's the hard truth: you need InfiniBand. Ethernet won't cut it for training.
The problem is gradient synchronization. PyTorch's DistributedDataParallel (DDP) uses all-reduce to average gradients across GPUs. For a 70B model with 8 GPUs, that's about 140GB of gradients per iteration. Over 25Gb Ethernet, that's 45 seconds of just waiting. Over 400Gb InfiniBand, it's 2.8 seconds.
Most people think "we'll use gradient compression." We tested that. ZeRO-3 and FSDP help, but they don't eliminate the need for fast interconnects. The introduction to distributed systems paper Nancy Lynch wrote in 2009 got this right: communication latency is the bottleneck, not computation.
For the best GPU cluster for scientific computing, you want:
- Intra-node: NVLink 4.0 (900GB/s per GPU pair)
- Inter-node: InfiniBand NDR400 (400Gb/s per link)
- Topology: Fat tree with 4:1 oversubscription at the spine
Don't try mesh topology. It sounds good on paper. In practice, path congestion kills you. Fat tree with proper oversubscription handles burst traffic better.
Cooling: the silent killer
Nobody talks about cooling in GPU cluster guides. It's the most important thing I learned.
We did a test in August 2025. Ambient temperature hit 38°C. Our H100s throttled down to 60% utilization. We lost $12,000/hour in compute capacity.
Modern H100 SXM modules draw 700W each. A typical 4U chassis with 8 GPUs pulls 5.6kW just for the GPUs. Plus CPUs, networking, and storage, you're looking at 7-8kW per chassis. That's the heat output of 80 space heaters in a 2-foot-tall box.
Direct liquid cooling (DLC) is the answer. Not air cooling. Most data centers can handle 15-20kW per rack with air. You need 40-50kW per rack.
We switched to cold plate DLC in Q4 2025. Inlet water at 25°C, outlet at 45°C. GPU temps dropped to 55°C under load. No throttling. Ever.
Software stack: pick your battles
I've tried everything. Here's what works.
Slurm vs Kubernetes
Everyone asks me this. The answer depends on what you're doing.
For training: use Slurm. It's purpose-built for HPC workloads. Job scheduling is deterministic. Resource allocation is static. You don't want Kubernetes moving your pods around during an 8-day training run.
For inference: use Kubernetes. You need auto-scaling, rolling updates, and the ability to handle traffic spikes. Kubernetes + KubeRay gives you the best of both worlds.
We run both at SIVARO. Slurm on a dedicated training partition, Kubernetes on a separate inference partition. They share the same physical cluster through a thin orchestration layer.
Model parallelism strategy
Here's where most implementations go wrong. People over-optimize the parallelism strategy.
For models under 13B parameters: data parallelism is fine. Use DDP or FSDP with sharding degree 1.
For 13B-70B parameters: use tensor parallelism with ZeRO-3. Shard optimizers and gradients across GPUs inside a node.
For 70B+ parameters: combine tensor parallelism (intra-node) with pipeline parallelism (inter-node). We use DeepSpeed with 4-way tensor parallelism per node and 8-way pipeline parallelism across nodes.
yaml
# DeepSpeed configuration for 70B model training
deepspeed_config:
train_batch_size: 128
gradient_accumulation_steps: 4
zero_optimization:
stage: 3
reduce_bucket_size: 5e8
reduce_scatter: true
pipeline:
stages: 8
partition: "balanced"
tensor_parallel:
enabled: true
degree: 4
The distributed architecture of your software stack has to match your hardware topology. Don't let the tool dictate your design.
Common mistakes I've made (so you don't have to)
Mistake 1: Over-subscribing NVLink. We tried running 16 GPUs per node with two NVSwitch chips. Each GPU got half the bandwidth. Training throughput dropped 40%. Stick to 8 GPUs per node with full NVLink.
Mistake 2: Using DAS storage. Direct-attached storage sounds cheap. It is. It's also slow. Your GPUs will spend 30% of their time waiting for data to load. Use parallel filesystems (Lustre, GPFS, or WEKA).
Mistake 3: Ignoring job scheduling. We let researchers submit jobs ad-hoc. Three people would schedule 7-day training runs. They'd all fail because of resource conflicts. Implement fair-share scheduling with priority queues.
Mistake 4: Not testing networking under load. ib_write_bw passes? Great. Run NCCL all-reduce benchmarks under full GPU load. The numbers will shock you. PCIe congestion kills performance when all GPUs are moving data simultaneously.
Monitoring: what to watch
You can't fix what you can't see. Here's our monitoring stack:
- GPU metrics: nvidia-smi is not enough. We use DCGM-exporter with Prometheus for per-SM utilization, memory bandwidth, and temperature per GPU die.
- Network metrics: Watch NCCL collectives. If all-reduce latency exceeds 2ms for 8-GPU sync, something is wrong.
- Power metrics: H100s should run at 700W each under load. If they're pulling less, they're being underutilized.
- Thermal metrics: Junction temperature above 85°C means imminent throttling. Fix your cooling.
We use Grafana dashboards with per-node views. Each researcher can see exactly how their job is performing. When someone says "my training is slow," we can tell them it's because they're sharing a node with someone funneling gradients through a single NIC.
The best GPU cluster for scientific computing in 2026
If I were building from scratch today, here's what I'd spec:
Minimum viable cluster (for teams getting started):
- 4 nodes, each with 8x A100 80GB
- InfiniBand HDR (200Gb/s) — not NDR, the cost difference isn't worth it at this scale
- NVLink 3.0 intra-node
- 2TB SSDs per node for scratch
- Network-attached storage with Lustre
- Slurm for job scheduling
- Total cost: ~$400k
Production cluster (for serious training):
- 32 nodes, each with 8x H100 SXM 80GB
- InfiniBand NDR400 with fat tree topology
- NVLink 4.0 intra-node
- 8TB NVMe per node
- WEKA parallel filesystem (80GB/s throughput)
- Slurm + Kubernetes dual scheduler
- Direct liquid cooling
- Total cost: ~$4.2 million
Experimental cluster (for bleeding edge):
- 128 nodes, mix of H100 and B200 GPUs
- InfiniBand NDR800 (waiting for Broadcom to ship)
- NVLink 5.0 (when it drops)
- On-node HBM memory pool (360GB aggregate per node)
- Total cost: "if you have to ask, you can't afford it"
The configuration you'll actually use
Here's the Slurm partition config we run in production:
bash
# slurm-gpu-partition.conf
PartitionName=gpu-training
Nodes=gpu[01-32]
Default=YES
MaxTime=7-00:00:00
State=UP
DefaultNodeCount=4
Gres=gpu:h100:8
Oversubscribe=NO
QOS=normal,high,express
# GPU resource configuration
NodeName=gpu[01-16] Gres=gpu:h100:8
CPUs=96 Boards=1 SocketsPerBoard=1
CoresPerSocket=48 ThreadsPerCore=2
RealMemory=2000000
NodeName=gpu[17-32] Gres=gpu:h100:8
CPUs=192 Boards=1 SocketsPerBoard=2
CoresPerSocket=48 ThreadsPerCore=2
RealMemory=4000000
Feature=highmem
This gives researchers exactly what they need: guaranteed GPU access, clear resource limits, and no ability to over-commit. The best GPU cluster configuration for deep learning is the one that actually enforces boundaries.
FAQ
Q: How many GPUs do I need to start?
A: Start with 8 (one node). You can train most models up to 13B parameters on a single node. Go to 32+ when you need 70B models.
Q: Should I use NVLink or PCIe?
A: NVLink. Always NVLink for training. PCIe is fine for inference if you're running single-GPU models. For anything requiring model parallelism, NVLink saves your throughput.
Q: What about AMD GPUs?
A: We tested MI300X in early 2026. Performance is competitive with H100 for FP16 workloads. The software ecosystem (ROCm) still lags behind CUDA. If you're doing custom kernels or optimized training, stick with NVIDIA. If you're using standard PyTorch, AMD works.
Q: How do I handle multiple users?
A: Slurm with fair-share scheduling. Each user gets a priority score based on recent usage. Running a 7-day job drops your priority. Short jobs keep it high. Implement QoS levels: express (1 hour max), normal (24 hours), long (7 days).
Q: Do I need InfiniBand for small clusters?
A: For 8 GPUs (1 node), no. NVLink handles intra-node communication. For 16+ GPUs (2+ nodes), yes. We tested 8-node clusters with 100Gb Ethernet. All-reduce took 18 seconds per iteration over Ethernet vs 2.5 seconds over InfiniBand. That's 7x slower.
Q: How much storage do I need?
A: 10TB per researcher minimum. Datasets are getting bigger. The common crawl dataset is 200TB. Llama 3 training data was 15TB. Don't be the cluster where scientists can't load their own data.
Q: What about power delivery?
A: Each H100 draws 700W at peak. A 4U chassis with 8 GPUs plus CPUs draws 7-8kW. Standard racks handle 15-20kW. You need special high-density racks (40-50kW capacity) or liquid cooling.
Q: Is spot/cloud cheaper than on-prem?
A: For burst capacity, yes. For sustained training (running >6 months) on the best GPU cluster for scientific computing, on-prem is 60% cheaper. We did the math: $4.2M on-prem vs $11M cloud over 3 years for equivalent throughput.
Final thoughts
Setting up a GPU cluster isn't a one-time project. It's an ongoing process of tuning, monitoring, and occasionally ripping out components that don't work.
The cluster I designed in 2024 is nothing like what we run today. Different cooling. Different networking. Different scheduler. We're on our fourth iteration of the job queue. That's not failure. That's learning.
Start small. Buy a single node. Train something. Watch the metrics. Add a second node. Fix what breaks. Scale.
The companies that get this right — OpenAI, Anthropic, Google DeepMind — didn't build the perfect cluster on day one. They built something that worked, broke it, and rebuilt it better.
That's the real lesson of how to set up a GPU cluster for AI. Not the hardware specs. The willingness to iterate.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.