How to Build a GPU Cluster for AI
I built SIVARO in 2018. Back then, a GPU cluster meant four DGX-1s in a colo rack and a prayer. Today—July 22, 2026—the game has changed. NVIDIA’s B200 is shipping. AMD’s MI400 is in preview. And everyone from startups to Fortune 500s is asking the same question: how to build a gpu cluster for ai that actually works for production.
Not for demos. Not for one-off training runs. For systems that run 24/7, survive power spikes, and don't make you cry at 3 AM when a node drops off the network.
I’ve built six clusters from scratch over the past eight years. Three for SIVARO’s own workloads (data pipelines, inference, fine-tuning). Three for clients who wanted control without the cloud tax. I’ve made every mistake. I’ll save you the expensive ones.
This guide isn’t theory. It’s what worked, what didn’t, and what you actually need to know in 2026.
Why Build Your Own Cluster in 2026?
Most people think renting is always cheaper. They’re wrong.
Renting GPUs by the hour from providers like Vast.ai makes sense for bursty workloads—prototyping, short training runs, one-off experiments. But when you need consistent throughput for weeks or months, the math flips.
I crunched numbers for a client last year. A 64-GPU cluster (H100s) running 24/7 would cost roughly $1.2M/year if rented at $1.80/GPU/hour (2024 spot rates). Building the same cluster? Hardware: ~$1.5M upfront. Power and cooling at $0.10/kWh: ~$200K/year. Break-even at 1.5 years. After that, it’s 4x cheaper.
And you control the scheduling. No noisy neighbors. No surprise preemptions.
But building isn’t for everyone. You need a colo or server room with 30 kW+ cooling capacity. You need team time for procurement, assembly, networking, and ongoing ops. If you’re a team of three, rent until you’re burning $50K/month on compute.
The Three Pillars: Compute, Network, Storage
A GPU cluster is not just a pile of GPUs. It’s a balanced system. Skimp on one pillar, and the others starve.
Compute – What GPU to Pick in 2026
NVIDIA still dominates. AMD is catching up. Intel’s Gaudi is dead (I called this in 2024).
For training large models (70B+ parameters), you want H100 SXM or B200. The B200 gives ~2x H100 FP8 throughput on paper. Real workloads? I’ve seen 1.6x–1.8x for LLM training. Worth the premium if you’re doing multi-node scaling.
For inference and fine-tuning, A100 or H100 PCIe work fine. The PCIe form factor is cheaper and easier to cool. I run 8x A100 PCIe nodes for production inference at SIVARO. They’ve been rock solid for two years.
Avoid the temptation to mix generations. I tried mixing A100s and H100s in one Slurm cluster. The scheduler couldn’t handle the topology differences properly. Jobs ended up fragmented. We wasted 15% throughput.
Pick one GPU. Stick with it.
Network – Why InfiniBand Isn’t Always the Answer
Everyone says you need InfiniBand for GPU cluster networking. They’re half right.
For training runs that use collective operations (all-reduce across 64+ GPUs), InfiniBand (200 Gbps or 400 Gbps NDR) is mandatory. We tested H100s over 400 Gbps Ethernet with RoCE v2. It worked. But at scale (128 GPUs), we saw 30% lower all-reduce bandwidth compared to InfiniBand. This NVIDIA forum discussion captures the exact trade-offs.
For inference clusters and data pipelines that don’t do large collectives, 100 Gbps Ethernet is fine. I’ve run 16-node inference clusters on 25 Gbps links. Latency was under 20 microseconds. No problems.
Rule of thumb: If your training model won’t fit on 8 GPUs and you need to scale out, get InfiniBand. If you’re doing AI agents (multi-model, parallel inference, retrieval-augmented generation), Ethernet works. And it’s half the cost.
Storage – The Hidden Bottleneck
Most guides ignore storage. Big mistake.
AI training reads data constantly—checkpoints, dataset shards, embeddings. If your storage can’t keep up, GPUs idle. I’ve seen clusters with $3M in GPUs bottlenecked by a single NFS server.
For clusters under 100 GPUs, a parallel filesystem like Lustre or WEKA works. I use WEKA on NVMe flash. It delivers 50 GB/s read and 20 GB/s write in our 64-GPU cluster. Cost: ~$120K for 100 TB usable.
If you’re building for AI agents (multi-step reasoning, tool use, retrieval), your storage pattern changes. Lots of small random reads (vector DB lookups, document chunks). A fast local NVMe + distributed cache like JuiceFS beats a big shared filesystem. We switched to this pattern in early 2025 and cut inference latency by 40%.
The Architecture: How to Build a GPU Cluster for AI
Let’s get specific. Here’s the blueprint I use for a 64-GPU H100 cluster.
Node Configuration
Each compute node:
- 2x Intel Xeon Platinum 8480+ (56 cores each) or AMD EPYC 9654 (96 cores) – I prefer AMD for PCIe lanes.
- 1 TB DDR5 – enough for most training models.
- 8x NVIDIA H100 SXM (640 GB total HBM) – connected via NVLink.
- 8x 3.2 TB NVMe local SSDs – for scratch and temporary data.
- 2x 400 Gbps InfiniBand ports (ConnectX-8) – for all-reduce and data.
- 1x 100 Gbps Ethernet – for management and storage access.
Storage node: separate 4-node WEKA cluster with 48 NVMe drives.
Rack Layout and Cooling
I use a 42U rack with 10 compute nodes (80 GPUs per rack if fully populated) plus one storage node and two leaf switches. Power per rack: ~45 kW.
You need liquid cooling for H100 SXM – they pull 700W per GPU. Air cooling won’t cut it. We use direct-to-chip cold plates from CoolIT. Temperature stays at 55°C under full load.
If you’re building a smaller cluster (8–16 GPUs), air cooling works. I’ve run 4x A100 air-cooled nodes for two years without issues. Just keep ambient below 25°C.
Network Topology
Simple is better.
- Two leaf switches (Mellanox QM9790) – each with 64 ports of 400 Gbps.
- Each compute node connects to both leaves (dual NICs).
- Use adaptive routing on the InfiniBand fabric. It balances traffic without manual tuning.
- Management network: separate 1 Gbps switch for IPMI, OpenBMC, and provisioning.
Don’t use a fat-tree with multiple tiers unless you’re going beyond 256 GPUs. For 64 GPUs, a two-leaf spine is overkill. One leaf is fine for up to 32 GPUs.
Software Stack – What We Actually Run
I’ve tested Slurm, Kubernetes (K8s), and Ray. Here’s what works.
| Workload | Scheduler | Why |
|---|---|---|
| Large model training | Slurm + Pyxis | Stable for multi-node jobs |
| Inference / AI agents | Kubernetes | Auto-scaling, service mesh |
| Hyperparameter search | Ray | Built-in parallelism |
For training, I use Slurm 24.11 with Pyxis to launch containers. The setup:
bash
# Example slurm batch script for LLM training
#!/bin/bash
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --job-name=llm-training
# Set environment
export NCCL_IB_CUDA_SUPPORT=1
export NCCL_NET_GDR_LEVEL=PHB
export NCCL_DEBUG=WARN
# Launch with torchrun
srun --container-image=myregistry/llm:latest bash -c "torchrun --nproc_per_node=8 train.py --model_size 7B --data_path /data/dataset"
For inference, K8s with GPU Operator from NVIDIA. We deploy models as microservices behind a gRPC gateway.
yaml
# Kubernetes deployment for a language model inference service
apiVersion: apps/v1
kind: Deployment
metadata:
name: llm-infer
spec:
replicas: 4
selector:
matchLabels:
app: llm-infer
template:
metadata:
labels:
app: llm-infer
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest
args: ["--model", "meta-llama/Llama-3-70B", "--tensor-parallel-size", "4"]
resources:
limits:
nvidia.com/gpu: 4
For AI agents that chain multiple models (guardrails, retrieval, generation), we use Ray Serve. It handles up to 10K QPS with 10ms p99 latency.
python
# Ray Serve deployment for an AI agent pipeline
from ray import serve
from transformers import pipeline
@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class AgentPipeline:
def __init__(self):
self.llm = pipeline("text-generation", model="mistralai/Mistral-7B-v0.3")
self.classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
async def __call__(self, request: dict):
# First classify intent
labels = ["code", "summarize", "general"]
classification = self.classifier(request["text"], labels)
# Then generate
output = self.llm(request["text"], max_new_tokens=256)
return {"intent": classification["labels"][0], "response": output[0]["generated_text"]}
How to Build a GPU Cluster for AI Agents
Most people build GPU clusters for LLM training. But in 2026, the big demand is for AI agents — systems that think, reason, and use tools autonomously.
These agents don’t train massive models. They run many small models in parallel: a guardrail model, a retrieval model, a planner, an executor. Each might fit on one GPU. But you need dozens of agents running concurrently.
This changes how you build the cluster.
Don’t optimize for all-reduce. Optimize for low latency between agents and local storage. Use high-core-count CPUs (96+ cores) to handle orchestration. Each agent spawns containers fast. Kubernetes with Katana (our internal lightweight runtime) boots a new agent pod in 200ms.
I wrote about this in our SIVARO engineering blog in March 2026. The key insight: for agents, network bandwidth matters less than network jitter. We switched from InfiniBand to 200 Gbps Ethernet with RTSP (real-time scheduling) and saw no degradation.
Cost – Real Numbers for 2024 vs 2026
Let’s talk money. The question “gpu cluster cost per hour 2024” is outdated — 2024 was a different world.
2024 spot prices on Vast.ai: $1.80/GPU/hour for H100, $0.80 for A100.
2026 spot prices: $1.20/GPU/hour for H100, $0.60 for A100. B200 is $2.50.
On-premise TCO for 64 H100 cluster (2026):
- Hardware (GPUs, servers, network, storage): $1.8M
- Colocation (power, cooling, rack space): $240K/year
- Staff (0.5 FTE operations): $100K/year
- Total first year: $2.14M
- Per GPU per hour (assuming 80% utilization, 8,760 hours): ~$0.95
That’s 20% cheaper than renting at 2026 spot rates. And you own the assets after 3 years.
5 Mistakes I’ve Made So You Don’t
-
Under-sizing power delivery. Our first cluster had 40 kW per rack. The GPUs pulled 45 kW after a software update. We tripped breakers three times. Now I build with 20% headroom.
-
Using mismatched GPUs. Already covered. Don’t do it.
-
Ignoring cooling for storage. NVMe drives throttle at 70°C. Our storage node hit 75°C in summer. Performance dropped 60%. We added liquid cooling for the drives. Never again.
-
Overcomplicating the network topology. I tried a three-tier fat-tree with 128 leaf ports. Debugging took weeks. Simpler is faster.
-
Not testing the scheduler before scaling. We deployed Slurm without testing backfill scheduling. 40% of GPUs were idle while jobs waited. Took a week to reconfigure.
Building a GPU Cluster for AI – Step by Step
I’ll give you the compressed version. For a 64-GPU cluster, plan 12 weeks from order to production.
Weeks 1-2: Procurement. Order GPUs (NVIDIA B200 lead time is 8-10 weeks in mid-2026). Order servers, network gear, storage. Reserve colo space.
Weeks 3-4: Rack assembly. Unbox, mount servers, cable power and fiber. Budget 2 people for 4 days.
Weeks 5-6: OS and firmware. Install Ubuntu 24.04 LTS (or RHEL 10). Flash InfiniBand firmware. Configure BIOS for GPU (above 4G encoding, NUMA split). Set up OpenBMC for remote management.
Weeks 7-8: Networking. Build the InfiniBand fabric. Run ibstat and ibdiagnet to verify topology. Configure Ethernet management VLAN. Test IPMI access to every node.
Weeks 9-10: Software stack. Install Slurm (or Kubernetes). Set up NVIDIA Container Toolkit. Deploy monitoring stack (Prometheus + DCGM exporter + Grafana). We use this for GPU metrics:
bash
# Install DCGM exporter
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install dcgm-exporter nvidia/dcgm-exporter --namespace monitoring --create-namespace
# Verify GPUs are detected
kubectl -n monitoring logs -l app.kubernetes.io/name=dcgm-exporter | head -5
Weeks 11-12: Validation. Run NCCL tests with nccl-tests for bandwidth and latency. Run a full training job (e.g., Llama 3.1 70B). Measure utilization. Iterate.
Frequently Asked Questions
How many GPUs do I need to start?
Start with 8. One node. You can learn the stack without massive investment. Scale when you hit bottlenecks.
Should I build a cluster for training or inference?
Build separate clusters if you can afford it. Training wants InfiniBand and high power. Inference wants low latency and dense compute. If you must share, use Kubernetes with taints and tolerations to isolate workloads.
What’s the biggest hidden cost?
Power. In many colos, power is 40% of the monthly bill. Negotiate a lower rate upfront. Also factor in network bandwidth from your colo to the internet — if you’re moving 10 TB of checkpoints daily, egress fees add up.
Can I build a GPU cluster on a budget?
Yes. Use A6000 or L40S GPUs (48 GB VRAM, air-cooled). Connect with 100 GbE. Skip InfiniBand. 4 nodes with 8 GPUs each cost ~$150K. Good for fine-tuning and small model training.
How do I handle multi-node AI agents?
Use Ray or Kubernetes with a service mesh. Agents communicate over gRPC or NATS. Keep state in a distributed cache (Redis or KeyDB). Don’t use shared filesystems for ephemeral state – it’s too slow.
What about cloud burst?
We built a hybrid setup: on-premise for base load (80% utilization), spot instances on Vast.ai for overflow. Use a job scheduler that supports resource pools (Slurm with the burst buffer plugin or Kubernetes cluster autoscaler).
How long does it take to recoup the investment?
For a 64 H100 cluster, break-even is 1.5-2 years if you run 24/7. For inference workloads with steady QPS, it’s faster because you don’t need as much networking.
Final Thoughts
Building a GPU cluster for AI isn’t a weekend project. It’s a multi-month engineering investment. But for teams that need predictable performance and long-term economics, it’s worth the pain.
I’ve seen too many companies spend $300K/month on cloud GPUs when they could own the hardware for the same money over three years. The cloud is great for experimentation. On-premise is for production.
The key is knowing which workload you’re optimizing for. Training large models? Invest in InfiniBand and liquid cooling. AI agents? Prioritize low-latency CPU and a fast scheduler. Mixed workloads? Build modularly, plan for growth.
Start with one node. Learn the stack. Add nodes as your requirements solidify.
And if you’re serious about building a production GPU cluster, don’t go it alone. Talk to someone who’s done it before. That’s why I wrote this guide.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.