Here's the short version: Disaggregated inference splits the inference pipeline into separate, independently scalable services — typically separating the prefill phase (processing the input) from the decode phase (generating tokens), and often isolating the KV cache management entirely.
Most people think this is just fancy cloud architecture. They're wrong. It's a fundamental rethinking of how GPUs spend their time.
Let me show you what I mean.
The Problem With Monolithic Inference
You've probably served a model like Llama 2 or Mistral on a single GPU. Works fine for small workloads. But scale it up.
When you send a prompt to a monolithic server, here's what happens:
- Prefill phase: GPU processes your input tokens in parallel. High compute, high memory bandwidth. GPU utilization hits 80-95%.
- Decode phase: GPU generates tokens one at a time. Low arithmetic intensity. GPU utilization drops to 5-15%.
- KV cache grows: Every generated token increases memory pressure. Eventually, you OOM.
The GPU is a sprinter during prefill and a slow jogger during decode. But the monolithic server treats both the same — same GPU, same memory allocation, same scheduling.
Result: You either overprovision GPUs for peak prefill demand (wasting money during decode), or you underprovision and your decode latency spikes during prefill bursts.
That's not a resource problem. That's an architecture problem.
What Is a Disaggregated Inference? (The Architecture)
Disaggregated inference breaks the pipeline into separate services that can run on different hardware.
At minimum, you get three components:
- Prefill service: Handles input processing. Needs high compute (GPU with large matrix units). Can batch many prompts together.
- Decode service: Handles token generation. Needs low latency and efficient KV cache management. Benefits from high memory bandwidth but doesn't need peak compute.
- KV cache service: Stores and manages the key-value cache across both phases. This is often the bottleneck you don't see coming.
Some architectures go further and separate the tokenizer, the embedding layer, and the attention computation. But the core insight is this: you're no longer forced to keep everything on one monolithic GPU.
How It Works
Request Flow
┌─────────────────┐
│ Load Balancer │
└────────┬──────────┘
│
┌────────▼──────────┐
│ Prefill Service │ ← GPU cluster optimized for batch compute
└────────┬──────────┘
│ (KV cache transferred)
┌────────▼──────────┐
│ Decode Service │ ← GPU cluster optimized for low latency
└────────┬──────────┘
│
┌────────▼──────────┐
│ KV Cache Store │ ← Memory-optimized nodes or remote memory
└──────────────────────┘
In practice, the prefill service runs on A100s or H100s with high TFLOPs. The decode service runs on L4s or even CPUs for small models. The KV cache lives in high-bandwidth memory, sometimes on separate nodes with fast interconnects.
Why This Matters Now
The timing isn't accidental.
Memory limits on single GPUs: The largest consumer GPU today has 80GB HBM3. A Llama 2 70B model in FP16 needs 140GB just for weights. You can't fit it on one GPU even if you wanted to. Disaggregation becomes inevitable.
KV cache explosion: For long-context models (128K tokens or more), the KV cache per request can hit several GB. Monolithic servers either evict history or OOM. Disaggregated systems can allocate cache storage independently of compute.
Cost pressure: GPU rental costs haven't dropped. Spot pricing fluctuates wildly. Disaggregation lets you mix on-demand and spot instances — prefill can tolerate preemption, decode can't.
Multi-model serving: If you're serving different models, sharing a pool of decoders across models reduces idle time compared to dedicated monolithic servers per model.
The Hard Parts Nobody Talks About
Transfer Latency Is Real
Moving KV cache from prefill to decode takes time. On a single machine with NVLink, it's microseconds. Across machines over InfiniBand or Ethernet, it's milliseconds. For real-time applications, those milliseconds matter.
Fix: co-locate prefill and decode on the same NUMA node, or use remote direct memory access (RDMA).
Scheduling Is Non-Trivial
In a monolithic system, scheduling is trivial — one request, one GPU, done. In disaggregated systems, you need to coordinate:
- Which prefill instance handles which request
- Where the KV cache goes after prefill
- Which decode instance gets it next
- How to handle stragglers (fast prefill + slow decode = pipeline stalls)
Kubernetes-based approaches struggle here. Kubernetes doesn't understand inference phases. It schedules pods, not work-stealing across compute phases.
Memory Fragmentation
The KV cache is variable-sized. Different sequences have different lengths. Over time, memory becomes fragmented. In monolithic systems, you just OOM. In disaggregated systems, you need a defragmentation strategy.
One approach is a slab allocator, adapted for GPU memory. Each slab handles fixed-size KV cache blocks. Blocks get freed and reused.
Debugging Is a Nightmare
When a monolithic server fails, you look at one log. When a disaggregated system fails, you have three services, two networks, and a distributed cache. Finding the bottleneck requires distributed tracing.
OpenTelemetry with custom spans for each inference phase helps. But it's still harder than monolithic debugging. The trade-off is worth it for scale, but don't pretend it's clean.
When You Should (and Shouldn't) Disaggregate
Do It If
- You serve multiple models from the same cluster
- Your workload has bursty traffic patterns
- You need to support 128K+ token contexts
- Your GPU utilization is below 50% on average
- You're paying for A100s but using them mostly for decode
Don't Do It If
- You serve one model with predictable, consistent throughput
- Your latency requirements are under 50ms end-to-end (disaggregation adds overhead)
- You're running on single-node clusters with no network bandwidth between GPUs
- Your model fits comfortably on one GPU with 30% headroom
Don't disaggregate a single Llama 3 8B on one GPU. A monolithic server handles that fine. Disaggregation adds complexity without benefit.
Code Examples
Simple Request Routing
# Pseudo-code for a disaggregated inference scheduler
class DisaggregatedScheduler:
def __init__(self, prefill_nodes, decode_nodes, cache_store):
self.prefill = prefill_nodes
self.decode = decode_nodes
self.cache = cache_store
def serve_request(self, prompt, max_tokens=1024):
# Step 1: Route to prefill
prefill_node = self.select_prefill_node(len(prompt))
# Step 2: Execute prefill, get KV cache reference
cache_key = prefill_node.prefill(prompt)
# Step 3: Store cache in shared memory
self.cache.store(cache_key, prefill_node.get_cache())
# Step 4: Route to decode node that has this cache
decode_node = self.select_decode_near_cache(cache_key)
# Step 5: Generate tokens
tokens = decode_node.decode(cache_key, max_tokens)
return tokens
def select_prefill_node(self, prompt_length):
# Pick node with lowest queue depth for compute-heavy prefill
return min(self.prefill, key=lambda n: n.prefill_queue_depth())
def select_decode_near_cache(self, cache_key):
# Pick node closest to cached KV data
cache_location = self.cache.locate(cache_key)
return min(self.decode, key=lambda n: n.distance_to(cache_location))
KV Cache Transfer With RDMA
# Simplified RDMA transfer between prefill and decode nodes
import pyverbs # Python bindings for RDMA
def transfer_kv_cache(source_node, dest_node, cache_handle):
# Allocate memory on destination
dest_mr = dest_node.register_memory(size=256*1024*1024) # 256MB cache
# Create RDMA write request
wr = pyverbs.WorkRequest(
opcode=pyverbs.Opcode.RDMA_WRITE,
source_addr=source_node.cache_addr(cache_handle),
dest_addr=dest_mr.addr,
length=dest_mr.size,
send_flags=pyverbs.SendFlags.SIGNALED
)
# Post to completion queue
source_node.qp.post_send(wr)
# Wait for completion
comp = source_node.cq.poll(timeout_ms=500)
if comp.status == pyverbs.Status.SUCCESS:
return dest_mr.handle
else:
raise TransferError(f"RDMA failed with status {comp.status}")
Dynamic Decode Scaling
# Auto-scaler for decode nodes based on queue depth
import asyncio
class DecodeAutoScaler:
def __init__(self, min_nodes=2, max_nodes=12):
self.nodes = self.initialize_nodes(min_nodes)
self.target_queue_depth = 100 # requests per node
async def scale_loop(self):
while True:
total_requests = sum(n.running_requests() for n in self.nodes)
nodes_needed = max(self.min_nodes,
min(self.max_nodes,
total_requests / self.target_queue_depth))
if nodes_needed > len(self.nodes):
self.add_nodes(nodes_needed - len(self.nodes))
elif nodes_needed < len(self.nodes) - 1:
self.remove_nodes(len(self.nodes) - nodes_needed)
await asyncio.sleep(5) # Don't thrash
The Future: Memory Disaggregation
The next frontier? Disaggregating memory itself.
Today, GPU memory is physically attached to GPU compute. You can't access HBM on one GPU from another easily. But new architectures like CXL and NVIDIA's upcoming Grace Hopper are changing that.
One emerging setup keeps the KV cache in a shared memory pool, accessed via CXL-attached memory nodes. Prefill nodes write to it, decode nodes read from it. No explicit data transfer needed — the memory controller handles it.
This will be standard in 2-3 years.
FAQ: What Is a Disaggregated Inference?
Does disaggregated inference require special hardware?
No. You can do it with standard GPUs and network. But you'll benefit from fast interconnects (NVLink, InfiniBand, or at least 100GbE). For large models, you want RDMA-capable networking.
How does batch size change with disaggregation?
Monolithic servers typically use small batch sizes (1-4) because the decode phase is sequential. Disaggregated prefill services can use large batches (64-512) since they only handle the parallel prefill phase. This dramatically improves GPU utilization during prefill.
What's the latency difference?
Disaggregated systems add overhead for cache transfer and scheduling. For batch inference, this is negligible. For real-time applications, you'll notice it. Trade-off: for long contexts, disaggregation can actually reduce latency because the decode phase isn't starved by prefill bursts.
Can I do this with open-source inference servers?
vLLM has experimental support for disaggregated prefill and decode as of their 0.5 release. TGI doesn't support it natively yet. For smaller deployments, vLLM's implementation works.
Does this work for vision models?
Yes, but differently. Vision models have a different compute profile — prefill is heavier, but decode doesn't have the same KV cache explosion. The prefill benefits from disaggregation more than the decode does for these models.
What about MoE models?
Mixture-of-Experts (like Mixtral 8x7B) actually benefit more from disaggregation. You can prefill on a cluster with all experts loaded, then decode on a cluster with only the frequently-used experts.
Is this worth it for hobby projects?
No. Stick with monolithic. Disaggregation adds operational complexity that doesn't pay off until you're serving 100+ reqs/sec or have multi-model workloads.
My Advice
-
Start monolithic, then disaggregate one piece at a time. Don't rewrite everything at once. Separating the KV cache from the compute is a natural first step; the prefill/decode split can come later.
-
Monitor GPU utilization by phase. If you don't know how your GPUs spend time, you're guessing. Memory stalls during decode are a solvable problem once you measure them.
-
Network is your new bottleneck. In monolithic systems, the GPU-to-GPU network barely matters. In disaggregated systems, it's everything.
-
Don't overdisaggregate. An architecture with 7 separate services for one model is not architecture, it's art. Keep it to 3-4 components max.
-
Plan for failure. When your decode node dies mid-request, the KV cache is gone. Checkpointing cache state lets a request recover instead of failing completely.
The Bottom Line
What is a disaggregated inference? It's the answer to the question: "Why are my GPUs idle 60% of the time?"
In 2024, monolithic inference is a luxury you can't afford at scale. The economics are clear. The technology exists. The only question left is: how long will you keep paying for idle GPU cycles?
If you're running production inference at scale, you'll hit this wall eventually. Either you choose to disaggregate on your terms, or the bottleneck will force your hand.
Start measuring. Start small. Then break apart the monolith.