Accelerating Distributed MoE Expert Placement
You’re sitting on a cluster of 256 H100s. Your Mixture-of-Experts model has 64 experts per layer. Every forward pass, the router picks the top-2 experts per token, and then you have to move those activations across the network. The moment you scale beyond a single node, everything slows down. Not because the compute is saturated — because the placement algorithm is dumb.
I’ve been there. At SIVARO, we build production AI systems that handle 200K events per second. We run MoE models for real-time recommendation and fraud detection. In 2024, our routing-to-placement latency was eating 40% of our total inference time. That’s insane. So we spent the next 18 months figuring out how to accelerate distributed MoE expert placement — not just with better hardware, but with smarter algorithms.
This guide is what I wish I’d read in early 2025. It’s not a survey paper. It’s the scars.
Why Expert Placement Is the Real Bottleneck
Most people think MoE scaling is about training stability. Wrong. Training MoE models is painful but well-studied — load balancing losses, expert capacity, all that. The untold story is inference placement. Once you deploy a distributed MoE model, the experts live on different GPUs. A token arrives at the router, gets routed to expert E1 and E5, but E1 is on GPU 3 and E5 is on GPU 7. You need to move the token’s hidden states to both, then gather the outputs.
That’s a collective operation. In a cluster with NVLink within nodes and RDMA between nodes, the placement decision (which expert lands where) determines whether you pay a 2µs overhead or a 200µs overhead.
At first I thought this was a networking problem — faster interconnects, smarter topologies. Then I realized it’s a scheduling problem disguised as a networking problem. You can’t outrun bad placement with faster cables.
Let’s define the problem formally: Given a set of experts with dynamic load, a set of GPUs with known inter-GPU bandwidths (from a topology graph), and a set of tokens assigned to experts each step, choose a mapping from experts to GPUs that minimizes total communication latency under capacity constraints. Oh, and you have to solve it in a few milliseconds.
GPU Cluster Explained: Architecture, Nodes and Use Cases covers the basics of cluster topology. But it doesn't tell you what to do when one GPU becomes a hot spot because all the popular experts landed there.
The Distributed System Architecture Problem
You can’t solve expert placement without understanding what is the distributed system architecture? of your serving stack. Most teams treat MoE serving as a stateless routing problem: tokens → router → experts. But the real architecture is a distributed key-value store where the keys are expert IDs and the values are the model parameters (or, during inference, the token hidden states). Except the keys move. And the values are gigabytes.
When we designed SIVARO’s inference engine, we started with a static placement: compile-time assignment of experts to GPUs based on expected load. That worked for a week. Then traffic patterns shifted — a new product launch caused one expert to get hit twice as hard. The static map became a disaster. Latency spiked from 12ms to 45ms.
We needed dynamic placement. That’s where the distributed systems challenge gets spicy. Now you’re solving an online bin packing problem in a continuously changing load environment, and you can’t afford to move experts around too often because moving a 5GB expert over the network takes time.
Three Levers to Accelerate Placement
After benchmarking dozens of approaches, we narrowed it down to three levers that actually move the needle:
1. Topology-Aware Initial Placement
If you’re starting training or deploying a new model, don’t randomly assign experts to GPUs. Use the cluster topology — specifically, the NVLink domains and leaf/spine bandwidths — to place frequently co-routed experts on the same node or same NVSwitch domain. This is obvious in hindsight, but most frameworks still use round‑robin.
2. Load-Balanced Dynamic Migration
Monitor expert load in real time using a lightweight profiling daemon. When the imbalance ratio (load on hottest GPU / average load) exceeds a threshold (we use 2.0 on H100s), trigger a migration. But don’t migrate entire experts — use partial replication: keep a shadow copy on a cold GPU, redirect new tokens there, then phase out the original. This is borrowed from distributed databases (think DynamoDB splitting).
3. Communication-Caching for Repeated Routings
This one is our secret sauce. In many production workloads, the same input patterns (e.g., certain user sessions) route to the same experts repeatedly. Cache the routing decision — the expert IDs — and pre‑fetch the hidden states into the GPU’s HBM before the token arrives. We call this predictive placement. It cut our latency by 30% for rec sys traffic.
Here’s a concrete code snippet showing the difference between a naive placement and our topology‑aware version:
python
# Naive placement: round-robin
def place_experts_naive(num_experts, num_gpus):
return [i % num_gpus for i in range(num_experts)]
# Topology-aware: use GPU affinity groups based on NVSwitch domains
def place_experts_topology(num_experts, gpu_topology, expert_co_route_affinity):
# gpu_topology: dict mapping domain_id -> list of GPU indices
# expert_co_route_affinity: sparse matrix of which experts often get same tokens
placement = [-1] * num_experts
domain_groups = list(gpu_topology.values())
# Assign co-routed experts to same domain
for i in range(num_experts):
best_domain = argmin(domain_load[domain] for domain in domain_groups
if can_place(i, domain))
placement[i] = best_domain[0] # pick first GPU in that domain
return placement
I’m oversimplifying — the real logic involves a min-cut clustering algorithm. But the gist is: don’t ignore the cost of cross‑node communication.
What Doesn’t Work: Over‑Optimizing the Router
I spent three months building a clever routing algorithm that predicted expert loads with a small neural net. It was beautiful. It improved load balance by 5% — but added 2ms of overhead per routing step. Net gain: zero. Actually negative, because the tokens had to wait for the router to finish its prediction.
The lesson: keep the routing logic simple. A two‑layer MLP with 256 neurons is enough. The placement algorithm matters far more than the router, because placement affects all future tokens, not just the current one.
AI Meets Cryptography Cloudflare Circl
Now here’s a tangent that became core to our system. In 2025, Cloudflare open‑sourced Circl — a cryptographic library for post‑quantum secure multi‑party computation. I had a crazy idea: what if we use secure aggregation protocols to compute expert load without revealing individual token routing patterns? In shared GPU clusters (like those rented via Vast.ai: Rent GPUs), the cloud provider shouldn’t know which experts are popular — that leaks business logic.
We integrated Circl’s threshold Paillier encryption to aggregate load statistics across GPUs privately. The coordinator receives encrypted load counts, decrypts the sum using distributed key shares, and decides placement without ever seeing per‑GPU loads. The overhead is ~100µs per aggregation — acceptable for placement decisions made every 100ms.
This is AI meets cryptography Cloudflare Circl in the real world. Not just for privacy — it also prevents adversarial tenants in a multi‑tenant cluster from inferring your model architecture by observing GPU utilization patterns.
Practical Considerations for On‑Premise Clusters
If you’re building an on‑premise GPU cluster for MoE, What is the best option to setup on premise GPU cluster for a small company gives a decent starting point. But the forum misses one key thing: you need at least two NVSwitch domains for MoE. Why? Because you want to reserve one domain for experts and one for router/non‑expert layers. Mixing them increases cross‑domain traffic by 60%.
Also, don’t overlook the network fabric. 5 Key Considerations when Building an AI & GPU Cluster mentions InfiniBand vs Ethernet. For MoE placement, we found that adaptive routing in InfiniBand is critical — it prevents packet reordering and allows the placement algorithm to rely on consistent latency estimates. Without adaptive routing, our placement decisions were stale within 10 seconds because the routing changed.
The Code You Actually Need
Enough theory. Let’s look at a real migration algorithm we use. This Python pseudocode implements a simple load‑balanced migration with expert replication:
python
def migrate_expert(exp_id, src_gpu, dst_gpu, state_dict):
# Step 1: Replicate expert to dst_gpu asynchronously
future = async_copy(state_dict, dst_gpu)
# Step 2: Update routing table atomically using NCCL allgather
with nccl_comm.allgather_barrier():
global_routing_table[exp_id] = [src_gpu, dst_gpu] # both available
# Step 3: Wait for copy to finish
future.wait()
# Step 4: Phased draining: new tokens go to both, old tokens finish on src
# After drain timeout (e.g., 5s), remove src from routing table
schedule_phase_out(exp_id, src_gpu, delay=5.0)
This avoids the dreaded “expert migration storm” where all tokens to an expert get stuck while the parameters move.
When Accelerating MoE Expert Placement Goes Wrong
I have to be honest — we broke production twice. Once because we migrated too aggressively. The routing table became inconsistent due to a race condition in our NCCL barrier. Tokent lost. Another time because we used GreenNode’s GPU cluster guide topology autodetection, which was wrong for our Supermicro chassis. The placement algorithm assigned experts to GPUs that didn’t share NVLink within the same node — our own fault for trusting a third‑party tool blindly.
Lesson: always verify topology with nvidia-smi topo -m and custom ping‑pong benchmarks.
The Future: Learned Placement Policies
As of mid‑2026, we’re deploying a reinforcement learning agent that learns placement policies from past batch traces. We call it PlaRL. It’s early, but it already beats our hand‑tuned heuristic by 12% in total token throughput for the DeepSeek‑MoE‑16B model. The agent runs on the scheduler node, using 1% of a single GPU. It reoptimizes every 500 steps.
If you’re working on this, don’t wait for a paper. “Accelerating distributed MoE expert placement” is an active problem, and the solutions are still primitive. The industry is obsessed with model size — nobody’s thinking about where the model sits.
Frequently Asked Questions
Q: What is the distributed system architecture?
A: In the context of MoE, it’s a multi‑agent system where routing nodes dispatch work to expert servers, which are stateful and communicate via all‑to‑all collectives. The architecture must support dynamic reconfiguration.
Q: Can I use a centralized scheduler?
A: For clusters under 64 GPUs, yes. Beyond that, use a two‑level hierarchy: per‑node schedulers that aggregate load and a global scheduler that makes placement decisions every 100ms.
Q: Does expert placement matter for training as well?
A: Yes, but the bottleneck switches to gradient communication. During training, focus on overlapping computation with communication rather than placement. For inference, placement is dominant.
Q: How do I measure if my placement is bad?
A: Profile the all‑to‑all latency for each layer. If any GPU’s send time is >50% of the total forward pass, your placement is unbalanced.
Q: Should I use Vast.ai for renting GPUs for MoE serving?
A: It’s viable if you ensure the GPUs you rent are within the same cluster (same NVSwitch domain). Vast.ai allows filtering by topology — use that.
Q: What about expert replication vs migration?
A: Replication is better for static hot spots. Migration is better for slowly shifting load. Use both: replicate to handle spikes, migrate to rebalance.
Q: How does Cryptography (Cloudflare Circl) help with placement?
A: It enables private load aggregation, protecting your model’s expert popularity distribution from leak to the cluster orchestrator.
Q: Does the number of experts matter?
A: Yes. With fewer than 8 experts, placement is trivial. With >128, you need hierarchical grouping. 32–64 is the sweet spot for current hardware.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.