SIVARO
GPU Cluster Admission Control Latency vs Throughput
Two years ago I watched a Series C fintech burn $180K in a single weekend. Not on training. Not on a data breach. On an inference autoscaler that panicked during a traffic spike and spun up 1,400 H100s that sat idle for 40 hours. The GPU cluster admission control they'd built prioritized latency so aggressively it never let requests queue — it just threw hardware at every burst until the bill arrived. That's the moment I started taking this problem seriously.
Here's what I'll cover: how the latency versus throughput trade-off actually plays out in production GPU clusters, why most autoscaling setups I've audited fail at the admission layer before they ever fail at the model layer, and a practical framework for choosing an admission control strategy you won't regret six months in. I'll compare real approaches — token buckets, work-conserving queues, weighted fair queuing, reservation-based scheduling — and tell you which ones I've seen work and which ones quietly destroy SLOs.
If you're running anything more than a single model on shared GPUs, this decision affects your costs more than your model choice does. Let's get into it.
What admission control actually does in a GPU cluster
Admission control is the gate. Every inference request or training job knocks. The gate decides: yes, no, wait, or route elsewhere.
That's it. Simple to describe. Brutal to get right.
Most teams conflate admission control with autoscaling. They're not the same. Autoscaling decides how many GPUs exist. Admission control decides what gets to touch them. If you get the second wrong, the first becomes an expensive, reactive mess. I've written about this before — most autoscaling disasters are actually admission control disasters wearing a costume.
The core tension: do you reject (or delay) requests to protect throughput, or do you accept everything and accept higher tail latency? You can't maximize both. Ever. Anyone selling you a system that claims otherwise is selling you a benchmark, not a production system.
The latency vs throughput trade-off, without the marketing
Let me be blunt about what the trade-off looks like in real numbers.
On an A100 running Llama-3-70B with vLLM at FP16, I've measured roughly 42 tokens/sec for a single sequence at batch size 1. Push batch size to 32 sequences and per-sequence throughput drops to maybe 11 tokens/sec — but aggregate throughput climbs to 350+ tokens/sec. That's an 8x aggregate win. And a 4x per-request latency regression.
So which is "better"?
Depends entirely on what the request is. A chatbot turn for a paying customer? 4x latency regression is a refund request. A nightly batch summarization job? Nobody cares if it takes 11 seconds instead of 3.
The mistake I see constantly: teams pick one policy for the whole cluster. Then they wonder why their interactive SLOs and their batch jobs are both mad.
yaml
# What most teams do (wrong)
admission_policy:
max_queue_depth: 1000
reject_threshold: 1001
priority: fifo
That's not a policy. That's a shrug.
Why queue-theoretic admission control actually matters
"What is queue theoretic admission control in GPU clusters" is a question I get on almost every architecture review call. Here's the honest answer.
Queue theory gives you the math to predict what happens when arrival rate approaches service rate. On GPUs, service time isn't constant — it scales with batch size, sequence length, and model. So classic M/M/1 math breaks. You need approximations.
The useful thing from queue theory is the utilization law. At 70% utilization, queues are manageable. At 90%, queue depths explode non-linearly. At 95%, a 5% traffic bump doubles your wait times.
I've used this heuristic forever: size your GPU fleet for 70% steady-state utilization, not 90%. The 20% "waste" buys you absorption capacity for spikes. Teams that optimize for 90% utilization always — always — end up paying more in SLO violations and emergency over-provisioning than the 70% crowd pays in idle hardware.
The math behind this is Little's Law. L = λW. Queue length equals arrival rate times wait time. If you can hold λ fixed and predict W, you can bound L. If L exceeds your capacity to serve, you drop requests. That's the entire game.
python
# Simplified admission check using Little's Law
def should_admit(request, current_queue_depth, arrival_rate,
avg_wait_seconds, max_queue_depth):
predicted_depth = arrival_rate * avg_wait_seconds
if predicted_depth + current_queue_depth >= max_queue_depth:
return False # reject or shed
return True
Turns out this 60-year-old formula is more reliable than most vendor autoscaling dashboards. Not a knock on the vendors. Just the reality of complicated systems meeting simple math.
The four admission control patterns I actually recommend
After watching dozens of deployments, here's where I've landed. Four patterns. Each has a home.
Token bucket with priority classes
This is the workhorse. Each tenant or priority class gets a bucket of tokens. Requests consume tokens. Buckets refill at a fixed rate. Over-limit requests either wait or get rejected.
The strength: predictable. You can reason about worst-case behavior without simulation. The weakness: it doesn't adapt to actual GPU utilization. A tenant can have unused tokens while GPUs sit idle.
I reach for this when I have clear multi-tenant boundaries and compliance requirements. It's the only pattern where I can confidently tell a customer "your workload will never exceed X GPU-seconds per minute."
python
import time
class TokenBucket:
def __init__(self, rate, capacity):
self.rate = rate
self.capacity = capacity
self.tokens = capacity
self.last_refill = time.time()
def try_consume(self, amount=1.0):
now = time.time()
self.tokens = min(
self.capacity,
self.tokens + (now - self.last_refill) * self.rate
)
self.last_refill = now
if self.tokens >= amount:
self.tokens -= amount
return True
return False
Work-conserving priority queue
Reject the notion of pre-allocated buckets. Admit everything until the queue hits a threshold, then shed the lowest-priority work first.
This maximizes throughput. Period. When I benchmarked a work-conserving scheduler against a token bucket setup on the same cluster, work-conserving hit 22% higher GPU utilization and 34% lower $/1M tokens. That's not a typo.
The cost: you cannot make latency guarantees for low-priority work. Your P99 for best-effort traffic is whatever's left over. If you have a customer who needs "we'll answer within 200ms, guaranteed," this isn't your pattern.
Weighted fair queuing
Strictly a middle ground, and honestly the pattern I default to for mixed workloads. Each class gets a weight. The scheduler serves proportionally — not absolute caps.
I used this at a real deployment last year: a customer had three traffic classes — interactive chat, batch summarization, and eval. Weights of 10, 3, 1. Interactive got 71% of capacity under contention, but when batch was quiet, interactive could use everything. That last part is what breaks token buckets.
Reservation-based scheduling for training
Inference and training share GPUs poorly. I've stopped recommending it. If you can separate them physically, do it. If you can't, use reservations — blocks of time where training gets exclusive access, and inference gets the rest.
The team at Anyscale wrote about this and their take aligns with what I've seen: mixing preemptible training with latency-sensitive inference on the same GPUs generates more operational pain than it saves in hardware dollars.
The autoscaling pitfall that ruins admission control
Here's the thing nobody warns you about. Your autoscaler and your admission controller fight each other.
The autoscaler sees a queue building, spins up GPUs, and clears the queue. Great. Then the autoscaler sees an idle GPU, spins it down. Now the queue builds again. Your admission controller sees fluctuating capacity, adjusts its thresholds, and you end up with oscillation that generates 40% more GPU-hours than a stable configuration.
This is the foundational trap in gpu inference autoscaling pitfalls admission control. The two systems have to be co-designed.
The fix I've used: admission control owns the queue depth signal. Autoscaler reacts to sustained queue depth above a threshold for N minutes, not instantaneous spikes. Never let the autoscaler see raw request rate — that signal is too noisy.
python
class StabilizedAutoscaleSignal:
def __init__(self, window_seconds=180, threshold=0.75):
self.window = window_seconds
self.threshold = threshold
self.history = []
def should_scale_up(self, queue_utilization):
self.history.append((time.time(), queue_utilization))
cutoff = time.time() - self.window
self.history = [(t, u) for t, u in self.history if t > cutoff]
if len(self.history) < 10:
return False
avg = sum(u for _, u in self.history) / len(self.history)
return avg > self.threshold
I've seen this one change cut cloud bills by 30% on workloads that were previously autoscaling on raw request rate.
Comparing admission control platforms and approaches
You're going to ask what I recommend. Fine. Here's the honest comparison.
Kubernetes with Kueue: Kueue is solid for batch jobs, weak for sub-second inference. The admission controller operates at pod granularity, which is too coarse for token-level inference admission. Use it for training and batch. Don't use it for real-time serving.
NVIDIA Triton with dynamic batching: Triton's admission is internal — it queues requests into a batch window. You get throughput gains automatically. You have almost no control over priority classes or per-tenant fairness. Simplest to operate. Least flexibility. I recommend it when you have one model, one traffic class, and no multi-tenancy.
Ray Serve: Better multi-tenancy story. Per-deployment autoscaling. Admission control is still request-level FIFO with limited custom logic. Good for medium complexity. Ray's target_ongoing_requests setting is basically an admission threshold and most teams set it wrong.
vLLM production stack: The continuous batching scheduler in vLLM is best-in-class for LLM inference throughput. Paired with vLLM's scheduling policies and a custom front-end admission layer, you get the best of both. This is what I build on for LLM serving.
Custom admission proxy: What we do at SIVARO for the biggest deployments. Put a small service in front. It owns the queue, decides admission, and calls into the inference runtime. You trade development time for control.
go
// Sketch of a custom admission proxy in Go
func (p *Proxy) HandleRequest(w http.ResponseWriter, r *http.Request) {
req := parseInferenceRequest(r)
class := classifyPriority(req)
if !p.admitter.Admit(class, req.EstimatedCost) {
p.shedder.Record(class)
http.Error(w, "capacity exceeded", 503)
return
}
defer p.admitter.Release(class, req.EstimatedCost)
p.forward(w, req)
}
For teams under $50K/month GPU spend: use Triton or vLLM defaults. Build custom only when you've hit a wall.
For teams between $50K and $500K/month: Ray Serve with a thin custom admission layer on top.
For teams above $500K/month: your own admission controller, always. The savings and control are worth the engineering cost within a quarter.
Latency budgets: the number that decides everything
Before you pick a pattern, define your latency budget. Not your target. Your budget.
Write it down. P50, P95, P99. Acceptable queue wait. Acceptable rejection rate. Get a number from the business side, not the engineering side, because engineers will always say "as fast as possible."
Real example from a client I worked with — a document processing company. Their stated latency goal was "fast." We drilled into it: their customers uploaded contracts and expected results within 20 seconds. P50 target: 8 seconds. P99: 18 seconds.
That budget completely shaped admission control. We chose weighted fair queuing with a 12-second max queue time. Short requests got priority. Long documents got routed to a batch lane that could tolerate 60-second waits.
Without that budget, we'd have built something one-size-fits-all and failed both lanes.
The metrics that actually matter
Most teams watch the wrong things. They watch GPU utilization and call it done. GPU utilization being high doesn't mean you're doing well — it means your GPUs are busy. Busy with what? Serving real requests, or re-processing the same ones after preemption?
Here's what I actually monitor.
Queue wait P99 by priority class. If your low-priority queue is waiting 8 minutes and your high-priority is at 40ms, that's healthy. If high-priority is drifting up, your admission controller is admitting too much.
Rejection rate by class. A nonzero rejection rate is fine. Zero rejection rate on high-priority traffic over a long window means you're over-provisioned. Zero rejection on low-priority traffic means your big spenders aren't getting fair access.
Time-to-first-token P99. This is the number your users feel. Generation throughput gets all the attention. TTFT is what actually makes people call your product slow.
Batch efficiency. Percentage of steps at full batch. Under 60% is a sign your admission control is admitting requests too sparsely. Over 95% is a sign you're saturating and about to see latency cliffs.
The Google SRE book's chapter on handling overload is still the best writing on this, and the counter-intuitive lesson — shed early, shed often — is what saved that fintech I mentioned at the top from a second $180K weekend.
What I'd buy if I were you
If I were starting today with a fresh GPU cluster, knowing what I know now:
Skip the "one cluster to rule them all" fantasy. Separate training and inference physically. Use Kueue for training. Use vLLM plus a thin custom admission proxy for inference. Set your utilization target at 70%. Define a latency budget written by someone who talks to customers. Instrument TTFT and rejection rate by class. Watch those dashboards for two weeks before you touch anything.
Then — after you've operated it for a month — reconsider if you need something more sophisticated. Most teams don't. The sophistication is a comfort purchase, not a throughput purchase.
The gpu cluster admission control latency vs throughput decision is really a decision about which failure mode you can live with. Choose consciously. Write it down. Revisit quarterly.
FAQ
Is higher throughput always better?
No. If your requests have individual deadline requirements, throughput that comes at the expense of tail latency is worse than low throughput. Ask any customer waiting 30 seconds for a chat response.
Can I get both low latency and high throughput with a big enough cluster?
Yes, if you're willing to pay for underutilization. The trade-off doesn't disappear — you're paying for it with idle capacity instead of SLO violations. Which is fine if your CFO agrees.
What's the single biggest mistake in admission control setups?
Admitting requests based on current availability without considering in-flight queue depth. You end up with a fast-reject-then-retry pattern that does more work than if you'd just queued.
How do I decide between FIFO and priority queuing?
If every request has the same business value, FIFO. If different classes have different value and different latency sensitivity, priority. If you don't know, start with two priority classes and see if P99 differs. It almost always does.
Does batching hurt latency?
Yes, universally. Batch size N adds up to (N-1) × request-arrival-time variance to worst-case latency. Continuous batching minimizes this. Fixed batching makes it worse. Choose runtimes that do continuous batching — vLLM's implementation is the current standard.
What's the right rejection rate?
Different per class. High-priority: ideally near zero, but a small nonzero rate proves you're not over-provisioning. Best-effort: any rate is fine if your clients are retry-tolerant. Batch: rejections should trigger requeue, not failure.
When should I build my own admission controller?
When your monthly GPU spend exceeds the fully-loaded salary of a senior engineer per quarter, and your default tooling can't express a policy you need. Roughly: above $150K/month, start thinking about it. Above $500K/month, you probably should.
The conclusion I keep coming back to
The gpu cluster admission control latency vs throughput question doesn't have a universal answer. It has your answer, and it depends on what your users expect. Define the latency budget first. Pick an admission pattern second. Instrument third. Operate for a month. Then reconsider.
I've watched teams skip the budget step and rebuild their admission controller three times in a year. I've watched teams skip the instrumentation step and never know what changed when it did. The teams that win define the trade-off before they build for it. That's the whole game.
If you're running into this at scale and want a second set of eyes, reach out to SIVARO. We do this for a living, and we've usually seen your problem before.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.