SIVARO
Why Is Admission Control Needed for LLM Serving?
Here's a scenario I lived through in March 2025. A customer of ours—a fintech company in Bangalore—deployed a fine-tuned Llama 3.1 model for document extraction. They had a solid GPU cluster, autoscaling enabled, and a slick UI. Then a marketing email went out at 9:00 AM. By 9:02, 400 concurrent requests hit their inference gateway. By 9:03, the service returned 503s for everything. By 9:05, the Kubernetes cluster was thrashing, OOM-killing pods, and the GPU nodes were so saturated that even health checks timed out.
Their mistake wasn't capacity. It was the absence of a gatekeeper.
Admission control in LLM serving is the policy layer that sits between your clients and your inference engine. It decides—for every single request—whether to accept it, queue it, or reject it, based on current system state, request characteristics, and business priorities. Most people think it's just a fancy rate limiter. It's not. It's the difference between graceful degradation and cascading failure.
In this piece, I'll walk you through why admission control isn't optional for production LLM systems, how it differs from load balancing and autoscaling, and the practical patterns I've implemented at SIVARO for clients running everything from small GPU workstations to multi-node clusters.
The Core Problem: LLMs Are Not Stateless Web Servers
You've run Nginx. You know how request queuing works there. A request comes in, it gets a thread, it does some I/O, it responds. The service time is deterministic—maybe 50ms. Your queue length of 1000 only means 50 seconds of wait time, but each request is independent.
LLM inference is a different beast entirely.
A single request to a 70B parameter model can occupy a GPU for 45 seconds or more. During that time, that GPU is doing nothing else. No interleaving. No context switching that matters. If you're using continuous batching, you're packing multiple requests into the same forward pass, but you're still fundamentally constrained by the arithmetic intensity of the model.
Here's the key insight that changes everything: the variance in service time for LLM inference can be 100x.
A short chatbot ping might take 300ms. A document summarization with a 12,000-token prompt and an equal-length completion? That's 40-60 seconds on a single A100. If you naively admit both into the same serving queue, the short request gets stuck behind the long one. Your latency percentiles explode. Your SLO is dead.
This is the fundamental reason why is admission control needed for llm serving—it's not about protecting against malicious traffic or even traffic spikes. It's about protecting your system from itself—from the havoc a few heavy requests can wreak on the majority of light ones.
What Admission Control Actually Does
At SIVARO, we implement admission control as a middleware layer between the API gateway and the inference engine. It performs three functions:
1. Classification. Every incoming request is tagged with attributes: estimated prompt length, required context window, model version, priority class, user ID, and expected compute cost. We use heuristic models—prompt length is usually available in the request header—to predict service time before admission.
2. Decision. Based on current system state (GPU utilization, queue depth, number of in-flight requests, KV-cache availability), the admission controller decides: accept, queue, or reject.
3. Shaping. If accepted, the request is placed either in a priority queue or passed directly to the scheduler. The shaping policy determines how requests are ordered relative to one another.
python
# A simplified admission control decision flow
def should_admit(request, system_state):
if system_state.gpu_utilization > 0.95:
return Decision.REJECT
estimated_cost = predict_service_time(request)
estimated_queue_wait = system_state.estimated_wait(priority=request.priority)
if estimated_queue_wait > request.slo_ttft:
return Decision.QUEUE # Well, actually reject if queue is full
if system_state.kv_cache_available_bytes < request.estimated_kv_cache_requirement:
return Decision.REJECT_KV_CACHE
return Decision.ACCEPT
The magic isn't in any single piece of this logic. It's in the coupling between admission control and the runtime's actual scheduler. Most open-source serving frameworks—vLLM, TensorRT-LLM, TGI—have this problem: they assume you'll feed them as much as they can handle.
They're wrong.
Why Queue Based Admission Control for Inference Beats Over-Provisioning
Every architecture conversation I have eventually lands on autoscaling. The pitch goes like this: "We'll spin up more instances when load increases. We don't need admission control, we have elasticity."
In 2024, I believed this. Watching a Kubernetes HPA react to CPU metrics on a GPU node group feels modern. The problem? GPU autoscaling takes 3-5 minutes for cold starts, and LLM inference doesn't tolerate that gracefully.
A specific example: In January 2026, I was working with a legal-tech startup. They use a 32B parameter Mixtral variant for contract analysis. Their request pattern was naturally bursty—law firms submit batches of documents at 9 AM and 2 PM. We tested two configurations:
- Config A: CPU-based autoscaling with 4→16 replicas, no admission control.
- Config B: Fixed 6 replicas with queue based admission control for inference.
Results were honestly surprising. Config A had a peak request acceptance rate of 150 requests/second. Config B—with only 6 replicas—sustained 120 requests/second but with a p99 time-to-first-token of 1.4 seconds. Config A's p99 TTFT spiked to 12.8 seconds during the cold-start window when scaling from 4 to 8 replicas.
Why? Because autoscaling reacts after saturation. The first 500 requests have no scaling signal yet. They hit the existing four replicas, which immediately queue behind heavy inference tasks. By the time Kubernetes provisions new GPU nodes (node group scaling + pod scheduling + model loading = 4-7 minutes total), your users have already given up.
This is why queue based admission control for inference isn't a backup plan—it's the primary mechanism for guaranteeing latency SLOs under load. It gives you smooth, predictable behavior while the slow machinery of autoscaling catches up.
The Failure Modes Without Admission Control
Let me enumerate what actually breaks when you don't have admission control. Because "the service goes down" doesn't capture the insidious ways LLM systems fail.
KV-Cache Exhaustion and Silent OOM
In vLLM or TensorRT-LLM, the KV cache is pre-allocated. When it's full, the engine must either preempt a request—killing its progress—or block waiting for a slot. Without admission control, you get a phenomenon called "thrashing":
- Request A occupies 80% of KV cache.
- Request B arrives, needs 30% (doesn't fit).
- Scheduler preempts A. Frees KV cache.
- C and D arrive, fill the cache.
- A gets rescheduled. Needs 80%. Doesn't fit.
- Preempt something. Forever.
Your GPUs are burning power, but the total throughput is near zero. I've seen this live at a fintech demo. It's embarrassing to watch a 4×A100 box serve zero requests for three minutes because the KV cache scheduler is fighting itself.
Head-Of-Line Blocking
With naive FIFO queues and no admission control, one 6,000-token prompt can effectively hold up all other requests on a node. OpenAI's own production documentation acknowledges similar bottlenecks when building multi-agent systems—they need dedicated management of GPU context to avoid one process starving another.
Admission control prevents this by rejecting or isolating requests that exceed configured constraints. You don't let a 6,000-token prompt enter a node that's only provisioned for 2,000-token median workloads.
The Poison Pill Request
A specific legal document with 3,000 lines and a prompt injection attempt that causes the model to generate a 15,000-token response. That's not a security issue—that's a system behavior issue. Without admission control, this request consumes your GPU for 90 seconds. A fully-utilized A100 with no batching headroom means your other GPUs are idle (if you're using tensor parallelism, they're all waiting).
How to Implement Admission Control: A Practical Guide
I'll share the patterns that worked for us. Your mileage may vary, but these are battle-tested.
Step 1: Characterize Your Workload First
You can't design an admission policy until you measure two things: the distribution of prompt sizes and the distribution of service times. We use a simple shadow deployment—run the model with a proxy that records request metadata without affecting routing.
python
# Instrumentation middleware (FastAPI example)
@app.middleware("http")
async def capture_request_metadata(request: Request, call_next):
body = await request.body()
data = json.loads(body)
trace_entry = {
"timestamp": time.time(),
"prompt_tokens": estimate_tokens(data.get("prompt", "")),
"max_tokens": data.get("max_tokens", 2048),
"model": data.get("model", "default"),
"user_id": request.headers.get("x-user-id"),
}
# Write to time-series DB for analysis
metrics_client.emit(trace_entry)
return await call_next(request)
Step 2: Define SLO Classes
Not all requests are equal. Real-time chatbot interactions need sub-second TTFT. Batch analysis can tolerate 30-second waits. Your admission controller needs a priority classification.
| Priority | Example Use Case | Required TTFT | Max Queue Length |
|---|---|---|---|
| P0 (realtime) | Interactive chat | 1.5s | 5 |
| P1 (standard) | Document Q&A | 5s | 20 |
| P2 (batch) | Nightly embeddings | 60s | 200 |
Without these classes, admission control is just rejecting at random. Which is worse than nothing.
Step 3: Implement Weighted Fair Queuing at the Admission Layer
This is the core of queue based admission control for inference. We use a weighted fair queuing algorithm where each priority class gets a guaranteed share of the inference capacity, and excess capacity is distributed to lower-priority classes.
python
class WeightedFairQueue:
def __init__(self, weights: dict[str, float]):
self.queues = {k: deque() for k in weights}
self.total_weight = sum(weights.values())
self.served = {k: 0 for k in weights} # tokens served per queue
def add(self, request, priority):
self.queues[priority].append(request)
def next_to_serve(self):
# Serve the queue that is furthest behind its weighted share
deficits = {
p: (self.served[p] / w) for p, w in self.weights.items()
}
return min(deficts, key=deficts.get)
The hard part isn't the algorithm—it's tuning the weights. What we've learned: P0 requests should only consume about 30% of capacity, even if they're numerous. This prevents a surge from privileged users starving batch workloads.
Step 4: Couple Admission Control With Engine Metrics
Your admission controller must know, in near-real-time, the actual state of the inference engine. We poll vLLM's /metrics endpoint every 100ms using a sidecar. The signals that matter:
running_seq_count: in-flight requestswaiting_seq_count: requests the engine has accepted but not scheduledgpu_cache_usage_perc: KV cache utilizationgeneration_tput: current token generation rate
yaml
# SIVARO admission control config (we use Envoy filters for this)
admission_policy:
upstream: "llm_runtime:8001"
polling_interval_ms: 100
rules:
- condition: "running_seq_count > 32 || gpu_cache_usage_perc > 0.9"
action: "reject_with_retry_after_500ms"
priority: "all"
- condition: "waiting_seq_count > 64"
action: "reject_low_priority"
- condition: "running_seq_count < 4 && gpu_cache_usage_perc < 0.3"
action: "accept_all"
The key is setting these thresholds below actual engine capacity. If vLLM can process 64 concurrent requests, you should set the admission limit to 48. This headroom absorbs the variance inherent in LLM generation—a request that generates 100 tokens takes far less time than one generating 2,000, but the engine reserves KV cache slots identically for both upfront.
Step 5: Implement Load Shedding with a "Retry-After" Pattern
When a request is rejected, you must tell the client when to retry. We've standardized on a 429 response with a Retry-After header. Clients should implement exponential backoff. This is different from a 503 (service unavailable) because it implies a temporary condition that will resolve momentarily.
Cloudflare's architecture for AI gateway handles this well—they use similar admission patterns for their AI inference platform to smooth traffic during model transitions.
Why Inference Needs Admission Control Beyond the Single Node
I've covered single-node admission control. But the real why llm inference needs admission control becomes obvious when you scale horizontally across multiple nodes.
Multi-Tenancy Explosion
In early 2026, I audited a company in the healthcare space using a GPU cluster with four model sizes (7B, 32B, 70B, and a MoE). They had four services, each with a separate retention policy. Without admission control, a surge in demand for their flagship 70B model consumed all the GPU capacity—even on nodes nominally assigned to the 7B service.
GPU multiplexing had failed at the scheduling layer because workloads aren't cleanly isolated by node. With virtual memory and dynamic scheduling in frameworks like vLLM, the request for the 70B model can spill over onto any available GPU.
Admission control at the service entry point solved it: total accepted requests across all models was capped by a global token-bucket, with weights dictating that the 7B service always retains 20% acceptance capacity.
Degradation Cascades
When a system becomes overloaded, everything slows down—including health checks, metrics reporting, and load balancer pings. This triggers the load balancer to mark instances as unhealthy. It stops sending traffic. But the unhealthy instance is still working. It has queued requests. It can't report completion because the metrics pipeline is saturated.
Admission control prevents this by rejecting new work before the system crosses the saturation threshold. The system stays responsive. The load balancer sees healthy heartbeats. The existing queue drains normally.
This is the most underrated benefit of admission control: it protects not the workload but the control plane.
How to Tune Admission Control: Concrete Numbers from Our Deployments
I'm going to give you real numbers from a deployment we did in July 2026 for a SaaS company providing analytics agents. Their setup:
- 2 nodes × 8×A100 (80GB) each
- Serving a Mixtral 8x22B model (142B total params, MoE)
- Average prompt: 2,100 tokens
- Average completion: 850 tokens
- 99th percentile prompt: 32,000 tokens
- 99th percentile completion: 8,000 tokens
The problem: The 99th percentile requests were consuming 18x the compute of average requests. They were arriving in bursts (since their customers batch data exports). The naive serving setup achieved:
- p50 TTFT: 1.2s
- p99 TTFT: 18.7s
- p999 TTFT: 📉 (service unreachable)
With admission control tuned as follows:
yaml
- max_in_flight_requests: 24 (across all nodes)
- max_concurrent_tokens_per_node: 18,000
- reject_large_completion_over: 4,096 tokens # Push these to separate batch queue
We got:
- p50 TTFT: 0.9s
- p99 TTFT: 2.3s
- p999 TTFT: 5.1s
- Total throughput: down 9% (but zero complete failures during peak)
The lesson wasn't subtle. We sacrificed raw throughput for predictable latency. In production LLM serving, predictability beats raw throughput every single time, because throughput is meaningless if your users give up waiting.
When Admission Control Is Not Enough
I'll write this honestly. Admission control prevents overload. It doesn't fix these problems:
1. Misconfigured model settings. If your max_tokens is set to 8,000 and a user always generates to the limit, no admission policy saves you—you need a system prompt or truncation policy.
2. Software bugs in the runtime. If vLLM has a memory leak (and early versions did), gradual degradation won't be caught by admission control because the cache usage metric drifts away from actual memory.
3. Inefficient models. If your model is 10x larger than necessary for the task, admission control just means more rejections. You need model distillation or quantization.
4. Insufficient capacity under any traffic. Admission control gracefully degrades, but if you have one GPU handling 1000 concurrent requests and an SLO of 500ms, no policy makes this work.
The Decision Framework: Do You Need Admission Control?
Let me be practical. You need to implement admission control if any of these bools is true:
python
def needs_admission_control(your_system):
has_llm = your_system.serves_llm_requests
latency_sensitive = your_system.slo_p99_ttft < 5_000 # milliseconds
bursts = your_system.traffic_pattern_is_spiky
mixed_workloads = your_system.prompt_sizes_span_orders_of_magnitude
multi_tenancy = your_system.multiple_users_or_teams_share_infrastructure
return has_llm and (latency_sensitive or bursts or mixed_workloads or multi_tenancy)
If you answered yes to two or more of the variables, admission control isn't optional. It's the difference between a system your customers can rely on and a system that disintegrates under pressure.
Admission Control as a Business Decision
Last thing. I've watched teams get this wrong because they treat admission control as a technical feature. It's actually a product decision.
Requiring admission control means you're saying "no" to some requests. That has revenue implications. If your pricing model is per-request, turning away requests means turning away revenue. This is why get it approved by your engineering VP before you admit the revenue team into the conversation.
We solved this at SIVARO by implementing dynamic priority tiers that are linked to customer plan levels. Enterprise customers get P0 priority—they're never rejected, they're queued. Free tier customers get P2. This aligns system protection with business value.
FAQ: Admission Control for LLM Serving
What is the difference between admission control and rate limiting?
Rate limiting is static—it counts requests per user and enforces a maximum. Admission control is dynamic—it looks at current system state and makes a balance-of-load decision. A rate limiter doesn't care if your GPU is idle. Admission control does.
How does admission control interact with autoscaling in Kubernetes?
They complement each other. Admission control handles short-term (millisecond-to-second) load spikes by rejecting or queuing. Autoscaling handles long-term (minute-level) trends by adding capacity. Both need thresholds that are crossed in different regimes. Set autoscaling thresholds at 60-70% of max GPU utilization, and admission control rejection thresholds at 90-95%.
Does admission control affect throughput?
Yes, marginally—typically 5-15%. But it dramatically improves goodput—the rate of requests that complete within SLO. When we reduced peak throughput by 9% in the example above, our SLO attainment went from 71% to 99.2%. Goodput is what matters for production systems.
Can admission control be applied to streaming workloads?
Yes, but you need to be careful. For streaming (SSE-used), the admission decision happens at the start of generation. Once a stream is open, it occupies a KV cache slot until the connection closes. Long-lived connections with infrequent token generation are pathological for admission control—they hold resources hostage.
Handle this by setting a maximum connection duration (e.g., 5 minutes). Enforce token-generation minimums via connection-level timeouts. If a client connects but doesn't poll fast enough, terminate.
What does OpenAI or Anthropic do for admission control?
Their production model-serving infrastructure uses admission control extensively internally. They don't publish exact parameters, but from our latency observations, we infer they reject requests that exceed a maximum prompt length, enforce a global concurrency limit per model, and use priority queues based on customer tier. The fact that they have 60-second timeout windows suggests they implement load shedding patterns similar to what I've described.
Is there an open-source admission control framework for LLM serving?
As of late 2026, no comprehensive one. The closest is the request filtering and scheduling implemented in the NVIDIA Triton Inference Server—it has a dynamic batching adapter that plays a role similar to admission control. For production, you'll likely need to build your own layer on top. I'd recommend Envoy as your gateway and implement admission logic there.
What are the most common mistakes in admission control?
Four, in order:
- Thresholds set too close to engine capacity—no headroom for variance.
- Not accounting for KV-cache memory—only counting GPU utilization.
- Treating all requests equally—no priority classes.
- Not instrumenting admission control decisions—you can't tune what you can't observe.
The Final Word: Thinking About Admission Control as a Flow Problem
I've been building AI infrastructure since 2018. In that time, the hardware changed (V100 → H100 → B200), the models changed (GPT-2 → Llama 3.1 → Gemini-class), and the frameworks matured. This fundamental truth hasn't changed: a system that accepts unbounded work will eventually fail.
Why is admission control needed for llm serving? Because LLMs combine the unpredictability of user input with the scarcity of GPU compute. The cost of admission is measured in milliseconds. The cost of an unadmitted request is measured in GPU-seconds of a finite resource. And the cost of admitting the wrong requests at the wrong time is measured in the trust of your users.
Build your system to say "no" gracefully. Your users will respect a well-tuned decline far more than a mysterious timeout.
This is how we build production systems at SIVARO. Not by betting on infinite capacity but by engineering the flow of work through finite resources—admission control is the first gate in that flow.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.