Why LLM Inference Needs Admission Control

Why LLM Inference Needs Admission Control

Queue-based admission control for inference isn't optional infrastructure. It's the difference between a system that degrades gracefully and one that falls over at 2 PM on a Tuesday when the sales team runs their demo.

I learned this the hard way. In early 2025, we were serving a customer-facing summarization feature for a healthcare SaaS company. Their traffic spiked 4x when they announced the feature in their newsletter. The model was fine. The GPUs were fine. The ingress layer collapsed because we had no mechanism to say "no" to requests we couldn't serve.

This article explains what admission control means for LLM serving, why your inference stack needs it, and how to implement queue-based admission control for inference without destroying your user experience.

What you'll learn: Why admission control is the forgotten pillar of production AI systems, how to design admission policies that protect your SLOs, and concrete implementation patterns with code.

---

The Core Problem: LLMs Break Normal Backpressure Assumptions

Most distributed systems handle overload through backpressure. Your database hits a connection limit, clients retry with exponential backoff, and everything stabilizes.

LLM inference doesn't work that way.

Here's why. A typical REST API has a predictable response time. A database query might take 50ms or 500ms depending on load. But a single LLM request can take 2 seconds or 40 seconds depending on prompt length, output tokens, and what else is running on your GPUs.

At SIVARO, we measured request latency variance of 20x on the same model with the same hardware. The difference was queue depth and preemption behavior.

When you throw 200 concurrent requests at an inference server that can handle 50, you don't get slow responses. You get timeout cascades. Requests pile up, each one's time-to-first-token (TTFT) explodes, clients retry, and now you have 400 requests contending for the same GPUs.

This is why LLM inference needs admission control: without it, the system doesn't degrade predictably. It collapses.

What Admission Control Actually Means for Inference

Admission control is the practice of deciding whether to accept a request before you commit compute resources to it.

For traditional web services, admission control is a nice-to-have. You shed load, return 503s, and move on.

For LLM inference, admission control is a requirement because of three constraints:

1. GPU memory is finite and non-preemptible. You can't swap a model's KV cache to disk mid-generation without losing the request.
2. Generation time is unbounded. You don't know how long a request will run until it completes.
3. Batching is everything. Throughput optimization requires grouping requests, which means you need a scheduling layer, not just a load balancer.

The practical implication: your inference server needs to accept requests into a bounded queue, evaluate whether it has resources to serve them, and reject early when it doesn't.

Anthropic's outage on June 2025 demonstrated what happens without this. A sudden traffic surge led to queue backlogs of 30+ minutes for API customers. The company had to deprioritize non-critical features to recover. Their infrastructure eventually stabilized, but the reputational damage from "expected 30 minute wait" messages was significant.

Why Traditional Rate Limiting Isn't Enough

Rate limiting is not admission control. They solve different problems.

Rate limiting prevents clients from sending more than X requests per second. It protects against abusive or buggy clients. It's a client-side contract.

Admission control protects your serving capacity. It's a server-side decision based on current load, not just request frequency.

Example: you might have a rate limit of 10 requests per second per API key. But if all 10 requests arrive with 4,000-token prompts and request 2,000-token completions, that's roughly 24,000 tokens of compute per second per key. Ten active keys means 240K tokens/second of demand. Your GPU cluster can maybe handle half that.

Rate limiting won't save you. Your clients are following the rules perfectly. But your latency SLO is already broken.

A queue-based admission control for inference system looks at work (tokens to process), not just requests. This is a critical distinction that most tutorials miss.

The Admission Control Algorithm That Works

We've settled on a layered approach at SIVARO. It's not elegant. It works.

Layer 1: Token bucket rate limit per API key (coarse protection)
Layer 2: Global admission queue with max queue depth
Layer 3: Bounded work estimator (prompt tokens + max_tokens)
Layer 4: Load shedding based on GPU queue depth

Here's what this looks like in pseudocode:

python
class InferenceAdmissionController:
    def __init__(self, max_queue_depth, max_batch_work):
        self.queue = asyncio.PriorityQueue(maxsize=max_queue_depth)
        self.max_batch_work = max_batch_work
        self.current_batch_work = 0
    
    async def admit(self, request):
        estimated_work = estimate_tokens(request.prompt) + request.max_tokens
        
        # Layer 4: Check if GPUs are already saturated
        if self.current_batch_work + estimated_work > self.max_batch_work:
            return AdmissionRejected(reason="GPU saturation", retry_after_ms=1000)
        
        # Layer 2: bounded queue
        try:
            self.queue.put_nowait((request.priority, request, estimated_work))
            return AdmissionAccepted()
        except asyncio.QueueFull:
            return AdmissionRejected(reason="Queue full", retry_after_ms=500)

That's the skeleton. The real implementation needs more nuance.

Priority Classes Matter More Than You Think

Not all requests deserve the same treatment. An interactive chat request from a paying customer waiting for a response has different tolerance than a batch summarization job that can wait 5 minutes.

We use three priority classes:

yaml
priority_classes:
  interactive:
    max_queue_time_ms: 500
    queue_weight: 10
    preemption: true
  standard:
    max_queue_time_ms: 5000
    queue_weight: 3
    preemption: false
  batch:
    max_queue_time_ms: 60000
    queue_weight: 1
    preemption: false

Interactive requests bypass the queue when possible. Batch requests get deprioritized when load is high.

The key insight: admission control isn't just about rejecting requests. It's about making sure the right requests get served at the right time.

Preemption Is Tricky but Powerful

LMCache's work on prefix caching in late 2025 showed that prompt caching reduces TTFT by up to 60% for repeated prompts. Preemption interacts with this in complex ways.

If you preempt a batch request mid-generation, you lose the KV cache for that sequence. If you're running a model with a 128K context window, that's potentially gigabytes of GPU memory you just freed. But if that request was 80% complete, you've wasted the compute.

We implemented a cost-benefit preemption policy:

python
def should_preempt(current_request, incoming_request):
    # Don't preempt requests that are >70% complete
    if current_request.completion_ratio > 0.7:
        return False
    
    # Preempt batch jobs for interactive requests
    if current_request.priority == 'batch' and incoming_request.priority == 'interactive':
        return True
    
    # Preempt only if incoming has high priority AND we can cache the current
    if incoming_request.priority == 'interactive' and current_request.prompt_cacheable:
        est_wasted_work = current_request.completion_ratio * current_request.est_total_work
        est_saved_latency = incoming_request.est_latency * 0.5
        return est_saved_latency > est_wasted_work
    
    return False

Preemption decisions need to happen in under 5 milliseconds. You can't afford a complex evaluation when GPUs are idling waiting for the scheduler.

Implementation: Queue-Based Admission Control for Inference

The most practical pattern we've validated in production is a token-based admission queue paired with a work estimator.

Full implementation:

python
import asyncio
import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class AdmissionDecision:
    accepted: bool
    estimated_wait_ms: int
    max_queue_time_ms: int

class TokenAdmissionController:
    """
    Queue-based admission control for inference systems.
    Uses both request count and estimated token work.
    """
    
    def __init__(self, 
                 max_concurrent_tokens=1_000_000,
                 max_queue_depth=500,
                 token_scaling_factor=0.8):
        self.max_tokens = max_concurrent_tokens
        self.max_queue = max_queue_depth
        self.scaling = token_scaling_factor
        
        # Track active work
        self.active_tokens = 0
        self.queue = asyncio.PriorityQueue(maxsize=max_queue)
        self.queue_depth = 0
        
        # For metrics
        self.total_accepted = 0
        self.total_rejected = 0
        self.total_queued = 0
        
    def estimate_work(self, prompt_token_count, max_completion_tokens):
        """Rough estimate of GPU work for a request."""
        # 2x multiplier for attention compute
        return int((prompt_token_count + max_completion_tokens) * 
                   self.scaling * 2)
    
    def can_accept_immediately(self, estimated_work, request_priority='standard'):
        if self.active_tokens + estimated_work < self.max_tokens * 0.8:
            # Headroom available for interactive and standard requests
            if request_priority == 'batch':
                return self.active_tokens + estimated_work < self.max_tokens * 0.5
            return True
        return False
    
    async def admit(self, 
                    request_id: str,
                    prompt_token_count: int, 
                    max_completion_tokens: int,
                    priority: str = 'standard',
                    max_wait_ms: int = 5000) -> AdmissionDecision:
        
        estimated_work = self.estimate_work(prompt_token_count, max_completion_tokens)
        
        # Immediate acceptance check
        if self.can_accept_immediately(estimated_work, priority):
            self.active_tokens += estimated_work
            self.total_accepted += 1
            return AdmissionDecision(accepted=True, estimated_wait_ms=0, 
                                     max_queue_time_ms=0)
        
        # Queue admission check
        if self.queue_depth >= self.max_queue:
            self.total_rejected += 1
            return AdmissionDecision(accepted=False, 
                                     estimated_wait_ms=-1,
                                     max_queue_time_ms=0)
        
        # Estimate queue wait time based on current load
        avg_processing_rate = self.estimate_processing_rate()
        estimated_wait_ms = int((self.active_tokens / avg_processing_rate) * 1000)
        
        if estimated_wait_ms > max_wait_ms:
            self.total_rejected += 1
            return AdmissionDecision(accepted=False, 
                                     estimated_wait_ms=estimated_wait_ms,
                                     max_queue_time_ms=max_wait_ms)
        
        # Accept into queue
        try:
            self.queue.put_nowait((priority, request_id, estimated_work))
            self.queue_depth += 1
            self.total_queued += 1
            return AdmissionDecision(accepted=True, 
                                     estimated_wait_ms=estimated_wait_ms,
                                     max_queue_time_ms=max_wait_ms)
        except asyncio.QueueFull:
            self.total_rejected += 1
            return AdmissionDecision(accepted=False, 
                                     estimated_wait_ms=-1,
                                     max_queue_time_ms=0)
    
    def request_completed(self, estimated_work):
        """Call this when a request finishes."""
        self.active_tokens = max(0, self.active_tokens - estimated_work)
        self.promote_from_queue()
    
    def promote_from_queue(self):
        """Pull next request from queue when capacity frees."""
        try:
            priority, request_id, estimated_work = self.queue.get_nowait()
            self.queue_depth -= 1
            self.active_tokens += estimated_work
            # Signal the scheduler that this request is ready
            # In practice: pass to execution engine
            return (priority, request_id)
        except asyncio.QueueEmpty:
            return None

Watch out for one thing: estimating work is hard. vLLM's continuous batching and NVIDIA's TensorRT-LLM both handle scheduling at the framework level. Your admission controller should complement, not fight, the framework's own batching logic.

The best rule of thumb we've found: your admission controller should know roughly how many requests the inference engine can handle per second, and gate at 70-80% of that number based on current conditions.

Measuring the Right Things

Most teams measure tokens-per-second and latency percentiles. Those are fine. But for admission control, you need different metrics.

Queue depth over time. If your queue is consistently above zero during off-peak hours, you have a misconfiguration. If it peaks and drains, you're fine.

Admission rejection rate by priority class. If interactive requests are getting rejected more than 1% of the time, your capacity planning is wrong or your admission thresholds are too aggressive.

Effective GPU utilization. Here's the trick: admission control that's too conservative leaves GPUs idle. The GPU at 60% utilization with zero queue is wasteful. You want it at 85-90% utilization with a small queue.

Metric	Target	Description
Queue depth at P95	< 20% of max	Queue shouldn't be consistently full
Rejection rate (interactive)	< 1%	Over-aggressive shedding hurts UX
Rejection rate (batch)	< 10%	Tiered rejection is acceptable
Request-to-token latency	< 2 seconds	Includes admission decision time

We sample these metrics every 5 seconds and feed them into a dashboard. The hardest part is tuning the thresholds. We iterated for three weeks with our infrastructure team before landing on values that worked for our traffic patterns.

Admission Control and Autoscaling: They Need Each Other

Admission control isn't a substitute for autoscaling. It's a bridge.

When traffic spikes, admission control rejects requests that would otherwise queue for minutes. That gives your autoscaler time to spin up new pods. RunPod's June 2025 update documented this pattern: they use admission control combined with Kubernetes Event-driven Autoscaling (KEDA) to handle burst traffic.

The sequence:

1. Traffic increases 3x
2. Admission controller starts rejecting low-priority batch requests
3. Rejection triggers autoscaling alerts
4. New GPU instances spin up (60-120 second lag)
5. Admission thresholds increase automatically
6. System returns to equilibrium

Without step 2, the system would struggle through step 4 with degraded performance. The queued requests would create timeout cascades that force clients to retry, making everything worse.

We built this pattern into our reference implementation:

yaml
admission_control:
  autoscale_cooldown_seconds: 120
  min_reject_rate_for_autoscale: 0.02
  max_queue_delay_before_scale: 5000

If rejection rate exceeds 2% for more than 30 seconds, trigger autoscale. This prevents both under-scaling (waiting too long to scale) and over-scaling (scaling on noise).

Priority Queues Need Careful Design

The naive approach to priority queues: implement a PriorityQueue and assign priorities. Then realize that starvation is real for low-priority requests.

A batch summarization job with priority 1 might sit in the queue forever while interactive requests with priority 10 flood in.

We use a weighted fair queuing model. Each priority class gets a minimum service share:

python
class WeightedFairAdmissionQueue:
    def __init__(self):
        self.classes = {
            'interactive': {'weight': 10, 'queue': [], 'last_service': time.time()},
            'standard': {'weight': 3, 'queue': [], 'last_service': time.time()},
            'batch': {'weight': 1, 'queue': [], 'last_service': time.time()},
        }
        self.total_weight = 14
    
    def next_request(self):
        now = time.time()
        eligible_classes = []
        
        for name, cls in self.classes.items():
            if cls['queue']:
                # Check if this class is due for service based on weight
                share_interval = 1.0 / (cls['weight'] / self.total_weight)
                if now - cls['last_service'] >= share_interval:
                    eligible_classes.append((name, cls))
        
        if not eligible_classes:
            # Fallback: serve whoever has been waiting longest
            return self.serve_oldest()
        
        # Serve the highest priority eligible class
        eligible_classes.sort(key=lambda x: -x[1]['weight'])
        return self.serve(eligible_classes[0][0])

This ensures batch requests get served even under high interactive load, just less frequently.

The Production Realities Nobody Talks About

Ghost Requests

Admission control decisions happen in under 10ms. But between decision and actual execution, things change. The GPU might have failed. Another request might have been promoted early. You need to handle the case where a request is accepted but never executes.

We call these ghost requests. They leak reservation slots and eventually block legitimate traffic.

Solution: add a lease system. Each accepted request gets a lease that expires after X milliseconds. If the request isn't picked up by the scheduler within the lease period, the slot is freed.

python
class AdmissionLease:
    def __init__(self, request_id, expires_in_ms=1000):
        self.request_id = request_id
        self.expires_at = time.time() + (expires_in_ms / 1000)
    
    def is_valid(self):
        return time.time() < self.expires_at
    
    def renew(self, additional_ms):
        self.expires_at = time.time() + (additional_ms / 1000)

Cold Starts

When you're using serverless GPU platforms like Modal or Replicate, cold starts can take 5-10 seconds. If your admission controller assumes a request will hit a warm GPU, it might make poor decisions.

Our implementation tracks "warm" vs "cold" capacity separately. Requests going to cold instances get a worse admission decision because the effective time-to-completion is longer.

Model Size Matters

Admission control for a 7B parameter model is different from admission control for a 70B parameter model. The KV cache for a 70B model with 100 concurrent requests can consume 200+ GB of HBM. With one model per GPU, you're looking at multiple A100s just for cache.

For multi-model serving, admission control needs to factor in model placement and GPU memory. SGLang's RadixAttention does some of this at runtime, but your admission layer should understand which models are co-located and whether a request can be served by cached prefixes from previous requests.

Admission Control Is the Opponent of GPU Utilization

The uncomfortable truth: aggressive admission control fights against GPU utilization. And GPU utilization is how you justify your infrastructure spend to the CFO.

I spent two months at a finance customer's site in late 2025 tuning this tradeoff. They had 8x A100s running a 70B model. The CFO asked why utilization was only 42% after we implemented admission control. Previously, it was at 91% because requests were always going through — but their error rate was 5.7% and p99 latency was 28 seconds.

The answer: we shifted from "keep GPUs busy" to "meet SLOs" and the right utilization happens to be 42-55% at the current scale. If they doubled traffic, utilization would rise. But they were paying for readiness, not just utilization.

That's the real insight around why LLM inference needs admission control: GPUs are expensive and finite, so you optimize for the customer contract, not the utilization metric.

Troubleshooting Common Admission Control Failures

Problem: Rejection Rate Too High

Check for:

• Autoscaling lag (Kubernetes nodes taking >90s to join)
• Work estimator overestimating token counts
• Queue depth cap set too low
• Priority classification incorrectly flagging requests

Problem: SLOs Still Missed With Admission Control Active

Check for:

• Admission decisions without actual resource reservation
• Scheduler ignoring admission decisions (common with multi-worker inference engines)
• Token count estimates way off (happens when prompts get truncated or completion lengths vary wildly)

Problem: GPU Utilization Dropped After Implementing Admission Control

Check for:

• Scaling factor too conservative (start with 0.7-0.8 of estimated capacity)
• Queue waiting times too restrictive
• Lack of retry logic at the client level causing request abandonment

Practical Implementation Steps

If you're building this today, here's the sequence I recommend:

Week 1: Instrument before you control. Track request accept/reject, queue depth, and estimated work. Don't implement admission control yet. Just observe.

Week 2: Add a soft admission threshold. Set it at 80% of your estimated GPU capacity. Reject requests beyond that but return a HTTP 429 with a Retry-After header.

Week 3: Implement priorities. Add interactive vs batch.

Week 4: Tie into autoscaling. Use admission metrics to trigger scale events.

Week 5: Hardening. Add lease times, handle edge cases, tune thresholds.

python
# Week 1 instrumentation example
class InstrumentedInferenceServer:
    async def handle(self, request):
        start = time.time()
        
        # Measure prompt tokens accurately
        prompt_tokens = count_tokens(request.prompt)
        est_completion = request.max_tokens if hasattr(request, 'max_tokens') else 2048
        
        # Log every request characteristics
        metrics.record('request_incoming', {
            'prompt_tokens': prompt_tokens,
            'est_completion_tokens': est_completion,
            'total_est_work': prompt_tokens + est_completion,
            'model': request.model,
            'priority': request.priority
        })
        
        # Process...
        result = await self.process(request)
        
        # Record actual usage
        metrics.record('request_processed', {
            'actual_completion_tokens': result.completion_tokens,
            'latency_ms': (time.time() - start) * 1000,
            'time_to_first_token_ms': result.first_token_time
        })

Frequently Asked Questions

Why is admission control needed for LLM serving when we have autoscaling?

Autoscaling has latency. GPU instances take 60-180 seconds to provision. Admission control bridges that gap by shedding load while new capacity comes online. Without admission control, you experience a sustained period of broken SLOs during scale-up events.

Why LLM inference needs admission control even for internal deployments?

Internal deployments still have contention. If you're running batch jobs alongside interactive requests on a shared internal cluster, batch jobs will starve interactive requests without admission control. Internal users are less patient than external ones when the system degrades.

What's the difference between rate limiting, load shedding, and admission control?

Rate limiting constrains client behavior. Load shedding rejects requests when the system is saturated. Admission control is broader: it evaluates requests based on current system state and projected resource availability, potentially rejecting or delaying requests before they reach the execution layer. It's preventive rather than reactive.

What's a good starting point for the max queue depth?

Start with 2x your expected max concurrent requests. If you expect 50 concurrent requests, cap the queue at 100. You'll tune this up or down based on your actual service distribution and client retry behavior.

Should the admission controller know about model architecture?

Yes, at least at a basic level. Models with large KV caches (long context, many heads) consume memory differently than small models. If you're serving multiple models, the admission controller needs to know the memory footprint per request to make good decisions.

How do you test admission control under production conditions?

Use Grafana k6 or Locust to generate synthetic traffic with realistic token distributions. Simulate traffic surges at 2x, 3x, and 5x normal load. Verify that admission control rejects requests at the edges rather than degrading all requests.

Does admission control affect time-to-first-token?

It adds 1-5ms for the decision itself. The bigger effect is indirect: by preventing queue overflow, you keep TTFT for accepted requests low, because they don't queue behind a backlog of requests that should have been rejected.

What happens when admission control makes a mistake?

Two failure modes. If it accepts too many, you see the same degradation you'd have without admission control. If it accepts too few, GPUs idle and you waste money. Both are recoverable, but acceptance bias is safer for SLAs in the short term.

The Bottom Line

Building production AI systems since 2018 has taught me one thing: every layer that assumes infinite capacity fails. This is true for databases, for message brokers, and it's true for LLM inference.

Admission control gives you a decision point. At that point, you can say "yes" to requests you can serve well and "no" to requests that would degrade the experience for everyone. The alternative is a system that doesn't fail on purpose. It fails by accident, usually at the worst possible moment.

Queue-based admission control for inference isn't glamorous. You won't write papers about it. But when your customer's production traffic peaks and your system holds steady while your competitor's falls over, you'll know exactly why LLM inference needs admission control.

The implementations I've shared here represent what we've validated. Start simple, measure relentlessly, and add sophistication only when you have evidence that the simple version isn't working.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.