llm inference admission control vs autoscaling

llm inference admission control vs autoscaling

Every Friday afternoon in mid-2026, I still see the same Slack message. Someone from platform engineering asks why their GPU bill doubled while p99 latency on the inference API went from 400ms to 2.3 seconds. And every time, the answer starts in the same place: they rerouted their whole capacity story through kubectl get hpa and hoped Kubernetes would figure out the rest.

I've built and debugged these systems since 2018. We've run vLLM fleets through 200K events/sec workloads, torn down more misconfigured HPA policies than I want to admit, and shipped revenue-critical inference for teams that couldn't afford a bad season. So when I say llm inference admission control vs autoscaling isn't a "pick one" question, I mean it. They do different jobs. Confusing them is expensive.

Here's the deal. Autoscaling answers "how much capacity should exist?" Admission control answers "which requests get to consume that capacity, right now, in what order?" You need both. The mistake is treating admission control like a fancy autoscaling trigger.

This piece is a buying guide. I'll compare both approaches, show where Kubernetes autoscaling breaks for LLMs, explain what admission control in Kubernetes GPU scheduling actually means, and give you the questions to ask before you pick a vendor or build it yourself.

What Kubernetes autoscaling actually does for GPUs (and doesn't)

Scope first. Kubernetes autoscaling has three layers. HPA scales replicas based on metrics. VPA adjusts per-pod resource requests. Cluster Autoscaler or Karpenter adds or removes nodes when pods can't schedule.

That's it. That's the whole thing. None of it understands that an LLM inference request has a variable cost based on prompt length, max tokens, cache hit rate, and KV pressure.

Autoscaling was designed for stateless web services. A request to /api/cart costs roughly the same CPU whether it's the first or thousandth. Inference doesn't work that way. A 30K-token context on a Llama-class model is a different universe of GPU time than a 200-token classification call.

Worse: HPA doesn't scale fast enough. Cold-starting a new GPU node in a managed cluster takes 90 seconds on a good day. On a bad day (spot preemption, region capacity crunch), it's minutes. Your p99 SLA is 500ms. You do the math.

I watched a fintech in March 2026 try to fix this with aggressive HPA thresholds. They set target utilization to 40%, scaled on tokens/sec instead of CPU. Bill went up 3.4x. Latency got better by maybe 12%. The autoscaler was fighting itself — flapping replicas up and down every 90 seconds while requests piled up.

What "admission control in Kubernetes GPU scheduling" really means

Here's where most teams get lost. They hear "admission control" and think of Kubernetes' built-in admission webhooks — the ones that validate or mutate pods before they're persisted.

Different thing. When I say admission control in Kubernetes GPU scheduling, I mean the runtime decision layer that inspects incoming inference requests and decides:

• Does this request get admitted now?
• Which replica/model/shard handles it?
• Does it queue, shed, or return a 429?
• What priority class does it belong to?
• How many tokens can it consume before preemption?

This layer lives in your inference server (vLLM, TGI, TensorRT-LLM, SGLang) or a gateway in front of it (NVIDIA Dynamo, Ray Serve, BentoML, KServe, or a custom Envoy filter). It's the bouncer, not the landlord.

The keys it uses:

1. Priority — is this a paying customer, internal batch job, or free-tier probe?
2. Deadline — does the caller care if this takes 8 seconds or 200ms?
3. Resource profile — how much KV cache and compute will this actually consume?
4. Current load — is the replica set at 60% or 94% utilization right now?
5. Fairness policy — has tenant A been starving tenant B for the last 30 seconds?

A good admission controller makes decisions in under 1ms. It has to. It's on the hot path of every single request.

Why queue-based GPU scheduling beats autoscaling for LLMs (most of the time)

This is my contrarian take and I'll defend it: queue based gpu scheduling vs kubernetes autoscaling isn't a fair fight for LLM workloads. Queues win. Autoscaling is a blunt instrument that responds in minutes; queues respond in milliseconds.

Here's why. LLM inference is a queueing problem with heterogeneous service times. Little's Law governs your concurrency. If arrival rate is λ and average service time is W, you need L = λ × W in-flight capacity. Autoscaling changes L in minute-scale steps. Queueing admission control reallocates L in real time.

Concrete example. We ran a customer support bot for a SaaS in Q1 2026. Traffic pattern: 40 rps baseline, spiking to 900 rps during their Tuesday 10am product announcements. Model: 8B parameters on L40S. Max context: 8K.

Autoscaling-only setup: pre-warmed 12 replicas, HPA scaled to 60 on spike, cold-start tail was 3-4 minutes. We dropped 2.1% of spike requests and p99 hit 4.8s. The 2.1% included paying enterprise accounts.

Same fleet with priority queueing admission control: we reserved 4 replicas for enterprise (guaranteed QoS), 8 shared for standard, and shed free-tier to a queue when load crossed 80%. No new capacity added during the spike. P99 for enterprise stayed under 700ms. Free-tier suffered. That's the deal you want to make.

Most teams try to solve overload with more GPUs. They should be solving it with smarter admission.

The four architectural options, compared honestly

You've really got four choices. I've shipped all four.

Option 1: HPA-only autoscaling. Simplest. Cheap to start. Breaks under variable load. Fine if your traffic is flat and your SLA is loose.

Option 2: Autoscaling + naive load balancing. Add a round-robin or least-conn LB. Better, but you're still overloading replicas because least-conn doesn't understand token cost. A replica with 3 long-context requests might look less loaded than one with 12 short ones.

Option 3: Queue-based scheduling with autoscaling as backpressure. Priority queues at the gateway, autoscaler reacts to queue depth rather than CPU. This is the sweet spot for 90% of teams.

Option 4: Full admission control with SLA-aware scheduling. Priority + deadlines + fairness + preemption + autoscaling. What you'd build at Stripe or OpenAI scale. Expensive. Necessary if you're serving multi-tenant revenue-critical inference.

The rest of this article helps you pick between them.

Feature-by-feature comparison

Latency response time

Autoscaling responds in 60-300 seconds (metric scrape + decision + node boot). Admission control responds in under 5ms. If your SLA is tighter than 2 minutes, admission control is not optional.

Cost predictability

HPA creates a feedback loop with cloud pricing and node availability. On AWS in 2026, p5.48xlarge spot capacity is not guaranteed. Your cluster autoscaler will wait. Meanwhile your requests queue. Admission control gives you bounded behavior — you shed load before you shed money.

Multi-tenancy and fairness

Autoscaling doesn't know what a tenant is. Admission control does. If you're selling inference as a product, this is the whole ballgame.

Cold-start handling

Autoscaling can't help you if the box isn't there. Admission control lets you pre-warm a small always-on pool and shape traffic into it. We now run "warm floor" setups where 20% of capacity is reserved for burst absorption and the autoscaler adds above that.

Observability

Both need good metrics. Admission control gives you richer signals — queue wait time, shed rate, per-tenant latency percentiles. HPA gives you replica count and utilization.

Operational complexity

Autoscaling: moderate. Admission control: high. You're building a scheduler. Get ready for state machines, policy versioning, and the deeply unpleasant task of explaining preemption to your customers.

Code: a minimal queue-based admission controller

Here's the shape of what we ship. FastAPI gateway, Redis-backed priority queue, per-tenant token bucket. This is simplified but real.

python
from fastapi import FastAPI, HTTPException, Request
from pydantic import BaseModel
import redis.asyncio as redis
import time
import uuid

app = FastAPI()
rdb = redis.from_url("redis://redis:6379")

PRIORITY_TIERS = {"enterprise": 1, "standard": 2, "free": 3}
MAX_QUEUE_DEPTH = {"enterprise": 200, "standard": 500, "free": 1000}
SHED_THRESHOLD = {"enterprise": 0.98, "standard": 0.85, "free": 0.60}

class InferenceRequest(BaseModel):
    tenant_id: str
    tier: str
    prompt_tokens: int
    max_tokens: int

@app.post("/v1/completions")
async def admit(req: InferenceRequest, request: Request):
    if req.tier not in PRIORITY_TIERS:
        raise HTTPException(400, "unknown tier")

    util = float(await rdb.get("gpu_utilization") or 0.0)
    if util > SHED_THRESHOLD[req.tier]:
        # Slow-path: enqueue instead of admitting
        depth = await rdb.llen(f"queue:{req.tier}")
        if depth >= MAX_QUEUE_DEPTH[req.tier]:
            raise HTTPException(429, "capacity exceeded")
        job_id = str(uuid.uuid4())
        await rdb.zadd(f"queue:{req.tier}", {job_id: PRIORITY_TIERS[req.tier]})
        return {"status": "queued", "job_id": job_id, "eta_ms": depth * 120}

    # Fast-path: admit directly
    return await dispatch(req)

async def dispatch(req: InferenceRequest):
    # Route to least-loaded replica that fits KV budget
    return {"status": "admitted", "ts": time.time()}

That SHED_THRESHOLD map is the whole product. Tune it against real traffic, not synthetic benchmarks.

Code: instrumenting autoscaling on queue depth, not CPU

If you're going to use HPA, don't scale on CPU. Scale on queue depth or shed rate. Here's a Prometheus adapter config we use.

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-inference
  minReplicas: 4
  maxReplicas: 60
  metrics:
    - type: Pods
      pods:
        metric:
          name: inference_queue_depth
        target:
          type: AverageValue
          averageValue: "40"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Percent
          value: 20
          periodSeconds: 120

The scaleDown stabilization of 600 seconds is deliberate. Nobody wins when replicas flap. And yes, the 4-replica warm floor is what absorbs the spike while the autoscaler is still deciding.

When autoscaling alone is actually fine

Let me not be dogmatic. Autoscaling-only is fine when:

• Traffic is smooth and predictable (internal tools, batch jobs, low-volume SaaS)
• SLA is measured in seconds, not milliseconds
• You have one tenant or a very small number
• Your GPU pool is large enough that you're never near saturation
• You genuinely can't afford the engineering time for admission control

A B2B analytics company we work with runs 3 replicas of a Mistral-class model for internal summarization. Traffic peaks at 20 rps. HPA with a queue-depth metric is more than enough. We didn't build them admission control. We saved them $40K in engineering.

Context matters. Don't over-engineer.

When you need admission control (and can't skip it)

You need it when any of these are true:

1. Multiple tenants share a GPU pool and paying customers can't be starved
2. Your p99 SLA is under 2 seconds
3. You have more than one model class or context size on the same hardware
4. You need preemption — batch jobs must yield to interactive traffic
5. Your traffic has spikes you can't absorb with pre-warming
6. You're being charged by the token and need per-tenant cost accounting

That's a checklist. If you check 2+ boxes, build or buy admission control. If you check 0-1, HPA is probably fine.

Providers and stacks: what's actually on the market in 2026

NVIDIA Dynamo (GA late 2025) is the most serious production-grade option. Its planner and router do admission-control-style scheduling. Expensive, opinionated, fast. If you're on NIM, this is the default answer.

Ray Serve has grown up. Its DeploymentHandle and request router do priority and queueing well. The autoscaler hooks (KubeRay) are decent. Good middle ground.

KServe with the LLM runtime (llm-d initiative) is the open-source path. Less polished than Dynamo, more flexible. If you have strong platform engineering, this is where I'd start.

vLLM's native --max-num-seqs and priority scheduling are useful primitives. Not a full admission controller. Pair it with a gateway.

BentoML / Modal / Together / Fireworks handle this for you if you're buying managed inference. Cost per token is higher, engineering cost is lower. Reasonable trade for a lot of teams.

We build custom gateways on Envoy + Redis for clients who need multi-tenant fairness that no vendor ships out of the box. It's not glamorous. It works.

The trap: buying a scheduler to fix a pricing problem

At first I thought the "admission control vs autoscaling" question was a branding problem — vendors wanted new SKUs. Turns out a lot of teams are actually trying to fix a pricing and quota problem with schedulers.

If free-tier users consume 70% of your GPU because you never set hard quotas, no admission controller saves you. You need to say no to free users before you need to say no at the scheduler. Fix the business rule first.

Similarly: if your autoscaler is thrashing because your metrics are garbage, an admission controller will just thrash in milliseconds instead of minutes. Same disease, faster symptoms.

FAQ

What's the difference between admission control and autoscaling for LLM inference?
Autoscaling changes how much GPU capacity exists. Admission control decides which requests consume existing capacity and in what order. You need both, but they solve different problems.

Is queue-based GPU scheduling always better than Kubernetes autoscaling?
No. If your traffic is smooth and your SLA is loose, HPA on queue depth is simpler and cheaper. Queues shine when traffic is spiky or multi-tenant.

What is admission control in Kubernetes GPU scheduling?
It's the request-time decision layer — inside the inference server or a gateway — that inspects each incoming request and decides admit, queue, shed, or route. It's distinct from Kubernetes' pod admission webhooks.

Can HPA scale on queue depth instead of CPU?
Yes, via a custom metrics adapter (Prometheus Adapter, KEDA). It's usually a big improvement for LLM workloads. CPU utilization is a poor proxy for GPU-bound inference.

How fast does admission control need to be?
Under 5ms per request, ideally under 1ms. It's on the hot path. If your admission layer adds 20ms, you've given back the latency you saved.

Do I need admission control if I only have one tenant?
Not usually. Priority queueing matters when resources are contested. Single-tenant teams get more from better autoscaling and pre-warming.

Which is cheaper — building admission control or buying a managed inference platform?
Depends on volume. Below ~$40K/month GPU spend, managed usually wins. Above that, the math tilts toward building, but only if you have platform engineering bandwidth.

How do I measure whether admission control is working?
Track shed rate, queue wait p99, per-tenant latency percentiles, and cost per successful request. If shed rate is climbing without revenue impact, you're doing it right.

What I'd tell a team starting today

Pick based on your SLA, not your ambition. If you can tolerate multi-second tail latency and single-tenant traffic, HPA on queue depth plus pre-warmed replicas will carry you far. That's a six-week project, not a six-month one.

If you're serving multiple customers with real money on the line and your p99 target is sub-second, budget for admission control. Start with a priority queue at the gateway. Add deadlines. Add fairness. Add preemption only if you actually have preemptable work.

The llm inference admission control vs autoscaling debate isn't really a debate — it's two layers of the same system. Autoscaling without admission control is a GPU bill with a surprise factor. Admission control without autoscaling is a static fleet pretending to be elastic. Do both, in that order of confidence.

And whatever you build, tune it against your real traffic. Benchmark numbers from a vendor's blog post won't tell you where your queue tips over. Your Tuesday 10am spike will.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.