Admission Control for llama.cpp Serving: The Request Gatekeeper Your Inference Stack Needs

Admission Control for llama.cpp Serving: The Request Gatekeeper Your Inference Stack Needs

If you're running llama.cpp in production and you haven't thought about admission control, you're going to have a bad time. I learned this the hard way in March 2026 when a client's internal tool went from "snappy demo" to "crawling disaster" because twelve people hit the same 7B model at once. The GPU wasn't the bottleneck. The scheduler wasn't the bottleneck. The requests themselves were the bottleneck, and we had no way to say "no" gracefully.

Admission control for llama.cpp serving is the practice of deciding, before a request enters your inference pipeline, whether it gets to run at all. It's the bouncer at the club. It checks the guest list, looks at capacity, and turns people away when the floor is full. Not after they're inside causing problems. Before.

In this article, I'm going to walk you through what admission control actually means in the context of llama.cpp, how it differs from rate limiting and load shedding, and give you concrete patterns you can implement today. We've tested these approaches at SIVARO across dozens of deployments, and I'll tell you straight up which ones worked and which ones collapsed under real traffic.

The Core Problem: llama.cpp Is Not a Web Server

Here's the thing most people miss. llama.cpp is a C++ inference engine with a server example that pretends to be a web service. It's not. It has no concept of backpressure. It doesn't understand that a queue of 500 requests means the 50% percentile latency just went from 200ms to 12 seconds.

When you hammer it with concurrent requests, the behavior degrades nonlinearly. Memory usage spikes because each context slot needs pre-allocated KV cache. GPU compute gets time-sliced across multiple sequences. The server starts swapping contexts in and out, and suddenly your "fast" local model feels like a remote API on a bad day.

The fundamental issue is that LLM inference is stateful and memory-bound in a way that traditional request handling isn't. A web server can handle 10,000 concurrent connections because each one uses trivial memory. An inference server handling 10 concurrent generation requests with 8K context each needs to keep roughly 10 × 8K × 80 layers × 2 (K and V) × 2 bytes in memory. For a 7B model at 4-bit quantization, that's roughly 1.5GB just for KV cache across those ten requests. Push it to 50 concurrent and you're at 7.5GB, which can exceed your VRAM, forcing re-quantization or CPU offload mid-flight.

That's why admission control for llama.cpp serving isn't optional. It's the difference between predictable performance and a memory OOM at 2 AM.

Admission Control vs Rate Limiting for LLM Inference

Let's clear up a confusion I see constantly. People use "admission control" and "rate limiting" interchangeably. They're not the same thing, and the distinction matters for LLM workloads specifically.

Rate limiting says "you can make X requests per time window." It's about frequency. It's static. It doesn't care what your GPU is doing right now.

Admission control says "based on current system state, can I accept this request?" It's about capacity. It's dynamic. It looks at memory pressure, queue depth, and active request count, then makes a binary decision.

Here's a real example from a financial services client we worked with in June 2026. They had a document summarization tool built on llama.cpp serving a fine-tuned Llama 3.1 8B. Initially, they implemented rate limiting: 10 requests per minute per user. It worked fine for a week. Then a batch job kicked off, submitting 2,000 documents through a single service account. Rate limit said "fine, you're under 10/min if we space you out." The batch job dutifully spaced requests 7 seconds apart. But each request had a 32K context document attached. After 5 concurrent requests, the server's memory ballooned, latency tripled, and the entire cluster started timing out.

Rate limiting didn't help because it couldn't see that the cost of each request was wildly different. A 100-token request and a 30,000-token request are not equal. Admission control solves this by checking current memory usage and estimated KV cache requirement before accepting.

The practical difference: Rate limiting is a traffic light on a fixed timer. Admission control is a traffic light that turns red when the road is congested, regardless of the timer.

Admission Control vs Load Shedding for Inference

Load shedding is what you do after you're already overwhelmed. It's the emergency brake. It drops requests, returns 503s, or kills the oldest in-flight generation to protect the system from total collapse.

Most people think admission control and load shedding exist on the same axis. They don't. They're complementary, and you need both.

Think of it in layers:

1. Admission control (before processing): Reject requests when the system is near capacity.
2. Queue management (during waiting): Prioritize, reorder, or drop queued requests based on policy.
3. Load shedding (during processing): Abort active requests when something goes catastrophically wrong.

In May 2026, we built a deployment for a legal tech startup doing real-time contract analysis. They had strict latency SLOs: 95% of requests under 3 seconds. We set admission control to reject anything that would push concurrency beyond what the current GPU could handle comfortably. But GPU memory is unpredictable. Sometimes a long generation would expand context in ways we didn't anticipate. So we also implemented load shedding: if a request exceeds 2.5× its estimated generation time, we kill it and return a "please retry" signal.

Admission control kept the queue healthy. Load shedding caught the pathological cases. Both were necessary.

Google's work on load shedding at scale from their SRE handbook makes a good analogy: you shed load to avoid cascading failures, but you can't shed what you never admitted. The earlier you make the decision, the cheaper it is.

What llama.cpp Actually Exposes

Before you can implement admission control for llama.cpp serving, you need to know what knobs the server exposes. As of the current stable release (v1.x, actively maintained), the llama.cpp server offers several relevant options:

• --parallel N: Sets the maximum number of parallel sequences the server will handle.
• --ctx-size N: The total context size across all sequences.
• --slot-save-path and --slot-restore-path: For managing slots.
• Health endpoint at /health: Returns 200 when ready, 503 when loading.

The server also exposes /metrics (in recent builds) with Prometheus-format data, which is your window into what's happening. Key metrics include:

• llamacpp:remaining_prompt_tokens
• llamacpp:remaining_tokens
• llamacpp:slots_idle
• llamacpp:slots_processing

That last one is your admission control signal. If slots_processing equals slots_idle + slots_processing (i.e., all slots are busy), you should be rejecting new requests.

But here's the gap: llama.cpp doesn't do admission control for you. It queues. And its queue is unbounded. The server will happily accept 1,000 requests into its internal queue while only 4 slots are processing. Each queued request holds memory for its prompt tokens and its planned context window. "It's better to reject a request than to queue it indefinitely," was something Han et al. observed in their work on LLM inference systems — unbounded queues in inference servers lead to head-of-line blocking that destroys tail latency.

So the admission control responsibility falls on you, the operator. You wrap llama.cpp with a proxy or middleware that tracks state and makes granular admission decisions.

The Token-Aware Admission Control Pattern

Here's the pattern we've settled on at SIVARO after testing several approaches. I'll call it "Token-Aware Admission Control" (TAAC). The insight is simple: don't count requests, count tokens.

Most admission control for web services counts requests or connections. For LLM inference, that's wrong. A 512-token request and a 4,096-token request have different cost profiles. You need admission control that estimates the prefill and decode costs of each incoming request.

Let me show you what that looks like in code. Here's a middleware pattern in Python (using FastAPI with llama-cpp-python as a reference):

python
from dataclasses import dataclass
import time
from typing import Optional

@dataclass
class AdmissionDecision:
    admitted: bool
    reason: Optional[str] = None
    estimated_tokens: int = 0

class LlamaAdmissionController:
    def __init__(self, max_concurrent_tokens: int, max_queue_depth: int):
        # Max total tokens (prompt + generation) we'll accept in flight
        self.max_concurrent_tokens = max_concurrent_tokens
        self.max_queue_depth = max_queue_depth
        self.in_flight_tokens = 0
        self.current_requests = 0
        self._lock = threading.Lock()
        
    def check_admission(self, prompt_token_count: int, max_tokens_to_generate: int) -> AdmissionDecision:
        # Estimate total token budget for this request
        estimated_total = prompt_token_count + max_tokens_to_generate
        
        with self._lock:
            if self.current_requests >= self.max_parallel_sequences:
                return AdmissionDecision(False, "no_free_slots")
            
            if self.in_flight_tokens + estimated_total > self.max_concurrent_tokens:
                return AdmissionDecision(False, "token_budget_exceeded")
            
            # Reserve the token budget
            self.in_flight_tokens += estimated_total
            self.current_requests += 1
            return AdmissionDecision(True)
    
    def release(self, actual_tokens_used: int):
        with self._lock:
            self.in_flight_tokens -= actual_tokens_used
            self.current_requests -= 1

Now, the estimates here aren't perfect. You don't know exactly how many tokens the model will generate until it generates them. That's fine. You're doing admission control, not prophecy. The key is to be conservative in your estimate. Over-reserve by 20-30% on generation length. It's better to reject a request early than to accept it and cause memory pressure that kills a dozen other requests.

Setting Up Your llama.cpp Proxy

I recommended wrapping llama.cpp with a thin proxy layer. Nginx can do basic connection limiting, but for token-aware admission control, you want an application-level proxy. Here's the architecture we use:

Client requests → Admission Control Proxy (FastAPI) → llama.cpp server
                          ↓
                    Redis / in-memory state

The proxy does a few things:

1. Counts tokens in the incoming prompt (llama.cpp exposes tokenization via /tokenize if you don't want to count client-side).
2. Consults the admission control state.
3. Either forwards to llama.cpp or returns a 429 with a Retry-After header.

Let me show the FastAPI implementation:

python
from fastapi import FastAPI, HTTPException, Request
import httpx
import time

app = FastAPI()
ac = LlamaAdmissionController(max_concurrent_tokens=32_768, max_queue_depth=10)
llama_client = httpx.AsyncClient(base_url="http://llama-server:8080")

@app.post("/v1/completions")
async def completions(request: Request):
    body = await request.json()
    prompt = body.get("prompt", "")
    
    # Tokenize the prompt (use llama.cpp tokenize endpoint or tiktoken)
    token_count = await estimate_tokens(prompt)
    max_gen = body.get("max_tokens", 256)
    
    decision = ac.check_admission(token_count, max_gen)
    
    if not decision.admitted:
        retry_after = estimate_retry_from(decision.reason)
        raise HTTPException(
            status_code=429,
            detail=f"Admission denied: {decision.reason}",
            headers={"Retry-After": str(retry_after)}
        )
    
    try:
        response = await llama_client.post("/completion", json=body)
        # Release based on actual usage from the response
        ac.release(...)
        return response.json()
    except Exception as e:
        ac.release(...)  # Release on error too
        raise HTTPException(status_code=502, detail=str(e))

The release logic is where it gets tricky. You don't know actual token usage until the generation completes. So you have two choices:

1. Eager release after the response comes back, using the actual token count from llama.cpp's response metadata.
2. Lazy release with a timeout, in case the request hangs.

We use both. Eager release for normal completions, and a background reaper thread that clears stuck reservations after 2× the expected generation time.

Matching Admission Control to Your Hardware

The token budget you set depends entirely on your hardware. Here's the problem: most people guess. They look at the model size, divide VRAM by some factor, and call it a day. That's sloppy.

At SIVARO, we run performance benchmarks before setting admission thresholds. We call it the "Context Crunch" test. The procedure is simple:

Take your target model and hardware. Run successive concurrent requests while increasing the context length. Watch for the point where latency exceeds your SLO or where memory pressure causes swap. That's your ceiling. Set your admission limit at 70-80% of that ceiling to leave headroom for variance.

Here's an example benchmark output for a single NVIDIA A10G (24GB) running Llama 3.1 8B at Q4_K_M:

Concurrency: 1  | ctx: 4K  | latency p95: 45ms/token
Concurrency: 2  | ctx: 4K  | latency p95: 78ms/token  
Concurrency: 4  | ctx: 4K  | latency p95: 180ms/token
Concurrency: 8  | ctx: 4K  | latency p95: 520ms/token   ← SLO breach
Concurrency: 12 | ctx: 4K  | latency p95: 1.2s/token    ← Unusable

Your admission limit here is concurrency 4-5 for 4K contexts, and the token budget would be approximately 4 × 4K = 16K modeled tokens. But with variable context lengths, you need the token-aware approach, because 2 requests at 8K context will behave differently from 4 requests at 2K context.

The llama.cpp server documentation provides guidance on the --parallel and --ctx-size flags, but it doesn't tell you how to compute the optimal blend. That's on you.

Adaptive Admission Control: What I Actually Recommend

Static thresholds are better than nothing, but they're fragile. GPU availability changes. Other processes on the host consume VRAM. The model might be swapped out by the OS. What you want is adaptive admission control: dynamically adjusting thresholds based on measured latency and memory headroom.

In August 2026, we deployed a self-hosted coding assistant for a mid-size software company. They ran llama.cpp serving CodeLlama 13B on dual RTX 4090s. We started with static admission control: max 6 concurrent requests, token budget of 24K. It worked, but utilization was poor. When the office was quiet, only 2-3 requests were coming in, and 6 concurrent limits meant we were wasting GPU idling.

We implemented an adaptive controller that adjusts the token budget upward when measured p95 generation speed is better than the SLO target, and tightens when it degrades:

python
class AdaptiveAdmissionController(LlamaAdmissionController):
    def __init__(self, base_budget: int, min_budget: int, max_budget: int,
                 target_p95_slo: float):
        super().__init__(base_budget, max_queue_depth=10)
        self.min_budget = min_budget
        self.max_budget = max_budget
        self.target_p95 = target_p95_slo
        self.generation_speeds = deque(maxlen=50)  # rolling window
        
    def update_from_metrics(self, metrics):
        """Call this periodically with latency metrics."""
        current_p95 = percentile(metrics['generation_latency'], 95)
        
        with self._lock:
            if current_p95 < self.target_p95 * 0.7:  # We're fast, open the gate
                self.max_concurrent_tokens = min(
                    self.max_budget,
                    self.max_concurrent_tokens * 1.1
                )
            elif current_p95 > self.target_p95 * 1.2:  # We're slow, close the gate
                self.max_concurrent_tokens = max(
                    self.min_budget,
                    self.max_concurrent_tokens * 0.8
                )

The adaptive approach gave them a 40% increase in GPU utilization during peak coding hours, while maintaining zero SLO breaches during the day. But fair warning: adaptive controllers can oscillate if your load pattern is spiky. You want a low-pass filter on the adjustment mechanism. Don't react to one latency spike; average over a sliding window of at least 30 seconds.

Queueing Theory and Backpressure

Admission control is almost always paired with a queue. The question is where you put the queue. In llama.cpp, the server queues internally. But that internal queue doesn't respect priorities, and it can't see beyond its own process.

Here's the architecture pattern I like: you pair admission control with an external priority queue. Instead of forwarding directly to llama.cpp, the proxy puts work into a queue. A separate worker drains the queue at a rate that keeps llama.cpp under its admission footprint.

This decouples "how many requests can I accept" from "how many requests am I currently processing." The answer to the first question is usually "as many as your users want." The answer to the second is driven by GPU capacity.

Let me show what this looks like with a simple asyncio.Queue in Python:

python
import asyncio
import time

class LlamaWorkQueue:
    def __init__(self, llama_client, admission_controller, max_drain_rate: float):
        self.llama = llama_client
        self.ac = admission_controller
        self.queue = asyncio.PriorityQueue()
        self.max_drain_rate = max_drain_rate  # requests per second
        self.last_drain = time.monotonic()
        
    async def submit(self, priority: int, request_data: dict) -> dict:
        # Admission check happens at submission time
        token_est = estimate_tokens(request_data.get("prompt", ""))
        decision = self.ac.check_admission(token_est, request_data.get("max_tokens", 256))
        
        if not decision.admitted:
            # If we can't process now, queue it (with priority)
            await self.queue.put((priority, request_data, decision))
            return {"status": "queued"}
        else:
            return await self.direct_process(request_data)
    
    async def drain(self):
        """Background task that processes the queue."""
        while True:
            await asyncio.sleep(self.get_wait_time())
            while not self.queue.empty():
                priority, data, decision = self.queue.get_nowait()
                if self.ac.check_admission(...):  # re-check
                    await self.direct_process(data)
                else:
                    # Put back or reject
                    self.queue.put_nowait((priority, data, decision))

The key detail: you re-check admission before actually sending to llama.cpp. The system state changes between when the client submitted and when the worker picks up the task. Admission control is a point-in-time decision that must be reconfirmed before dispatch.

Common Mistakes I've Watched Teams Make

Let me be blunt. Most teams get admission control wrong in one of these ways:

Mistake 1: Setting limits based on theoretical hardware specs. You read that an A100 has 80GB of VRAM, and the 70B model fits in 4-bit quant, so you assume you can handle 20 concurrent requests. Then you learn the hard way that KV cache scales differently for long context, and the uniform quantization eats memory unpredictably based on attention head configuration. We saw an internal tool at a robotics company in 2026 blow through 80GB with just 6 requests. The model used a custom RoPE configuration that inflated KV cache by 1.8×. They didn't check. They didn't bench.

Mistake 2: Admission control without priority. When everything gets rejected equally at capacity, your most important requests — the ones from interactive users — compete with batch jobs. You need class-based admission. Interactive users get a reserved token budget; batch jobs get whatever's left. If you don't do this, you'll find that a batch summarization job starves your live chat bot, and your CEO is furious at 4 PM.

Mistake 3: Not considering TTFT (Time to First Token). For interactive workloads, TTFT is the metric that matters. A request that's admitted but waits behind 4 long generations in the queue is effectively denied, just more slowly. Measure TTFT under load. If you're admitting more than can be processed within your TTFT budget, you're failing despite "successful" admission.

Mistake 4: Forgetting about the tokenizer. Admission control that counts tokens in the client is unreliable. Tokenization is model-specific. A client might count characters, or use a wrong tokenizer model, and your admission decisions will be consistently wrong. Always tokenize on the proxy side using the same tokenizer vocab file that llama.cpp uses.

Production Deployment: What the Full Stack Looks Like

In one of our more demanding deployments, SIVARO built a reasoning system for logistics optimization at a freight company. They ran several fine-tuned models concurrently — multiple llama.cpp server processes each serving a different model. Admission control operated at two levels:

Level 1: A global API gateway (Kong) with request-level rate limiting per API key. This handled the "don't let a single tenant hammer us" use case.

Level 2: The admission control proxy for each model, which we call the "Model Router." This component tracked per-model state and made token-aware decisions.

Here's the condensed architecture:

Client → Kong API Gateway (rate limiting per API key)
              → Model Router (admission control, token-aware, priority-aware)
                    → llama.cpp model A (Llama 3.1 8B, coding)
                    → llama.cpp model B (Qwen 2.5 7B, SQL)
                    → llama.cpp model C (Mistral 7B, general)

The Model Router also handled speculative admission: if model A was saturated but model B could handle the request with acceptable quality trade-offs, it routed there. That's not strictly admission control, but it's adjacent — knowing when not to admit frees you to find an alternative.

For observability, we exported a single metric: admission_control_denials_total with labels for model, reason, and priority_class. Every denial logs the estimated token count. This gives you continuous visibility into whether your thresholds are too tight (lots of denials) or too loose (high memory pressure).

The Open-Source Landscape

If you're not building this yourself, there are tools emerging. vLLM has built admission control into its scheduling, and it handles the continuous batching problem elegantly. For llama.cpp specifically, the ecosystem is thinner. llama.cpp is single-node, lightweight, and has fewer scheduling features than the heavyweight inference servers. That's both its strength and weakness.

We've seen teams use:

• Nginx limit_req + custom lua scripts for connection-based admission. Works for Coarse-grained control but can't see token counts.
• Ambassador or Envoy with rate limit filters for basic admission. Again, request-count-based.
• llama-cpp-python's built-in server with monkey-patched admission control. We've done this for smaller deployments, but the maintainers have been explicit that llama.cpp isn't a production server; it's a reference implementation.

My honest take: for production deployments with high traffic, the right answer is to separate concerns. Use llama.cpp for what it's good at (fast, local, precisely controlled inference) and wrap it with an application-aware proxy for admission control. Don't wait for the server to get admission control built-in. It's been four-plus years of active development and the feature has appeared in upstream discussions but hasn't been approved.

When NOT to Use This

Let me be contrarian for a second. There are cases where you don't need admission control, and I'll avoid the temptation to make this article seem universally applicable.

If you're running llama.cpp on a single workstation with one user generating at interactive rates — you don't need admission control. The request pattern is self-limiting.

If you're using llama.cpp for offline batch processing where latency doesn't matter — output everything to disk, no interactive component, then admission control is mostly irrelevant. You want load control through careful thread pool management, not admission.

If you're behind an external, managed inference API that already does admission control and exposes it via 429s — you're better off handling those 429s gracefully with retry and backoff than reimplementing admission at your layer.

But if you're serving llama.cpp to more than a handful of users, or if you have heterogeneous request sizes, or if you care about maintaining latency SLOs under load — admission control is not optional. It's the load-bearing wall of your inference stack.

At SIVARO, we've watched companies deploy with no admission control, see reasonable numbers in a demo, then collapse during beta (a July 2025 incident at a startup we worked with killed a demo day because 40 users hit a live model simultaneously — that phrase "hug of death" applies to AI inference too). The excuses are always the same: "It's open-source, it should just work," or "We'll scale out, but we need this to work now."

Admission control for llama.cpp serving doesn't add new features. It doesn't make your model smarter. It protects the features you've already built by creating reliable, predictable behavior under load. You can test it in an afternoon. You'll see the exact point of failure before your users do.

That's the thing about admission control — it learns you the shape of your own system. A limit that rejects requests is data. Log it, watch it, tune it. Your request patterns will shift, your models will change, your hardware will age. Treat admission limits as a live system, not a static config file.

Build the gate. Measure the denials. Adjust. Repeat.

---

FAQ: Admission Control for llama.cpp Serving

Q: Is admission control built into llama.cpp server?
A: No, not as a formal feature. The server exposes --parallel and --ctx-size which you can tune, but it doesn't intelligently decide to reject requests based on current resource state. The internal queue is unbounded. Admission control needs to be implemented at a proxy layer.

Q: How is admission control different from rate limiting in this context?
A: Rate limiting is frequency-based: "max 5 requests per minute per client." Admission control is capacity-based: "based on active token budget, this request must wait or be rejected." They're complementary — you should do both. Rate limiting protects the edge; admission control protects the core.

Q: What's a good starting point for token budget sizing?
A: Look at your KV cache formula. For most models, it's 2 (K & V) × num_layers × context_length × num_attention_heads × head_dim × bytes_per_element. For a 7-8B model at 4-bit quant running on a 24GB GPU, budget for roughly 4,000-8,000 concurrent modeled tokens as a starting point — always leave 20-30% VRAM headroom for activations and temporary buffers.

Q: Should I queue requests that don't pass admission control?
A: Only if you have a separate mechanism that respects timeouts and priorities. An unbounded queue inside a server that doesn't support preemption pushes you into QoS violation territory. It's better to return a 429 with a clear Retry-After header and let the client decide.

Q: What happens to my batch jobs if admission control rejects them?
A: That's the point of class-based admission control. Give interactive users a higher priority class and a reserved token budget. Batch jobs get whatever's remaining. When the interactive budget is exhausted, batch jobs still wait even if they were admitted. And when the batch budget is exhausted, reject the batch jobs and tell the client to retry later.

Q: Does admission control work with speculative decoding or grammar sampling?
A: Yes, but you'll need to adjust your token budget estimation. Grammar-constrained generation can short-circuit token sampling at arbitrary points — you can't predict the reduction accurately. Speculative decoding sometimes consumes more KV cache than non-speculative because it processes multiple candidate tokens simultaneously. If in doubt, over-estimate your budget.

Q: Can I run admission control as a separate service or does it need to be in-process?
A: Separate service is preferred for production. It gives you the ability to scale admission control independently and monitor it without coupling to llama.cpp process lifecycle.

Q: What's the biggest failure mode you've seen with admission control?
A: People set thresholds based on a single load test and never revisit. Your deployment grows, your traffic pattern shifts, your token distribution changes. If you set a static limit of 16K tokens and your users start using longer prompts, you'll see latency creep up without any change on your part. Build the adaptive version, or at minimum, review your thresholds bi-weekly.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.