SIVARO
Circuit Breaker for LLM Inference Server
How to stop one bad tenant from burning your entire GPU fleet.
I watched a customer's retry storm take down a 32-node H100 cluster in under four minutes last November. Not a hardware failure. Not a model crash. A single misconfigured client that kept hammering a /v1/completions endpoint with 128K-context requests, each one timing out at 90 seconds, each timeout spawning three retries. The queue went from 40 to 11,000 in ninety seconds. Every other tenant on that cluster got served a 503.
That's the moment I stopped treating a circuit breaker for LLM inference server as optional plumbing. It's the difference between one tenant having a bad afternoon and every tenant having one.
Here's what I'll cover: what a circuit breaker actually does in an LLM context (it's not the same as the microservice version), how it differs from admission control, why you need both, and the exact patterns we run in production at SIVARO on vLLM and TensorRT-LLM deployments.
What a Circuit Breaker Actually Is (And Isn't)
A circuit breaker watches for repeated failures against a downstream dependency and, after a threshold, stops sending traffic to it. Classic pattern from Michael Nygard's Release It! — closed state passes traffic, open state fails fast, half-open state probes recovery.
In a microservice world, the boundary is clean. Service A calls Service B. B starts returning 500s. A opens the circuit.
LLM inference breaks that model in three ways most people miss.
First, there's no single "downstream." A request touches the tokenizer, the scheduler, the KV cache allocator, the attention kernel, and the sampling loop. Any of them can be the bottleneck, and the failure mode is rarely a clean 500. It's latency creeping from 800ms to 40 seconds while status codes stay 200.
Second, GPU inference is memory-bound in a way CPU services aren't. When your KV cache fills, you don't get a nice "resource exhausted" error. You get preemption, queue growth, and eventually OOM kills that take down the whole model replica. One tenant's 100 concurrent long-context requests can starve everyone else.
Third, retries are catastrophic. In a REST API, a retry costs you a few hundred microseconds. In LLM inference, a retry on a 4K-token generation costs 2-30 seconds of GPU time that someone else was queued for.
So a circuit breaker for LLM inference server needs to trip on latency and queue depth, not just error rates. And it needs to trip at the tenant boundary, not the replica boundary.
At first I thought this was just a rate-limiting problem. Turns out it's a fairness problem, and rate limits alone don't solve it.
Why Error-Rate Circuit Breakers Fail on GPU Inference
Most open-source circuit breaker libraries — resilience4j, Polly, the Envoy outlier detector — trip on error ratio. That works fine when errors are your signal. On GPU inference, your signal is almost never errors.
Let me show you what I mean. Here's a Prometheus query we run that shows the actual health signal:
promql
# The signal that matters: queue wait time P99
histogram_quantile(0.99,
sum(rate(vllm:request_queue_time_seconds_bucket[1m])) by (le, model_name)
)
When that number goes above 5 seconds, your GPU fleet is saturated. Requests are still succeeding. Error rate is still 0.1%. But your effective capacity has collapsed and every new request makes it worse.
Here's the failure curve nobody warns you about:
python
# Simplified model of LLM queue collapse
# throughput does NOT grow linearly with concurrency
def effective_throughput(concurrency, gpu_capacity=100):
# At low concurrency, throughput scales.
# Past a knee point, KV cache thrashing makes it DEGRADE.
knee = 60
if concurrency <= knee:
return concurrency * 1.0 # near-linear
# Past the knee, each extra concurrent request makes
# things worse because of KV cache eviction
excess = concurrency - knee
return knee - (excess ** 1.32) * 0.8
At concurrency 60, you get 60 units of throughput. At concurrency 200, you get roughly 12. That's not linear degradation. That's cardiac arrest. And it takes about 30 seconds to notice it in error rates.
A circuit breaker on error rate fires 30 seconds too late.
The right signal set:
- Queue wait time P99 (primary trip signal)
- KV cache utilization (leading indicator)
- Batch size trend (if it's dropping while queue grows, you're thrashing)
- Per-tenant request concurrency (the culprit)
I've seen teams try to sort this with autoscaling alone. Doesn't work. More on that below.
Circuit Breaker vs Admission Control — They're Not the Same
Most people conflate these. They solve different problems and you need both.
An admission controller decides whether to accept a request at all. It runs at the edge, before the request touches a GPU. It enforces budgets: per-tenant token rates, per-tenant concurrency caps, per-tier priority weights.
A circuit breaker decides whether to keep serving a client that's already inside the system and misbehaving. It runs closer to the model, watches for degradation patterns, and evicts noisy tenants.
Here's the concrete difference in a multi-tenant scenario:
| Failure | Admission control action | Circuit breaker action |
|---|---|---|
| Tenant exceeds token budget | Reject new requests with 429 | N/A — never got in |
| Tenant's requests pile up in queue | Throttle inbound rate | Trip after queue wait > 5s for 30s |
| GPU node degraded / slow | Route to healthy node | Trip node, drain connections |
| Tenant sends pathological prompt | Reject at edge | Trip tenant, quarantine for 60s |
| Model serving latency doubles | Autoscaler fires | Trip if P99 > threshold for sustained window |
For admission control for multi-tenant GPU inference, the design question is: what's the pool of shared resources, and how do you divide it without a global lock? For the circuit breaker, the question is: when the pool is stressed, who do you cut?
The mistake I see most often: teams build one or the other and think they've covered it. Admission control without a breaker means one tenant with a valid budget can still starve the queue if their requests are slow. A breaker without admission control means you're fighting fires constantly rather than preventing them.
We run both. Admission control at the gateway (Envoy + a custom Lua filter, or a Python gateway in front of vLLM). Circuit breaker as a sidecar that tails the scheduler's metrics.
Admission Control vs Autoscaling for LLM Serving — The Real Trade-off
This is where I have a strong opinion.
Autoscaling is the wrong first answer for LLM serving.
Yes, you need it. No, it does not save you from overload. Here's why: GPU pods take 60-180 seconds to become ready (model load, weight transfer, warmup). A traffic spike — a new customer onboarding, a batch job kicking off, a retry storm — reaches your cluster in 3 seconds. By the time your HPA (or KEDA, or Karpenter, or whatever you're on this week) notices and spins up replicas, the original fleet is already thrashing.
Autoscaling is a capacity answer to a time problem.
Admission control for multi-tenant GPU inference is a time answer to a time problem. It says: before you touch the pool, prove you're allowed to. Budgets are enforced in microseconds. No queue, no thrash, no OOM kills.
The pattern we settled on:
Request arrives
↓
[Rate limit per tenant] ← O(1), in-memory
↓
[Admission controller] ← concurrency + token budget check
↓
[Load balancer] ← picks replica with lowest queue depth
↓
[Model server]
↓
[Circuit breaker sidecar] ← watches queue wait, KV cache, TPS
↓
├─ healthy → serve
└─ tripped → evict tenant, open for 30s, half-open probe
Autoscaling runs behind all of this, reacting to sustained trends. It's the goalkeeper, not the defense.
I've written before that [admission control vs autoscaling llm serving] isn't really a versus — it's a layering question. But if you're forced to pick one because you're early, pick admission control. It's cheaper, it's faster, and it fails more gracefully.
Building a Circuit Breaker for LLM Inference Server — The Actual Code
Okay, enough theory. Here's the shape of what we run.
The state machine
python
from enum import Enum
from dataclasses import dataclass, field
import time
class BreakerState(Enum):
CLOSED = "closed" # normal traffic
OPEN = "open" # failing fast
HALF_OPEN = "half_open" # probing recovery
@dataclass
class TenantBreaker:
tenant_id: str
state: BreakerState = BreakerState.CLOSED
failure_count: int = 0
opened_at: float = 0.0
success_count: int = 0
# Trips on sustained queue wait, not errors
QUEUE_WAIT_THRESHOLD_S: float = 5.0
OPEN_DURATION_S: float = 30.0
HALF_OPEN_PROBE_LIMIT: int = 5
HALF_OPEN_SUCCESS_REQUIRED: int = 3
def record_signal(self, queue_wait_s: float) -> None:
if self.state == BreakerState.OPEN:
if time.time() - self.opened_at > self.OPEN_DURATION_S:
self.state = BreakerState.HALF_OPEN
self.success_count = 0
return
if queue_wait_s > self.QUEUE_WAIT_THRESHOLD_S:
self.failure_count += 1
# 3 sustained breaches in a 30s window → trip
if self.failure_count >= 3:
self._trip()
return
if self.state == BreakerState.HALF_OPEN:
if queue_wait_s <= self.QUEUE_WAIT_THRESHOLD_S:
self.success_count += 1
if self.success_count >= self.HALF_OPEN_SUCCESS_REQUIRED:
self._reset()
else:
self._trip()
def _trip(self) -> None:
self.state = BreakerState.OPEN
self.opened_at = time.time()
self.failure_count = 0
def _reset(self) -> None:
self.state = BreakerState.CLOSED
self.failure_count = 0
self.success_count = 0
Notice what's not in there: HTTP status codes. The breaker doesn't care whether the request returned 200. It cares whether the system handled it in reasonable time. A 200 after 40 seconds of queueing is a failure in LLM serving, no matter what the status line says.
The signal collector
You need per-tenant queue wait time. vLLM exposes this via its metrics endpoint but not always split by tenant. We inject a tenant label at the gateway and propagate it through request headers, then the model server tags its metrics.
python
from prometheus_client import Counter, Histogram
# In the model server, tagged at the gateway via request header
QUEUE_WAIT = Histogram(
'llm_tenant_queue_wait_seconds',
'Histogram of queue wait time per tenant',
['tenant_id', 'model_name'],
buckets=[0.1, 0.5, 1, 2, 5, 10, 30, 60]
)
def observe_request(tenant_id: str, model_name: str, wait_seconds: float):
QUEUE_WAIT.labels(tenant_id=tenant_id, model_name=model_name).observe(wait_seconds)
Then a sidecar process polls P99 per tenant every 10 seconds and feeds the breaker:
python
async def breaker_loop(breaker: TenantBreaker, prom: PromClient, tenant_id: str):
while True:
p99 = await prom.query(
'histogram_quantile(0.99, '
'sum(rate(llm_tenant_queue_wait_seconds_bucket'
f'{{tenant_id="{tenant_id}"}}[1m])) by (le))'
)
breaker.record_signal(p99)
await asyncio.sleep(10)
The gateway enforcement
When the breaker is OPEN, the gateway should return 503 (or 429 if you prefer retry semantics) without forwarding to the model server. That's the whole point — you're saving GPU cycles.
python
# Envoy Lua filter or your ingress of choice
def handle_request(tenant_id):
breaker = breakers.get(tenant_id)
if breaker.state == BreakerState.OPEN:
return {
"status": 429,
"headers": {
"Retry-After": "30",
"X-Circuit-Breaker": "open"
},
"body": {"error": "tenant throttled — retry after 30s"}
}
return forward_to_pool(tenant_id)
One gotcha: the client needs to actually honor the Retry-After. Most SDK clients do, but homegrown retry loops often don't. If your tenant's client ignores backoff, the breaker stays open longer. We've had to add firewall-level per-tenant request caps for two customers who refused to fix their retry logic. Not pretty, but it worked.
The Signals That Actually Matter
After running this on production clusters for the better part of a year, here's what I'd trip on and what I wouldn't:
Trip on:
- Queue wait P99 > 5s sustained over 30s (tenant-specific)
- KV cache utilization > 92% for 60s
- Time-to-first-token P95 > 3s (interactive apps only)
- Per-tenant concurrent requests > 4x their normal baseline
Don't trip on:
- Error rate (too lagging for LLM workloads)
- Raw request rate (spiky traffic is normal; queue depth is the real signal)
- Total tokens generated (useful for billing, useless for breakers)
- GPU utilization (often 100% during healthy batching — no signal)
That last one trips people up constantly. GPU utilization at 100% is good in LLM serving — it means your batching is working. A circuit breaker keyed on GPU utilization will fire on healthy load. Don't do that.
Where the Breaker Lives — Sidecar, Gateway, or Scheduler?
Three options. I've built two of them. Here's the verdict.
In the scheduler (vLLM/TGI code). Most accurate signals, but you're forking the model server. vLLM's scheduler doesn't have a tenant concept by default. You'd need to add one through the request metadata and expose it in the metrics. Doable, especially since vLLM 0.6+ made engine stats more pluggable. We do this for our own models.
In a sidecar. Polls metrics, feeds the breaker, signals the gateway via Redis or a small control plane. Works with any server. Adds 5-15s latency to detection. We use this for customers on managed endpoints where we can't touch the scheduler.
In the gateway. Only works if the gateway can see scheduler signals, which it usually can't. Envoy can't see per-tenant queue depth. Skip this unless you're doing full custom.
My recommendation: if you own the model server, put the breaker in the scheduler. If you don't, sidecar. Gateway-only is a half-measure.
The Retry Amplification Problem
I mentioned this at the top and it deserves its own section because it's the thing that kills clusters.
When a request times out at the client, the client retries. At scale, this creates multiplicative load. OpenAI's engineering team wrote about this in 2024 after their own outage — a retry storm from SDK clients compounded a transient failure into a multi-hour incident. Two years later, most teams still haven't internalized it.
The math: if 1% of your requests time out and each timeout spawns 2 retries, you've added 2% load. Fine. But at 10% timeouts and 3x retries, you've added 30% load to a system that was already at 90% capacity. That's the collapse.
A circuit breaker for LLM inference server should be paired with retry budgets at the gateway. Enforce that a tenant can only consume X% of their request budget on retries.
yaml
# Envoy retry policy — cap retries per tenant, not per request
retry_policy:
retry_on: "5xx,reset,connect-failure"
num_retries: 2
per_try_timeout: 30s
retry_budget:
budget_percent: 20
min_retry_concurrency: 3
20% budget means if a tenant burns 20% of their inflight budget on retries, Envoy stops retrying for that tenant until the window clears. This alone has saved us from three would-be incidents.
FAQ
What's the difference between a circuit breaker and rate limiting?
Rate limiting caps request count over time. A circuit breaker reacts to system health. You can be under your rate limit and still trip a breaker because your requests are slow or expensive. You need both — rate limits are static budgets, breakers are dynamic guardrails.
Should I trip on tenant identity or on endpoint?
Both, in different breakers. Tenant-level breakers protect against noisy neighbors (the multi-tenant fairness problem). Endpoint-level breakers protect against a specific model or route being broken. We run two breaker instances with different thresholds.
How long should a circuit stay open?
Start with 30 seconds. Exponential backoff after the third consecutive open is reasonable. But don't let it stay open for more than 5 minutes without human review — at that point you have a real incident and the breaker is masking it.
Does this work with vLLM and TensorRT-LLM?
Yes. Both expose per-request metrics you can tag with a tenant label. vLLM's Prometheus integration is more mature. TensorRT-LLM needs a small adapter if you're using the Triton backend. We support both in production.
Won't a circuit breaker hurt my latency-sensitive tenants?
Only if you trip on the wrong signals. Interactive tenants have lower queue-wait tolerance (2s P99) and batch tenants have higher (30s P99). Set per-tier thresholds, not a global one.
Can I use Envoy's built-in circuit breaker?
Yes, but it only sees HTTP-level signals. You need custom out-of-band signals (queue wait, KV cache) feeding back into Envoy via its outlier detection API. This is doable but fiddly. We ended up writing a small control plane that translates Prometheus alerts into Envoy cluster updates.
What's the failure mode when the breaker itself fails?
Fail-open by default for low-priority tenants, fail-closed for abuse-prone ones. Two of our customers asked for fail-closed on all tenants — they'd rather drop traffic than have a runaway bill. Both defaults are valid; pick per tenant.
How does this interact with autoscaling?
Autoscaling responds to sustained trends over minutes. The breaker responds to acute distress over seconds. They're complementary. If your breaker is firing constantly, you have a capacity problem, not a breaker problem — go tune your autoscaler.
What I'd Do Differently
Two things.
First, I would've built the per-tenant metrics pipeline before the breaker. We had to rebuild our signal collection twice because tenant labels weren't consistent between the gateway and the model server. Get your labels right, propagate them end-to-end, then build breakers. Doing it in the other order cost us a month.
Second, I would've started with a simpler signal set. We tried to trip on seven different conditions initially. Four of them produced false positives. Now we trip on two: tenant-scoped queue wait P99, and KV cache utilization. That's it. Everything else is a dashboard, not a trigger.
The circuit breaker for LLM inference server isn't a clever pattern — it's a boring one that most teams skip until they've been burned. Don't be most teams. Start with admission control at your gateway, add per-tenant queue metrics, then wire in a breaker that trips on latency rather than errors. Your cluster will thank you at 3 AM.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.