GPU Admission Control Open Source

GPU Admission Control Open Source

You're running a production inference service and the p99 latency just exploded. Again.

The typical story: you've got 40 pods scheduled onto a single A100, and while the GPU has 80GB of memory, it only has so many compute slices. The Kubernetes scheduler sees memory fits. It doesn't see the compute contention. Your Triton server is now juggling twelve concurrent requests when the hardware really can handle four without melting.

I hit this wall building real-time CV pipelines at SIVARO in 2024. The fix wasn't better scheduling or fancier autoscaling. It was admission control at the server level. And the open source ecosystem has quietly built some serious tooling for exactly this problem.

Here's what GPU admission control open source actually means, why it's the difference between a stable inference fleet and a fire drill, and the practical algorithms that work in production.

What GPU Admission Control Actually Is

Admission control is the gatekeeper between the request queue and the GPU itself. It decides "yes, this request can go into the GPU now" or "no, you wait in line." This isn't queueing theory at the edge. It's a decision engine that runs right before work gets dispatched to the device.

The key insight that took me too long to internalize: GPU memory is not the bottleneck for inference. Compute is.

CUDA kernels occupy the device. Memory copies occupy the copy engines. But when you're running small-batch inference (like real-time object detection), the launch overhead and kernel execution time dominate. You can pack 80GB of model weights into a card, but if the kernels need the streaming multiprocessors (SMs), you're competing for silicon, not memory.

GPU admission control algorithms for inference servers solve this by modeling the actual resource consumption of requests and rejecting or queueing work that would push the device past its sustainable throughput.

# Simple admission pseudocode
def admit_request(request_size_sm, available_sms, current_load):
    if current_load + request_size_sm > available_sms * 0.85:
        return REJECT_OR_QUEUE
    return ADMIT

Sounds simple. It's not.

Why This Is a Real Problem in 2026

The hardware landscape shifted hard. Hopper and Blackwell parts are everywhere now. But the pattern I see across startups and enterprises at SIVARO: people bought serious GPUs, then treat them like free-for-all compute pools because the K8s scheduler can't see kernel-level utilization.

Kubernetes device manager allocates whole GPUs. It's binary. You either get the card or you don't. For training, that's fine. For inference with multi-tenancy, it creates a tragedy of the commons where ten services share one card and everyone's latency dies.

Some folks say "just deploy multiple replicas." Sure, go pay for eight A100s when you really need two. That purchasing decision kills more AI startups than bad models.

The real answer used to be vendor-locked: NVIDIA MPS, MIG, or Triton's internal scheduler. Now there's good open source for heterogeneous GPU admission control that doesn't care if the GPU is NVIDIA, AMD, or something from a cloud provider's ASIC program.

The Admission Control Algorithm for Inference Servers

We tested five different approaches at SIVARO when building a multi-tenant inference gateway in 2025. Here's the ranking for real-time inference workloads:

• Static concurrency limits: works if you're serving identical requests. Falls apart with heterogeneous models.
• Memory-based admission: fails because it ignores SM contention.
• Throughput-based tokens: decent baseline, suffers when batch sizes change dynamically.
• Dynamic concurrency with feedback: works well. This is the money.
• ML-predicted latency: overkill for most. Adds more inference latency than it saves.

The winning pattern has three components:

• A token bucket that refills based on measured GPU idle time, not wall clock time.
• A classification phase that buckets requests by estimated SM cost.
• A rejection policy that favors queuing for real-time workloads over outright rejection, but with a bounded queue depth.

Here's the algorithm we open-sourced internally and I'm going to break down:

class GPUAdmissionController:
    def __init__(self, max_concurrency, target_sm_util):
        self.max_concurrency = max_concurrency
        self.target_sm_util = target_sm_util
        self.semaphore = threading.BoundedSemaphore(max_concurrency)
        self.sm_tracker = SMTracker()
        self.queue = deque(maxlen=32)

    def admit(self, request):
        # Estimate SM requirement based on model profile
        sm_cost = self.sm_tracker.estimate(request.model_id,
                                          request.batch_size)

        if self.semaphore.acquire(blocking=False):
            if self.sm_tracker.current_load() + sm_cost < self.target_sm_util:
                self.sm_tracker.book(sm_cost)
                return ADMIT
            else:
                self.semaphore.release()
                return self._queue_or_reject(request)

    def complete(self, request_handle):
        self.sm_tracker.release(request_handle.sm_cost)
        self.semaphore.release()
        self._drain_queue()

The _queue_or_reject path is critical. You need a policy distinction between interactive requests (must admit or reject fast) and batch jobs (can queue indefinitely). Mixing CRM and Slack notifications. It costs you user trust.

Open Source Tooling Worth Adopting

The open source landscape changed in the last eighteen months. What used to be "roll your own CUDA monitoring with DCGM" now has proper frameworks.

KServe with ModelMesh

ModelMesh has built-in admission control concepts. It's not perfect. The memory-capacity-based in-placement logic is coarse. But it has a concept of "memory pool" that you can abuse as a crude limiter.

What we found: ModelMesh's admission was too crude for heterogeneous SKUs. It assumed models dominate memory and forgot about dynamic batch size. We used it as a foundation and added our own token bank for the compute dimension. Or grab it here: KServe.

NVIDIA's Triton Inference Server with Enforcing

Triton's dynamic batching is great. But its admission control is basic — you can set max_queue_delay or a static max_batch_size. In 2025, NVIDIA shipped improvements to their model_control and sequence batcher, but we found their concurrency estimator too conservative for real-time.

Using a GPU admission control open source with Triton means putting a proxy in front, not relying on Triton's internals.

Ray Serve's Autoscaling Admission

Ray Serve has something approaching admission control through its autoscaling policy. Setting max_queued_requests per replica is a practical admission cap. The anti-aliasing window handles burst.

Sadly, Ray's policy is purely count-based. It doesn't look at GPU saturation. You'll deploy two replicas and one gets the requests that require high SM counts. Wait for the Ray 3.x improvements.

The Gateway Pattern: Envoy with Custom Filters

I know Envoy is about HTTP routing, not GPU scheduling. But if you're building a single admission point for heterogeneous models, running your GPU admission controller as an Envoy filter is the fastest path to production.

Our setup at SIVARO runs an Envoy sidecar that intercepts gRPC inference calls. A custom Python filter consults the admission algorithm before forwarding upstream.

python
# Envoy watcher pattern (simplified)
def main():
    admission = GPUAdmissionController(max_concurrency=16,
                                       target_sm_util=0.85)

    app = FastAPI()
    
    @app.post("/v1/infer")
    async def infer(request: InferenceRequest):
        decision = admission.admit(request)
        
        if decision.status == "ADMIT":
            async with IOManager(request.model_id):
                result = await upstream_client.infer(request)
            admission.complete(decision.handle)
            return result
            
        elif decision.status == "QUEUED":
            # Wait on retry-able channel
            timeout = min(50, decision.estimated_wait_ms)
            await asyncio.sleep(timeout / 1000)
            return HTTPException(503, "Retry later")
            
        else:
            raise HTTPException(429, "GPU saturated")

Target Utilization: The Hidden Knob

Most who "implement admission control" fail because they set targets wrong. Here's what I've measured across workloads:

• Transformer-based inference (LLM, BERT): target 60-65pct SM utilization for p99 < 50ms.
• CNN image models: target 80-85pct works fine. Kernels are shorter and better interleaved.
• Mixed multimodal: anything above 70pct triggers convoy effects.

Why? It's statistics. Kernel arrival has Poisson-like burstiness at the admission layer. If you consistently run at 92pct SM utilization, the queue lengths grow super-linearly. It's not linear queuing theory; it's multi-server with correlated arrivals.

At SIVARO, we set the target default to 80 percent for image models and 60 percent for diffusion models, and we saw p99 improvements of 4-6x.

How to Set Up an Open Source GPU Admission Control Actually Working

Let's walk through a Kubernetes-native deployment from scratch. This is the pattern from SIVARO's github that several startups have copied.

The Control Plane.

First, you need a scheduler that understands GPU compute. The stock Kubernetes scheduler won't do. As of 2026, the standard approach is to use a custom scheduler extender that talks to a Metrics API exposing DCGM metrics.

Here's the trick — don't use the K8s extended resources for fractional GPU. That's a dead end for compute. Instead, create a custom resource called sivaro.io/sm-utilization and use the scheduler's resource binding to nothing.

apiVersion: sivaro.io/v1
kind: GPUReservation
metadata:
  name: inference-profile-llama-7b
spec:
  gpuClass: A100
  maxSMUtilization: 60
  maxMemoryMB: 14000
  coLocatedWith: ["frontend", "text-embedding"]

The Telemetry Pipeline.

The admission controller is only as good as your utilization estimate. Don't use nvidia-smi polling; that has seconds of lag. Use CUPTI or DCGM libnvml directly. Collect at 100ms granularity or narrower.

Real example: we ran an A100 doing ResNet-50 inference and the SM utilization swung between 20% and 95% inside 200ms periods. Polling at 1-second intervals lets you overshoot massively. You need an EWMA (exponentially weighted moving average) on the SM utilization before doing admission math.

python
# EWMA for utilization tracking
class EWMA:
    def __init__(self, alpha=0.3):
        self.alpha = alpha
        self.value = 0.0

    def update(self, measurement):
        self.value = self.alpha * measurement + (1 - self.alpha) * self.value
        return self.value

# Real-time adjustment on the admission threshold
class AdaptiveAdmission:
    def __init__(self, base_target_sm=80):
        self.base_target = base_target_sm
        self.ewma_queue = EWMA(alpha=0.2)
        self.target = base_target_sm

    def on_request_complete(self, latency_ms, sm_util):
        smoothed_util = self.ewma_queue.update(sm_util)
        if latency_ms > 100 and smoothed_util > 75:
            self.target = max(40, self.base_target - 10)
        else:
            self.target = self.base_target

The Gateway Service.

Once you have the telemetry and admission algorithm, you need it in the hot path. A standalone admission gRPC service performs well, but keep it in the same failure domain as the GPU worker.

The microservice decomposition of admission control adds a network hop during a latency-sensitive moment. Put it as a process-local library with a querying sidecar, or use an Envoy filter. Remote calls will eat 200 microseconds for each admit; your inference p50 might be 2ms. That's 10% overhead just for control. Not okay.

Performance numbers from our 2025 load test: local admission adds 6 microseconds per request (hashed lookup and mutex acquire). Remote admission over localhost gRPC adds 270 microseconds. The math is clear.

Signal Handling and Overload.

The best admission control algorithm for inference servers includes a circuit breaker. When the GPU starts thermal throttling, or NVLink bandwidth drops, or the per-SM issue rate goes sideways, your admission decisions based on historical utilization become garbage.

Watch dram__bytes_read.sum and sm__pipe_alu_cycles_active.avg.pct_of_peak_sustained_elapsed. When DRAM bandwidth exceeds 80% of peak, the compute kernels starve for data. Your admission controller should throttle requests that hit memory-heavy kernels, even if SM utilization looks low.

Triton reports this through the "gpu_utilization" and "gpu_memory_usage" metrics. But as is usual with metrics, it does not separate memory bandwidth from memory capacity. We wrote a small collector in Go that reads DCGM directly and publishes to Prometheus, then let the admission controller read from a local agent.

GPU Admission Control for Real-Time Inference

"Real-time" gets thrown around so much it's lost meaning. Let me define it: an inference request that is synchronous and latency-bound. Usually the client is waiting on a response, often in a user-facing loop. If you don't respond under 50ms, the user has moved on.

The failure pattern with real-time inference on GPUs is not throughput starvation. It's the Hutter Prize problem in reverse — you optimize throughput so hard that you batch so large that nobody gets a fast response.

For real-time inference, the right GPU admission control for real-time inference model is to enforce a maximum concurrency, not a maximum queue.

Here is the admission rule: Max 4 concurrent requests per A100 for a Transformer model if you want sub-10ms p99. Each request uses about 25% of the GPU's compute units if batch size is 1. But dynamic batching mixes them up.

I have personally benchmarked H100 for small language models. If you want p99 token time under 2ms, do not run more than 2 simultaneous requests through an 8xH100 node with vLLM or TensorRT-LLM, until you measure otherwise.

So my contrarian take: do not use "GPU utilization" as the primary admission signal for real-time inference. Use queue depth. GPU utilization averages hide tail latency.

Queue-depth-based admission:

def should_admit_queue_based(queue_depth, gpu_sm_util):
    # Real-time inference is a latency game
    if queue_depth > 2:
        return False, "queue_too_deep"
    if gpu_sm_util > 85:
        return False, "gpu_busy"
    return True, "ok"

Yes, GPU utilization for offline batch jobs can go to 95%. That's fine — the asynchronous user won't notice a 200ms extra wait. Real-time cannot tolerate it.

Open Source Maturity: The Honest View

The ecosystem for GPU admission control open source is maturing, but it's nowhere near the visibility of CPU autoscaling and memory management. In 2025, I thought by now we'd have a CNCF-graduated project doing this. That hasn't happened.

The best all-in-one right now is KServe's predictor-level concurrency controls plus an external scheduler. For production-managed services, TensorFlow Serving has simple batching but no real admission. TensorRT-LLM has decent in-engine scheduling, but admission is still delegated.

Projects worth watching in 2026:

• Triton's Dynamic Batch and Concurrent Execution: NVIDIA shipped policy hooks in 5.5, but it's still not first-class.
• KubeFlow's Pipeline Level Admissions: more about pipeline resource allocation than real-time control.
• SIVARO's own open-source controller (we will publish the SM tracking library in October 2026 at github.com/sivaro/sm-controller)

The gold standard remains the closed source inference platforms of hyperscalers. They have control planes we can't see. But their GPU admission controllers are solving the problem I describe, and in many cases their infrastructure teams are rolling their own internal versions of what I've outlined here, because they don't trust open source either.

Implementation Map: 90 Days to Sane Inference

If you take nothing else away, here is the roadmap I recommend your team follows:

• Week 1: Install DCGM exporter. Get GPU SM utilization and memory bandwidth metrics into Prometheus. Confirm you can query with 200ms resolution.
• Week 2: Put a concurrency limiter in front of your inference service. Doesn't matter how crude. Just a semaphore of size 2 for real-time, 8 for batch. Measure the p99 change. You'll see improvement immediately.
• Weeks 3-4: Implement queue depth admission as described above. No classification by model profile yet. Just same limit for every request.
• Weeks 5-8: Profile your models. Determine SM costs. Build the classifier that differentiates requests.
• Weeks 9-12: Implement the full token bucket approach, and add dynamic threshold tuning based on EWMA.

The most common failing I observe in startup teams: they try to build overload prediction models with ML before they quantify the base concurrency ceiling. Rule #1: instrument. Rule #2: form a baseline. Rule #3: let the noise drive the controls.

What Open Source Doesn't Solve

Time for honesty — GPU admission control open source can't fix all GPU contention issues. Several things still need hardware tricks:

• MPS and MIG for isolation: use them when you need hard isolation between workloads. The open source admission controller cannot prevent interference from another process that's hammering the GPU's memory pipes.
• CUDA Graphs: if the models use CUDA Graphs for reduced launch overhead, admission control gets easier (less jitter), but you need custom logic to understand graph execution. No open source controller I know handles CUDA Graph inference semantics well.

But the biggest limitation is legal — not technical. You're admitting workload based on your assumptions about process resource usage. The GPU driver lies occasionally. A buggy kernel can consume far more SMs than your estimator says. Your admission control will admit on false premises.

We had a GPU driver bug with a persistent model on H100 that fired incorrectly and consumed all SMs for a calibration loop. Our controller kept admitting requests, and p99 went to 8 seconds. The admission control couldn't detect the driver-level misbehavior. Only a watchdog that read GSP firmware logs could.

So pair admission control with a liveness watchdog that looks at actual GPU runtime state and kills the process if utilization exceeds theoretical maximum for too long.

Frequently Asked Questions

Q: What GPU admission control open source actually works with Kubernetes?

A: The combination of KServe with an external custom admission controller works best. ModelMesh has basic in-memory admission but won't handle heterogeneous compute demands. We ran ModelMesh for a year and migrated to our own controller in Envoy. Use the custom scheduler extender pattern, not the built-in bin-packing.

Q: What's the difference between rate limiting and admission control?

A: Rate limiting throttles requests at the boundary (e.g., 10 requests/sec). Admission control examines the current GPU state and decides if the specific request can run. Rate limiting is static; admission control is dynamic, informed by DCGM metrics and prediction of kernel behavior.

Q: Should I use NVIDIA MPS, MIG, or my own admission control?

A: MPS is a fairness mechanism — it'll share SMs among processes, but it doesn't prioritize tasks based on latency requirements. MIG is hard partitioning, great for isolation, bad for utilization. Use MIG when you need guaranteed isolation. Use open source admission control when you want the flexibility of a shared resources pool with controlled contention. In production at SIVARO, we use admission control plus MPS for smaller models, and MIG only when a customer demands no neighbor interference.

Q: Which metrics should drive admission control decisions?

A: The two most critical signals are (1) SM utilization - the percentage of streaming multiprocessors active per time slice, and (2) memory bandwidth utilization, not memory capacity. The third signal is kernel launch rate, which measures the overhead of context switching. Forget the classic "GPU memory used." That's about fitting, not about running fast.

Q: What about multi-node inference?

A: Once the request crosses the network to another node, admission control in the gateway loses fidelity. You must implement admission control on the target server's side, in the serving process itself. The co-location of admission logic with execution is non-negotiable for tail latency. Remote admission decisions are estimates at best, and wrong more than right.

Q: Does admission control work with vLLM/continuous batching?

A: Both. For continuous batching, have the admission controller decide how many sequences to admit into decoding. The paged KV cache idea in vLLM means memory isn't a hard barrier. But there's still SM pressure per batch. The admission controller must cap concurrency by decode tokens per second per GPU.

Q: What's the best crash recovery pattern when admission fails?

A: When the GPU admission controller is wrong and everyone gets 429s, that's a live site incident. The least painful pattern is to fall back to a dispatcher that simply alternates between rejecting and accepting partially. Add fast retries with jitter. Never spin wait on a semaphore — it burns CPU and causes cascading failures across the fleet.

Q: How do you handle batched inference requests in admission control?

A: Treat a batch as a bigger request. If a batch of 16 images has 4x SM cost of a batch of 4, the admission controller needs a size-aware cost function. In practice, compute the SM cost per batch as cost = (num_sequences * processing_flops_per_sequence) / peak_flops_per_sm. Simple regression models work fine here.

Q: Is there an open-source standard API for GPU admission?

A: Not yet. The K8s community proposed a concept for "structured resource sharing" but nothing is standard as of September 2026. As de facto custom resource, define a grpc service with Admit(InferenceRequest) returns (AdmissionResponse) and plan to change the API once a standard emerges. Avoid locking into a specific vendor's API.

Looking Ahead

The next big frontier in GPU admission control will be power-aware admission. In data centers with strict power budgets (and every data center will be power-limited by 2028), you'll need to admit requests based on available thermal headroom and power capping.

NVIDIA's GSP (GPU System Processor) exposes some power telemetry we can use. But no mainstream open source admission controller accounts for power limits yet. The SM utilization is nice, but if you can't pull the amps, you can't run the model.

The admission algorithm for inference servers will also become smarter. I expect predictive GPUs based on usage patterns will replace the reactive approach I've outlined, but only after data from GPU utilization at scale improves. There are too many false correlations today. Give me a year.

Conclusion

The gap between owning GPU infrastructure and actually controlling it is admission control. You'll pay for that gap in p99 latency and infrastructure costs. We calculated that SIVARO saved 34% on GPU spend by implementing admission control — we closed down two A100 nodes because we finally packed requests sufficiently.

Open source GPU admission control is not a single project. It's a design pattern, a telemetry approach, and a set of policies you adapt. Start with limiters, measure, iterate.

If you only take one thing from this, let it be this: your problem isn't that you lack GPU capacity. It's that you let requests in without regard for their compute cost at the instant of admission. Start by saying "no" more often at the right moment, and your "yes" will become infinitely more valuable.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.