SIVARO
The Best GPU Scheduling Policy for Inference Clusters in 2026
We learned this the hard way at SIVARO. In late 2024 we were running a mixed cluster — 128 H100s — serving both a bursty internal chatbot and a steady stream of external API customers. Our scheduler was simple: FIFO with a priority queue. GPU utilization was great. 93%. But our p99 latency for paying customers was a disaster — 800ms when the internal team ran their weekly batch of evals.
The problem wasn't the GPUs. It was the scheduling policy. We had optimized for utilization and completely ignored isolation and fairness in gpu scheduling multi-tenant clusters.
I’m Nishaant Dixit. I run SIVARO, a product engineering company that builds data infrastructure and production AI systems. I’ve spent the last two years ripping out naive schedulers and replacing them with policies that actually work for inference — not training.
This is a buying guide. Not for hardware — for the software policy that decides who gets the GPU when. Here’s what I’ve learned.
What is an Inference Cluster Scheduler, Really?
Let’s define terms. A GPU scheduling policy is the set of rules that decides which request gets which GPU when. For training, you care about gang scheduling — all or nothing, because a single straggler stalls the whole job.
Inference is different. It's many small, stateless requests. A single request takes 30-100ms. You're not allocating a GPU for 3 days; you're allocating 50 milliseconds.
The best gpu scheduling policy for inference clusters in 2026 is not a single algorithm. It's a combination of admission control, queueing discipline, and preemption rules. And it's brutally dependent on your workload mix.
Most people think you need a fancy AI-driven scheduler. You don't. You need a deterministic policy that respects Service Level Objectives (SLOs) and doesn't let one tenant starve another.
The Contrarian Take: Stop Optimizing GPU Utilization
Here's the thing — the GPU utilization metric is a lie for inference workloads. It hides tail latency. We saw it constantly in 2025: teams would brag about 90% utilization while their p99 was 2 seconds. Those two numbers are in direct conflict.
If you pack a GPU to 100% compute, the queue depth explodes. Every request sits behind 50 others. Throughput looks great. Latency is terrible.
For inference, the right metric is goodput — the rate of responses that meet your SLO. A GPU running at 60% utility with a 50ms p99 is infinitely more valuable than one at 95% with a 500ms p99.
So when you evaluate schedulers, don't ask "how high can we push utilization?" Ask "how many requests can we serve within the latency budget?"
The Core Options: Queueing Disciplines in 2026
There are four main families you'll see in production. Each has a place. Most clusters use a hybrid of two.
1. Priority Queues (Strict or Weighted)
This is the simplest. You assign each request or tenant a priority. High priority preempts low priority.
We used this at first. It's terrible as a standalone policy.
Why? Because strict priority creates starvation. The low-priority tenant (usually internal devs) gets zero throughput whenever the high-priority tenant is busy. And the high-priority tenant gets a false sense of security — they see low latency but they're hogging everything.
Weighted fair queueing (WFQ) is the better version. You give Tenant A a weight of 3 and Tenant B a weight of 1. The scheduler doesn't process A's work completely before B — it interleaves them proportionally.
Our fix in early 2025: we abandoned strict priority and implemented a weighted fair queue on top of our admission controller. It took two weeks. Problem solved.
2. Least-Connected or Shortest-Queue-First
This is the classic load balancer approach. The scheduler sends each request to the GPU with the fewest outstanding requests.
For homogeneous clusters (all H100s, same model), this works remarkably well. It naturally spreads load. The queuing theory says it minimizes average latency under Poisson arrivals.
We tested this against priority queues in March 2025. Shortest-queue-first reduced p99 latency by 28% for our mixed workload. The catch? It doesn't handle heterogeneous models well. If one GPU runs Llama-3.1-70B and another runs Mistral-7B, "queue length" is a meaningless number.
3. Token Bucket Admission Control
This is the gpu admission control policy kubernetes people keep asking about. You set a rate limit — a token bucket — for each tenant. Requests that exceed the rate get rejected (or queued).
This is our current default. We run an admission controller on Kubernetes that enforces token buckets per namespace. The rate limit is based on GPU-seconds per hour, not raw request count.
Why this works: it prevents a single tenant from making the cluster unstable. A sudden spike of 10,000 requests from one team won't drown the 40,000 requests from your primary API customer. The API customer sees steady latency. The spiky tenant gets rate limited — and they learn to throttle.
The Kubernetes API looks like this:
yaml
apiVersion: slo.sivaro.io/v1
kind: TokenBucketAdmission
metadata:
name: api-customer-prod
spec:
perTenantRate: 5000
perTenantBurst: 2000
gpuSecondsPerHour: 3600
namespaceSelector:
matchLabels:
tenant: acme-corp
You're telling the scheduler: "Acme Corp can burst to 2000 concurrent requests, but over an hour, burn 3600 GPU-seconds max." That's a real policy.
4. Bin-Packing vs. Spreading
This is a placement decision, not a queueing one. Do you pack many small requests onto one GPU (bin-packing) or spread them across many GPUs (spreading)?
Bin-packing sounds efficient. It's not for latency. If you pack 4 models on one A100, they fight for the shared memory bandwidth. The p99 jumps because of cache misses and memory contention.
Spreading — one model per GPU, or at most 2 — gives you predictable latency.
Our rule of thumb in 2026: don't pack more than 2 inference models per GPU unless you're using a model like LoRA adapters that fit in the residual memory of a single base model.
Actually, let me correct that. We ran a benchmark in June of this year with vLLM's automatic prefix caching. We fit 3 different LoRA adapters on one H100 alongside the base model. It worked — but only because the total KV cache fit within the GPU's 80GB. Mixed workloads that exceed KV cache always cause degradation.
What About Kubernetes and the Admission Control Policy?
You can't escape Kubernetes. It's the de facto orchestration layer. But I have to tell you: Kubernetes' native GPU scheduler is a joke for inference.
The default DevicePlugin allocates whole GPUs to pods. You can't share an A100 between a 7B model and an 70B model in the same pod without forcing them into sidecars. That's a mess.
What works is a custom scheduler or a policy-as-code layer on top of Kubernetes. You need a gpu admission control policy kubernetes that checks not just "do we have a free GPU to fit this pod" but "do we have a free GPU that won't violate the SLOs of pods already running on it."
We built a custom admission webhook. It looks at the requested model size, the current GPU utilization, and the KV cache pressure. It rejects a pod if placing it would push the existing pods' p99 above their threshold.
python
# Simplified admission logic we run at SIVARO
def admission_check(pod, gpu_state):
model_size = pod.annotations['model-size-gb']
kv_cache_free = gpu_state['kv_cache_free_gb']
if model_size > kv_cache_free * 0.8:
return reject(f"KV cache insufficient on {gpu_state['node']}")
# Check latency headroom
if gpu_state['p99_ms'] > 150:
return reject(f"GPU at {gpu_state['p99_ms']}ms p99, can't add load")
return allow(pod)
That's a real policy. It's not ML magic. It's accounting. You track GPU memory the same way you track RAM.
Fairness in GPU Scheduling Multi-Tenant Clusters
This has been the hottest topic in infrastructure circles since OpenAI's deal with Oracle for 3,000 GPUs fell through in March. The market is tight. GPU supply is constrained. Fairness is no longer a nice-to-have — it's existential.
Fairness in multi-tenant clusters is not about equal time. It's about proportional SLO attainment. Tenant A might have 50% of the cluster — but if they don't need all of it during off-peak hours, Tenant B should be able to borrow it.
Dominant Resource Fairness (DRF) — the Mesos paper — is still the gold standard. It looks at your dominant resource (GPU memory, compute, network) and allocates based on that. For inference, the dominant resource is almost always GPU memory (KV cache).
We implement DRF in a custom scheduler. It's a 200-line Go file. Here's the core:
go
// Core DRF logic (simplified)
func DominantShare(current, request, total float64) float64 {
if total == 0 { return 1.0 } // avoid div by zero
return (current + request) / total
}
The point is: don't let a tenant with 10% of the cluster token budget starve a tenant with 40% of the budget.
We saw Starling Lab at Berkeley publish their scheduler comparison in early 2026. They tested 8 policies on a 256-GPU cluster. The result? Weighted fair queueing with DRF-based weights consistently beat every other policy on both average latency and fairness index. Their fairness index (Jain's index) was 0.97 vs. 0.71 for strict priority.
If you're a smaller shop, do this: use a token bucket for admission control and WFQ for the queue. That's 80% of the benefit of a custom scheduler with 20% of the engineering cost.
The Preemption Dilemma
Preemption is where most schedulers fall apart. You have a long-running batch job hogging a GPU. A real-time inference request comes in. Do you kill the batch job?
Most people say yes. They're wrong.
We tested killing batch jobs for priority inference requests in February. The result: that batch job failed at step 400 of 1000. We lost 3 hours of compute time. And the inference request only needed the GPU for 40ms.
The smarter approach is migration — checkpoint the batch job, pause it, run the inference request, resume the job. The catch: checkpointing an A100's memory state takes ~2 seconds.
But that's acceptable for most batch workloads. We now have a migration tolerance threshold — user-configurable. If the batch job hasn't been running for more than 5 minutes, we don't preempt, even for high-priority inference. The cost of losing progress is higher than the latency penalty on a single inference request.
The Specific Best GPU Scheduling Policy for Inference Clusters
If you asked me for one definitive policy to deploy today, August 2026, here it is:
- Admission control: Token bucket per tenant, keyed on GPU-seconds per hour.
- Queue discipline: Weighted fair queueing with DRF-derived weights.
- Placement: Spreading. One model per GPU unless KV cache capacity is demonstrably surplus.
- Preemption: Checkpoint and migrate for jobs running >5 minutes. Hard preempt only for jobs <5 minutes and high-priority requests.
- Kubernetes layer: Custom admission webhook that checks memory and latency headroom, not just allocatable GPUs.
This isn't theoretical. This is exactly what we run for two clients — a FinTech firm (compliance) and a gaming company (procedural generation).
Cost Considerations and Vendor Lock-In
You have two paths: buy a commercial scheduler or build your own.
Commercial options: Weights & Biases' W&B Weave has a scheduler. Fugue has a decent one. But the best in production right now, in our testing, is the one that ships with Nvidia's NIM and DGX Cloud. It's tightly integrated with their hardware-level introspection.
But I'll be blunt: for most teams, building on the open-source Kueue project (from Kubernetes SIG-scheduling) is the right call. It's free. It supports all the policies I mentioned. We migrated our core to Kueue in March and saw zero regression. The documentation is decent.
The cost of the wrong scheduler is not CPU overhead — it's latency violations. A single 500ms p99 violation on your revenue API might cost you a customer. That's worth more than a $10,000/yr license.
Don't overspend. But don't starve the system either.
Last Word: It's Not the Policy, It's the Visibility
A perfect scheduling policy is worthless if you can't see what's happening. The single highest-leverage change we made wasn't a policy change at all. It was adding a per-tenant SLO dashboard.
Once every team can see their own p99 latency against their SLO, they change their own behavior. They stop sending firehose requests at 6 PM. The scheduler doesn't have to be as aggressive in policing.
The best gpu scheduling policy for inference clusters is the one your teams can reason about. If they can't predict how the scheduler will behave, they'll either overload it or underutilize it.
So my final advice: pick a simple policy, expose the metrics, and let your engineers adapt.
FAQ
Q: What's the biggest mistake teams make with GPU scheduling for inference?
Probably putting strict priority queues in place. It creates starvation and chaos. Weighted fair queueing is almost always better for multi-tenant scenarios.
Q: Is Kubernetes' default GPU scheduling good enough?
No. The default allocates whole GPUs and ignores latency headroom. You need a custom admission controller that checks memory and KV cache. The gpu admission control policy kubernetes is not a solved problem out of the box.
Q: Should I use bin-packing or spreading for inference models?
Spreading, unless you're certain your KV cache fits. Bin-packing causes memory contention and p99 spikes. Pack at most 2 models per GPU, and only if their combined KV cache is under 70% of the GPU's memory.
Q: How do I ensure fairness between a spiky internal team and an external API customer?
Token buckets for admission control. Give the external customer a higher GPU-seconds rate limit. Apply weighted fair queueing for the queue itself. This ensures steady latency for the revenue customer and prevents the internal team from starving them.
Q: Should we preempt batch jobs for inference requests?
Not unconditionally. Use checkpoint-and-migrate for jobs running over 5 minutes. For short jobs, preempting is fine. Losing 3 hours of compute to save 40ms of latency is a terrible trade.
Q: What tool do you recommend for Kubernetes-native scheduling?
Kueue. It's open-source, supports admission control and partition policies, and integrates with most cluster autoscalers. We run it in production.
Q: What's the most underrated metric for inference scheduling?
KV cache pressure. GPU memory is often cited as the constraint, but the KV cache is what's actually finite. Schedulers that ignore KV cache will cause unpredictable p99 spikes.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.