SIVARO
Admission Control to Prevent GPU Fragmentation
March 2025. A fintech client in Singapore calls me at 2am. They've got 48 H100s across six nodes. A critical LLM fine-tuning job — 8 GPUs, tight deadline — has been pending for nine hours. The cluster dashboard says 12 GPUs are "available." But those 12 GPUs are scattered across four nodes as single-GPU inference pods. You can't squeeze an 8-way tensor-parallel training run out of that.
The GPUs aren't idle. They're fragmented. And the cluster scheduler, left to its own devices, will happily keep churning out one-GPU pods that make the problem worse.
This is the problem admission control to prevent GPU fragmentation solves. And no, I don't mean "just buy more GPUs." I mean a specific architectural pattern: intercepting pod creation requests at the API server level, validating them against cluster-wide allocation invariants, and rejecting anything that would push your cluster past a fragmentation threshold.
In this article, I'll walk you through how I've built this at SIVARO for three different clients, the exact webhook patterns, the quota logic, and where it breaks down. You'll leave with working code and a mental model you can adapt to your own cluster.
The Problem Nobody Talks About in GPU Clusters
Here's what most GPU cluster operators believe: Kubernetes scheduling is enough. You set nvidia.com/gpu: 1 on a pod, the scheduler finds a node with a free GPU, done. And for a dev environment with 4 GPUs and one team? Sure, works fine.
Now scale that to 200 GPUs, 15 teams, a mix of training jobs (need 8, 16, or 32 GPUs on the same node) and inference services (need 1 GPU, can live anywhere). The scheduler's greedy first-fit algorithm starts creating a patchwork. Node 1 has 7 GPUs busy, 1 free. Node 2 has 3 busy, 5 free. Node 3 is full. Your 8-GPU training job? Can't go on Node 1 (only 1 free). Can't go on Node 2 (only 5 free). Stuck.
I've seen this pattern on clusters ranging from 32 to 512 GPUs. The bigger the cluster and the more heterogeneous the workload mix, the worse it gets. By mid-2025, with H200s and Blackwell B200s entering production fleets at major cloud providers, this stopped being a "dev cluster quirk" and became a production reliability issue.
The Kubernetes Dynamic Resource Allocation (DRA) work is moving in the right direction — structured resources, partitionable GPUs, extended resources with attributes. But DRA is still maturing, and most production clusters I work with in 2026 are running 1.29–1.32 with device plugins, not full DRA. You need admission control now, not after DRA reaches GA everywhere.
What Admission Control Actually Does for GPU Scheduling
Let's get precise. A Kubernetes admission controller is a webhook (validating or mutating) that sits between a user's kubectl apply and the scheduler. The API server calls your webhook before the object is persisted. You get 10 seconds. You can:
- Reject the request (validating webhook, return
allowed: falsewith a message) - Modify the request (mutating webhook, patch the pod spec)
- Enrich the request (add labels, annotations, resource requests)
For GPU fragmentation prevention, the pattern is mostly validating with occasional mutation. You're saying: "This pod wants 1 GPU on a node that currently has 7 of 8 in use. If I let this through, that node becomes unschedulable for any 2+ GPU job. Rejected."
The key insight: you're not replacing the scheduler. You're adding a cluster-level invariant that the per-pod scheduler can't see. The scheduler asks "can this pod go on this node?" Your admission controller asks "will the cluster still be able to schedule future work if I allow this?"
That's the difference. And it's why this isn't just a ResourceQuota or a LimitRange. Those are per-namespace. You need cross-namespace, cluster-wide logic.
Building an Admission Controller for Your Cluster
Here's the skeleton I use. Go, net/http, talks to the K8s API to read current GPU allocation state, evaluates the request against a fragmentation model, returns allow/deny.
go
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
corev1 "k8s.io/api/core/v1"
admissionv1 "k8s.io/api/admission/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
)
const (
maxGPUsPerNode = 8
minContiguousForTraining = 4 // reject 1-GPU pods if < 4 GPUs remain on a node
)
type GPUState struct {
TotalGPUs int `json:"total_gpus"`
AllocatedGPUs int `json:"allocated_gpus"`
}
func handleAdmission(w http.ResponseWriter, r *http.Request) {
var review admissionv1.AdmissionReview
if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
http.Error(w, fmt.Sprintf("could not decode body: %v", err), http.StatusBadRequest)
return
}
resp := &admissionv1.AdmissionResponse{
UID: review.Request.UID,
Allowed: true,
}
// Decode the pod
var pod corev1.Pod
if err := json.Unmarshal(review.Request.Object.Raw, &pod); err != nil {
resp.Allowed = false
resp.Result = &metav1.Status{Message: fmt.Sprintf("decode error: %v", err)}
} else {
gpuRequest := getGPURequest(&pod)
if gpuRequest > 0 {
// Check cluster state, apply fragmentation logic
if shouldReject(pod.Namespace, gpuRequest) {
resp.Allowed = false
resp.Result = &metav1.Status{
Message: fmt.Sprintf(
"Admission denied: requesting %d GPU(s) would fragment cluster. "+
"Current free GPUs on best-fit nodes do not satisfy min-contiguous policy.",
gpuRequest,
),
}
}
}
}
review.Response = resp
review.Request = nil
json.NewEncoder(w).Encode(review)
}
func getGPURequest(pod *corev1.Pod) int {
total := 0
for _, container := range pod.Spec.Containers {
if gpu, ok := container.Resources.Requests["nvidia.com/gpu"]; ok {
total += int(gpu.Value())
}
}
return total
}
func main() {
http.HandleFunc("/admit", handleAdmission)
fmt.Println("GPU admission controller listening on :8443")
http.ListenAndServeTLS(":8443", "/certs/tls.crt", "/certs/tls.key")
}
The shouldReject function is where your policy lives. In my production deployments, it does three checks:
- Node-level contiguous availability: Query all nodes, sum free GPUs per node. If no node has ≥
minContiguousForTrainingfree GPUs, reject any 1-GPU request. - Namespace isolation: Track per-namespace GPU allocation. No single namespace can consume more than 60% of cluster GPUs (configurable).
- Temporal backfill window: If a large training job (≥8 GPUs) has been pending >30 min, tighten the threshold. Block even 2-GPU inference pods.
The last one is non-obvious. I added it after watching a 16-GPU fine-tuning job starve for 11 hours while a batch of one-GPU text-classification pods kept getting scheduled. The system was "working." The job wasn't running. Nobody was happy.
AI Training Cluster Quota Management Best Practices
This section gets asked about constantly, so let me be direct.
The word "quota" is misleading here. A traditional K8s ResourceQuota says "namespace X can have at most Y GPUs total." That's a ceiling. What you actually need for GPU clusters is a scheduling priority layered on top of a ceiling.
Here's what I've found works (and I've iterated on this across four different cluster deployments since 2023):
Tier 1: Hard ceiling per namespace. Your ResourceQuota. nvidia.com/gpu: 16 for the ML team, nvidia.com/gpu: 4 for the data science team. This prevents one team from eating the cluster. Boring, necessary, works.
Tier 2: Admission-level fragmentation guard. This is the webhook I described above. It's additive to the quota, not a replacement. The quota says "you can have at most 16." The admission controller says "even though you could take GPU #17 on node 4, you can't, because doing so leaves node 4 with only 1 free GPU and no other node has 4+ free."
Tier 3: Fair-share reclamation. Use Kueue or Volcano's gang-scheduling for the actual scheduling. Kueue (which went GA in the 1.30 era and has been stable through 1.32) handles the "queue up training jobs, batch-schedule them when enough GPUs free up" problem. The admission controller handles the "don't let small pods break the cluster" problem. They solve different layers.
yaml
# Kueue ResourceFlavor + ClusterQueue — simplified
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: training-h100
spec:
namespaceSelector:
matchLabels:
tier: training
resourceGroups:
- covers: [all]
flavorsQuotas:
- flavor: h100-8x
resources:
- name: nvidia.com/gpu
nominalQuota: 48
fairSharing:
weight: 10
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
name: default
namespace: ml-training
spec:
clusterQueue: training-h100
The critical detail: your admission controller and Kueue need to talk. Or rather, they need to share state. I handle this by having the webhook read a ConfigMap (or, in the fancier setup, a small Redis instance) that Kueue updates with pending job sizes. That way the admission controller knows "there's a 32-GPU job waiting" and tightens thresholds preemptively.
Admission Control for LLM Inference on Kubernetes
This is where it gets spicy, because LLM inference has different GPU characteristics than training.
A single LLaMA-3-70B in 4-bit quantization needs 1 GPU (H100). A Llama-3-405B in FP8 needs 8. A mixture-of-experts model like Mixtral 8x22B might want 2 GPUs for decent throughput. Your inference fleet is heterogeneous in GPU demand, and it's persistent — these pods run for days, not hours.
If you let 40 single-GPU inference pods scatter across 6 nodes, you've turned your 48-GPU cluster into 48 one-GPU islands. No training job fits. No larger inference model fits.
My approach for admission control for LLM inference on Kubernetes:
- Label inference pods with expected GPU lifetime. If
inference/persistent: "true", the admission controller applies stricter fragmentation rules. If it's a batch inference job (will finish in 2 hours), it's more flexible. - Prefer consolidation. The mutating webhook can add a
nodeAffinitypreference to pack single-GPU inference pods onto fewer nodes, leaving other nodes intact for larger requests. - Rate-limit new inference deployments. If cluster fragmentation (measured as "number of nodes with < 4 free GPUs") exceeds a threshold, new inference pods queue instead of deploying.
yaml
# Mutating webhook: pack inference pods onto fewer nodes
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: gpu-fragmentation-mutator
webhooks:
- name: gpu.packer.sivarо.io
admissionReviewVersions: ["v1"]
sideEffects: None
clientConfig:
service:
name: gpu-admission
namespace: kube-system
path: /mutate-pack
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
matchPolicy: Equivalent
objectSelector:
matchLabels:
workload-type: llm-inference
The packing logic is straightforward: query the scheduler's NodeResources or read nvidia.com/gpu allocation from pod objects. Find the node with the fewest free GPUs that can still host this pod. Assign it there. You'll get clusters where inference is dense on 3 nodes and training has 3 clean nodes.
I've seen this take a 48-GPU cluster from ~40% effective utilization (fragmentation tax) to ~75%. Not 100%. Nothing's 100%. But the gap between 40 and 75 is the difference between "we need to buy 24 more H100s" and "we're fine."
The Trade-Offs Nobody Mentions
I want to be honest here because the blogosphere won't.
Admission control adds latency to pod creation. Every kubectl apply now hits your webhook. If your webhook is slow (bad network to the API server, cold Redis lookup, heavy state computation), you're adding 200-800ms to every pod creation. For batch training jobs that spin up 64 pods? You're adding 30+ seconds to startup. I mitigate this by making the state read local (in-memory cache refreshed every 5 seconds) rather than hitting the API server per-request.
You can reject legitimate workloads. A data scientist who really needs that one GPU for a quick experiment gets a rejection message. You need a good escalation path. At SIVARO, we build a Slack alert into the rejection: "Pod X rejected by GPU admission controller. Reason: fragmentation threshold. Contact #gpu-ops to override." The override is a one-liner annotation on the pod that bypasses the webhook. It's an emergency hatch, not a regular path.
It doesn't solve bad scheduling policy. If your K8s scheduler is doing worst-fit and spreading pods across all nodes, no amount of admission control fixes that. You need a bin-packing scheduler (Volcano, or K8s 1.29+'s leastAllocated scoring plugin tuned aggressively) and admission control. They're complementary.
The state model gets complex fast. "How many GPUs are free" is simple. "How many contiguous GPUs are free on each node, considering which pods are gang-scheduled, considering which GPUs are in a NUMA topology that matters for NVLink bandwidth, considering that 2 GPUs on node 3 are in maintenance" — that's a graph problem. I've written a 400-line state evaluator for a client with H100 NVLink topology awareness. It works. It's not fun to maintain.
If you're running <32 GPUs, I'd skip the topology awareness. Just do the count-based logic. It catches 85% of fragmentation issues.
Getting It Right: A Practical Checklist
You don't need to build all of this on day one. Here's the sequence I recommend:
Week 1: Deploy a validating webhook. Simple count-based logic. Reject 1-GPU pods when any node drops below 4 free GPUs. Get the TLS cert and webhook registration working. This alone prevents the worst fragmentation.
Week 2: Add namespace-level tracking. ConfigMap with per-namespace GPU allocation. Reject if a namespace exceeds its soft cap (even under its hard ResourceQuota).
Week 3: Integrate with Kueue or Volcano. Share pending-job state. Tighten thresholds when large jobs are queued.
Week 4+: Add the mutating webhook for inference packing. Add topology awareness if you have NVLink/InfiniBand. Add the temporal backfill logic.
Monitor: expose a Prometheus metric gpu_cluster_fragmentation_ratio (free GPUs on best-fit node / total free GPUs). Alert when it drops below 0.5.
bash
# Quick smoke test: try to create a 1-GPU pod when cluster is fragmented
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: test-frag
labels:
workload-type: llm-inference
spec:
containers:
- name: dummy
image: nvidia/cuda:12.4.1-base-ubuntu22.04
resources:
requests:
nvidia.com/gpu: 1
command: ["sleep", "3600"]
EOF
# Expected (when fragmented):
# Error from server: admission denied: requesting 1 GPU(s) would
# fragment cluster. Current free GPUs on best-fit nodes do not
# satisfy min-contiguous policy.
FAQ
Does admission control replace a GPU-specific scheduler like Volcano or Kueue?
No. The admission controller runs before the scheduler. It prevents bad requests from entering the cluster. The scheduler (Volcano, Kueue, or vanilla K8s) decides where allowed pods go and handles gang scheduling. You need both. The admission layer is a gate; the scheduler is the traffic controller.
Can I do this with a simple Kubernetes ValidatingAdmissionPolicy (CEL) instead of a custom webhook?
For basic rules, maybe. "Reject pods requesting more than 8 GPUs" is a one-line CEL expression. But the fragmentation logic requires reading other pods' state, computing per-node GPU availability, and checking pending job queues. CEL policies can't do cross-object queries. You need a custom webhook with API server access. I've tried the CEL approach twice. Both times I hit the wall within a week.
How do I handle GPU types? What if I have H100s and A100s mixed?
Treat them as separate resource pools. nvidia.com/h100: 1 and nvidia.com/a100: 1 via NVIDIA Device Plugin labels. Your admission controller tracks fragmentation per GPU type. An 8-GPU H100 training job can't use A100s. The fragmentation calculation is per-type, per-node.
What happens if my admission controller goes down?
You set failurePolicy: Ignore on the webhook. If your controller is unreachable, the API server lets pods through. This is the correct default. A down admission controller shouldn't block all pod creation. You lose fragmentation protection temporarily, but your cluster doesn't freeze. I've had this happen twice in production (bad deploy, bad cert rotation). Five minutes of "unprotected" scheduling is fine. Three hours of "no pods can start" is an incident.
Is this applicable to multi-tenant clusters where different companies share the GPU pool?
Yes, and arguably more important there. The namespace-to-tenant mapping makes the per-tenant quota logic even more critical. I'd add a per-tenant "fair share" enforcement: if tenant A is using 70% of free GPUs and has no pending large jobs, tighten their admission threshold more aggressively. This is where NVIDIA's multi-tenancy patterns with GPU isolation meet K8s admission control.
How does this interact with spot/preemptible GPU instances?
It should, but most implementations I've seen ignore it. If 12 of your 48 GPUs are spot instances that can be revoked in 5 minutes, your "free GPU" calculation is wrong. I handle this by tagging spot GPUs with a label and weighting them at 0.5 in the fragmentation calculation. A node with 4 spot + 2 on-demand free GPUs is treated as having 5 effective free GPUs, not 6. Not perfect. Better than pretending spot GPUs are permanent.
Do I need this for a single-node 8-GPU setup?
No. If you have one node, there's no cross-node fragmentation. Your only problem is "is there a free GPU," which the scheduler handles. This pattern starts mattering at ~2 nodes minimum, but the pain really kicks in at 4+ nodes with mixed workload sizes. If you have 4 H100s on one box and one team, just use a ResourceQuota and call it a day.
Wrapping Up: What You Actually Need to Ship
You don't need a research paper. You need a 200-line Go webhook, a TLS cert, a webhook registration manifest, and a clear policy decision: "what fragmentation level do we tolerate?"
That last part is the hard part. Not the code. The policy. "Reject 1-GPU pods when < 4 free on best node" — is 4 right? Maybe it's 2 for your workload mix. Maybe it's 6 if you run a lot of 8-way training. You'll know after two weeks of watching your gpu_cluster_fragmentation_ratio metric.
Admission control to prevent GPU fragmentation isn't sexy. It's a webhook. It's a rejection message at 2am that a data scientist grumbles about. It's a ConfigMap that a junior SRE accidentally deletes (have a backup. Have two backups).
But it's the difference between a cluster that works and a cluster that works most of the time, except on the days your largest training job matters, when 11 GPUs are scattered and stuck and nothing runs.
I've fixed that problem four times now. It's always the same fix. Just nobody builds it until it hurts.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.