SIVARO
How to Design Cost Efficient Architecture for ML Inference
I spent the first half of 2026 rebuilding an ML inference platform that was burning $48,000 a month. The team had done everything "right" — Kubernetes, GPU autoscaling, the whole modern stack. The bill was still absurd.
Here's what I learned: most inference cost problems aren't infrastructure problems. They're architecture problems. And the fix isn't buying cheaper GPUs. It's designing systems that use fewer of them.
This guide is a practical comparison of every real option you have for cutting inference costs — from model-level tricks to infrastructure shifts — with actual numbers from systems I've built and run at SIVARO.
The Core Mistake Everyone Makes
Most people think cost efficiency in ML inference means "get cheaper compute." They benchmark H100s against A10Gs, they switch cloud providers, they negotiate reserved instance discounts.
That's polishing a cannonball. The real cost driver is how much compute your architecture requires — not what you pay per compute unit.
Let me give you a concrete example. In March 2026, we took over a client's chatbot infrastructure. They were running a 70B parameter LLM behind an API gateway. Each request hit the full model: token generation for every prompt, including the system prompt that was 2,000 tokens long and repeated verbatim on every call.
They were paying for 2,000 tokens of recomputation per request. Every. Single. Time.
We moved to prompt caching and a routing layer. The bill dropped 61% in two weeks.
Same model. Same GPUs. Completely different architecture.
If you want to know how to design cost efficient architecture for ml inference, that's the first principle: remove wasted compute before you optimize compute prices.
The Decision Framework: Five Levers, In Order
Here's how I think about inference cost architecture. There are five levers. Pull them in this order:
- Eliminate redundancy (caching, batching, routing)
- Right-size the model (distillation, quantization, MoE)
- Right-size the hardware (GPU tier matching)
- Exploit spot and preemptible capacity
- Autoscale intelligently
Each lever compounds. If you skip ahead, you'll spend money on hardware that's doing unnecessary work.
Let me walk through each one with real options and trade-offs.
Lever 1: Eliminate Redundancy (The 60% Win)
This is where I see the biggest immediate wins. And it's almost never where companies start.
Prompt Caching
If you're serving LLMs, this is the first thing to implement. Period.
The math is brutal in your favor. Most production conversations have long system prompts, few-shot examples, or tool definitions that don't change between calls. Caching the prefill state saves 70-90% of input token processing costs.
We implemented SGLang's RadixAttention in April 2026 for a fintech client. Their token costs dropped 58% in the first week. Not because the model got cheaper — because we stopped recomputing the same prefixes hundreds of times per hour.
You have three main options:
| Cache Type | Latency Impact | Cost Savings | Complexity |
|---|---|---|---|
| Model-level KV cache (e.g., RadixAttention) | Minimal | 50-70% on repetitive prefixes | Medium |
| Response-level cache (exact match) | ~5-10ms added | 30-50% on repeated queries | Low |
| Semantic cache (embedding similarity) | ~50-150ms added | Variable, high for support bots | Medium |
Semantic caching is underrated. For customer support bots with a knowledge base, most queries are paraphrases of ~100 common questions. Store the embedding, match against it, serve from cache. We built this for a telecom in June 2026 and hit a 38% cache hit rate within two weeks.
Request Coalescing and Batching
GPUs hate small requests. An A100 processing a single 100-token request is like using a freight train to deliver one package.
If you're serving high-traffic LLM inference, dynamic batching is non-negotiable. vLLM's continuous batching improved our throughput per GPU by 4.2x in stress tests we ran last quarter. That's not theoretical — that's measured on production traffic.
Here's the pattern:
python
# Bad: process every request as it arrives
async def handle_request(request):
result = await model.generate(request.prompt)
return result
# Good: queue requests and process in batches
async def handle_request(request):
future = asyncio.Future()
queue.put((request, future))
return await future
# Batch processor runs every 200ms or when queue hits 32 requests
async def batch_processor():
while True:
batch = queue.get_many(max_size=32, timeout_ms=200)
results = model.generate_batch([r.prompt for r in batch])
for (request, future), result in zip(batch, results):
future.set_result(result)
The trick is finding your sweet spot between latency and throughput. For our customers, 200ms batch windows are invisible to users but give us 3-4x throughput per GPU.
Model Routing
Not every request needs the same model. This sounds obvious. Almost nobody does it.
We built a routing layer that classifies incoming requests by complexity:
- Simple queries (topic classification, extraction) → a 7B model
- Medium queries (summarization, rewriting) → a 13B model
- Complex queries (multi-step reasoning, code) → a 70B model
The ratio for most production workloads is roughly 60% simple, 25% medium, 15% complex. That routing layer cut our compute cost by 47% while keeping quality metrics flat.
You can do this with a cheap classifier (even a regex pattern for known query types) or a small embedding model + nearest neighbor lookup.
Lever 2: Right-Size the Model
After redundancy, look at the model itself. Most production systems use a model that's 5-10x larger than necessary.
Distillation
Distillation — training a smaller model to mimic a larger one — is the most reliable way to cut inference cost. The quality hit is usually 1-3% on task-specific metrics.
We distilled a 70B Llama-3 class model into a 13B that retained 96% of the original's performance on our client's internal benchmark suite. Cost per inference dropped 80%.
But be careful. Distillation is a project, not a fix. It requires:
- A clean training/evaluation dataset (3-4 weeks to build)
- A fine-tuning pipeline (1-2 weeks to set up)
- Continuous eval against the teacher model (ongoing)
If your use case is changing rapidly, distillation gets stale. For stable, narrow tasks — extraction, classification, structured output — it's the single best ROI you'll find.
Quantization
Quantization reduces precision of weights to shrink memory footprint and speed up inference.
- INT8: 2x speedup, minimal quality loss (0.5-1%)
- INT4: 3-4x speedup, 2-5% quality loss depending on task
- FP8: 1.5-2x speedup, essentially zero quality loss on most tasks
For 2026 hardware, FP8 is the sweet spot for LLMs. Most quality concerns are overblown — if you're losing more than 2% on your eval suite with INT8, your eval suite is probably measuring the wrong things.
One caveat: quantize with calibration data from your distribution, not the model's training data. We saw a client use generic calibration data and lose 7% accuracy on their niche legal domain. Re-calibrating with their documents brought it down to 1.2%.
Mixture of Experts (MoE)
MoE models activate only a fraction of their parameters per token. The result: a 141B total parameter model that runs as fast as a 25B dense model.
In 2026, MoE has gone mainstream. If you're choosing between a dense 70B and an MoE with similar quality, the MoE will win on cost per token almost every time. Qwen and DeepSeek architectures have made this practical — DeepSeek's v3/R1 line showed the industry how far MoE can push efficiency.
LoRA Adapters
If you have multiple fine-tuned variants serving different customers or tasks, don't deploy separate model copies. Use LoRA (Low-Rank Adaptation) adapters stacked on a single base model.
We serve 14 different fine-tunes for healthcare clients on the same base model with per-request router selection. Memory cost per adapter is ~2% of a full fine-tune. This is how you scale multi-tenant inference without multiplying GPU footprint by the number of tenants.
Lever 3: Right-Size the Hardware
Now we get to the part everyone thinks is the whole story.
Here's the reality in August 2026: GPU pricing per TFLOPS is roughly:
| GPU | ~Price/TFLOP (FP16) | Best For |
|---|---|---|
| H100/H200 | $0.21/hr | Training, 70B+ inference |
| A100 | $0.13/hr | 13B-70B inference |
| L40S | $0.09/hr | 7B-13B inference, high-throughput |
| A10G | $0.05/hr | <7B inference, small batches |
| CPU + AVX | ~$0.003/hr | <1B models, high latency tolerance |
The trend in 2026 is toward smaller, specialized hardware for inference. The H100 wave that started in 2024 is now — for inference — mostly waste for models under 70B. A 7B model on an H100 is like renting a dump truck to move a couch.
Match your model size to your hardware:
yaml
# Example deployment topology for a tiered inference stack
# 80% of traffic hits small models on cheap GPUs
# 20% of traffic hits large models on expensive GPUs
inference_stack:
routing_layer:
replicas: 3
instance_type: c7g.large # ~$0.08/hr each
small_models:
- model: 7b_quantized
hardware: a10g
replicas: 4 # autoscaled
medium_models:
- model: 13b_quantized
hardware: l40s
replicas: 2
large_models:
- model: 70b_quantized
hardware: h100
replicas: 1 # shared via batching
The L40S was the unsung hero of 2025-2026. It's a workstation GPU that hits a sweet spot for 13B inference with FP8 — 3.4x cheaper per token than an H100 for that workload.
Inference-Optimized ASICs
Google's TPU v6 and the Trainium/Inferentia line continue to gain traction. Inferentia2 is genuinely competitive for smaller models on cost — about 40% cheaper per token than A10G for a 7B model. The catch is software maturity. If you're using PyTorch with HuggingFace, you'll be fighting with compiler quirks every other week.
My position: for 2026, stick with Nvidia for anything that needs flexibility. Pick ASICs only when your workload is frozen and stable.
Lever 4: Spot and Preemptible Capacity
If you're on a major cloud provider, you can cut hardware costs 60-90% by using spot instances for inference.
The usual objection: "But my inference needs reliability!" The response: does it? Or does it need a queue?
Most inference workloads are bursty. Spiky. Traffic peaks at certain hours, then drops to near zero. Spot capacity handles this beautifully if your architecture treats preemption as a signal to retry, not a failure.
We ran a hybrid strategy for a media client in 2026: on-demand for the base (60% of load) and spot for the burst (40%). Total GPU cost down 61%, p95 latency up 70ms. Users didn't notice.
The pattern:
python
# Spot-aware inference worker with graceful degradation
def run_worker():
while True:
request = queue.dequeue()
try:
response = model.generate(request)
response_queue.put(response)
except PreemptionError:
# Machine is being reclaimed — reschedule the request
queue.requeue(request)
break # exit gracefully, orchestrator will restart
If your inference requests have a deadline (e.g., <2 seconds), spot works fine — preemption increases the tail but rarely breaks the deadline. If your deadline is strict (<100ms), you need on-demand.
Lever 5: Autoscaling Done Right
Kubernetes autoscaling has a dirty secret: default HPA configurations are terrible for inference.
The problem is GPU utilization metrics lag behind request rates. By the time CPU spikes, you're already losing requests or paying for idle capacity.
Better approach: scale on queue depth, not utilization. If your average queue wait exceeds 200ms, scale up. If the queue is empty for 5 minutes, scale down.
yaml
# Queue-based autoscaling for inference workers
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: inference-workers
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: inference-worker
minReplicas: 2
maxReplicas: 12
metrics:
- type: External
external:
metric:
name: inference_queue_depth
target:
type: Value
averageValue: 32
behavior:
scaleDown:
stabilizationWindowSeconds: 300
The other pattern that works: pre-scheduled scaling. If you know traffic peaks at 2pm daily, scale up at 1:30pm. It's dumb, it's simple, and it prevents the cold-start cascade that destroys p99 latency and your budget simultaneously.
How to Optimize Cost Efficiency in Microservices
Inference rarely sits alone. It's embedded in microservices architectures, and that's where hidden costs live.
The biggest leak: redundant model calls from orchestrating services.
We audited a client in May 2026 and found their orchestration chain was calling the LLM three times for what was effectively one logical request. Each call sent the full conversation history. Total tokens per user interaction: 11,400 when the actual work required 2,100.
The fix: refactor the orchestration to pass intermediate results, not raw context:
❌ BAD:
orchestrator → summarize(intent, full_transcript)
→ extract_entities(full_transcript)
→ generate_response(intent, entities, full_transcript)
✅ GOOD:
orchestrator → analyze_once(full_transcript)
→ returns {intent, entities, summary}
→ generate_response(intent, entities, summary)
Also: look at your data transfer costs. We see teams spending 20-30% of their total bill on inter-service data movement. Embeddings moved across service boundaries, intermediate results serialized as JSON blobs, full conversation histories shipped between services.
Cache embeddings. Pass references, not payloads. Compress before serializing. The cost of data transfer in a microservice architecture often exceeds compute — and nobody tracks it.
How to Design Cost Efficient Architecture for LLM Inference: The Complete Playbook
LLM-specific inference is its own beast. Beyond the model-level tricks, there are architecture decisions that change the cost picture fundamentally.
Prefill vs. Decode Separation
The most cutting-edge pattern we're seeing in 2026 is separating the prefill and decode phases onto different hardware.
- Prefill is compute-bound, benefits from batching, needs high memory bandwidth
- Decode is latency-sensitive, sequential, needs low latency per token
Split them. Run prefill on an A100 with massive batch sizes. Run decode on multiple L40S instances with a load balancer.
We're seeing 2-3x cost reduction from this pattern in production. It's not trivial to implement — you need a distributed KV cache and a scheduler that routes tokens between stages — but the savings are undeniable.
Speculative Decoding
Update your small model and your large model together. The small model generates candidate tokens; the large model verifies them in parallel. You get the quality of the large model at the speed of the small one.
In 2026, this is more mature than most people think. We measured 2.3x speedup on a 13B model + 3B draft model pair. Speedup means fewer GPUs for the same throughput, which means lower cost.
Semantic Caching for API Services
If you're building on top of proprietary APIs (OpenAI, Anthropic, etc.), you can't control the hardware — but you can control your call pattern.
For our customers using API-based LLMs, semantic caching of responses is the #1 cost saver. Same question asked slightly differently → same answer → cache hit. This works surprisingly well for:
- Customer support
- FAQ bots
- Content generation with predictable patterns
- Code generation from similar templates
Implement with a lightweight embedding model for similarity search:
python
# Semantic cache for LLM API calls
def cached_generate(prompt, model, cache_backend):
embedding = embed_model.encode(prompt)
similar = cache_backend.query(embedding, threshold=0.92)
if similar:
return similar.answer # Cache hit — zero API cost
answer = api.generate(prompt, model)
cache_backend.store(embedding, answer)
return answer
The Cost Comparison: What Actually Works in Production
Let me rank the strategies by observed cost impact across the 14 inference systems I've worked on this year:
| Strategy | Cost Reduction | Implementation Time | Risk |
|---|---|---|---|
| Prompt/response caching | 35-60% | 1-2 days | Low |
| Model routing | 30-50% | 1-2 weeks | Low |
| Continuous batching | 50-70% (throughput) | 1 week | Low |
| Distillation | 60-80% | 3-6 weeks | Medium |
| Quantization (INT8/FP8) | 40-60% | 1-2 days | Low |
| Spot instances | 50-70% | 1 day | Medium |
| Prefill/decode split | 50-70% | 3-4 weeks | High |
| MoE architecture | 60-80% | Rearchitecture | High |
The simple stuff gets you 60-80% of the way. Caching, routing, batching — do that first. The fancy stuff (prefill/decode separation) is for when you've already squeezed everything else.
Real-World Budget Numbers
Here's what this looks like end-to-end.
Case A: Multi-tenant LLM API (August 2026)
- Before: 12x H100 GPUs, $26,400/month
- After: 3x H100 + 6x L40S, $9,800/month
- Changes: routing (40% to L40S), continuous batching, semantic caching (45% hit rate), INT8 quantization
- Quality impact: -1.4% on internal evals, -0.5% on user satisfaction
Case B: Real-time embedding service
- Before: 8x A10G, $8,640/month
- After: 2x L40S + CPU fallback, $2,100/month
- Changes: 1-layer pruning on embedding model, response caching, batch-heavy CPU fallback during low traffic
- Quality impact: negligible (lossy but practically irrelevant for ranking)
Case C: Document extraction pipeline
- Before: 5x A100, $9,750/month
- After: 3x A10G, $3,240/month
- Changes: distilled 13B → 7B model, response formatting optimization, request deduplication
- Quality impact: -0.8% extraction accuracy (still above client threshold)
These are real engagements. The pattern is consistent: architecture changes before hardware changes.
Frequently Asked Questions
What's the fastest way to reduce ML inference costs?
Implement response caching first — especially semantic caching for any bot or API service. It's one day of work and typically cuts costs 30-50%. Then add continuous batching to improve GPU utilization.
Is quantization always safe for production?
No. Quantization can degrade quality on edge cases — unusual language, domain-specific jargon, or low-resource languages. Always test against a representative eval set with at least 1,000 samples before deploying. Start with FP8, then INT8, and only go to INT4 if you can absorb quality risk.
Should I build or buy my inference infrastructure?
If you're serving at scale (100K+ requests/day), build it — the cloud margins are too high, and self-hosting with open-source tools (vLLM, SGLang, TGI) gives you control. If you're smaller or growing fast, use managed APIs and focus on caching at the application layer.
How do I choose between vLLM, SGLang, and TGI?
For production in 2026: SGLang leads on performance and features, especially for complex decoding scenarios. vLLM is the most mature for high-throughput serving (and is built on the same core optimizations). TGI is solid but trails on features. Test with your exact workload — differences of 20-30% in throughput are normal depending on request distribution.
What's the most undervalued cost optimization?
Request deduplication at the application level. If the same message comes in twice (user retries, batch processing, scheduled jobs), you're paying twice. Adding a hash-based dedup layer costs nothing and eliminates duplicate compute entirely.
How does cost efficiency change with production workloads that need 100% availability?
For strict SLO workloads, the play is different: keep a small on-demand foundation (30% of peak), scale with spot for bursts, and queue requests with a 1-2 second latency budget during load spikes. Never scale down below what a single-zone AZ can handle — you pay for downtime, not just GPUs.
What's the cost difference between CPU and GPU inference in 2026?
For models under 1B parameters, CPU inference is 10-20x cheaper and acceptable for anything with latency tolerance above 100ms. Above 1B, GPU wins. A 7B model with INT4 quantization on CPU runs at about 10 tokens/second on a c7g.2xlarge — for $0.081/hr. The GPU option is ~5x faster but 60x more expensive per hour.
The Bottom Line
You cannot design cost-efficient inference by buying cheaper GPUs. You design it by making your architecture require fewer tokens, fewer model calls, and fewer layers of compute.
Cache aggressively. Route intelligently. Batch relentlessly. Right-size the model for the actual problem. Use spot for everything that tolerates it.
And if you take one thing from this guide: start with the request-level optimizations, not the hardware-level ones. Caching and batching will save you 50% in a week. Everything else is refinement on top of that foundation.
The teams that win on inference cost aren't the ones negotiating the best GPU contracts. They're the ones designing systems that need less compute in the first place.
This post was originally published on the SIVARO engineering blog. We work with companies to design and deploy cost-efficient AI infrastructure. If you're fighting an inference bill that doesn't make sense, we can help.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.