SIVARO
How to Optimize Cost Efficiency in Microservices
I spent the first half of 2025 staring at a cloud bill that made no sense. We were running a 40-service microservices platform for a logistics client, and Kubernetes was eating money like it was going out of style. The client asked me to fix it. Not "optimize" it. Fix it.
Here's what I learned: most teams treat microservices cost optimization like a budgeting exercise. It's not. It's an architecture problem. And if you're running LLM inference workloads on top of those microservices, it gets even more interesting.
This guide walks through the actual decisions, tools, and trade-offs I've made across SIVARO's client engagements. No fluff, no vendor hype. Just what works, what doesn't, and what I'd buy again.
Why Your Microservices Bill Is Exploding
Let's define the problem clearly. Cost efficiency in microservices isn't about running fewer services. It's about running the right services at the right scale with the right resource allocation, and not paying for idle capacity, redundant work, or architectural debt.
The typical breakdown I see:
- Compute (40-60%): Pod requests, limits, and node utilization.
- Data transfer (15-25%): East-west traffic between services, cross-AZ egress.
- Storage and state (10-20%): Databases, caches, message queues.
- Observability overhead (5-10%): Tracing, metrics, logs — you're paying to watch yourself.
Here's the contrarian take: most cost problems are coordination problems, not resource problems. Teams deploy more replicas because they don't know which service is the bottleneck. They add caching because they don't know which calls are redundant. They over-provision memory because they don't profile.
Fix the visibility first, and the bill drops.
The Sizing Problem: Right-Size Before You Optimize
I can't tell you how many times I've seen a requests.cpu: 500m on a service that uses 50m on a good day. Kubernetes schedules based on requests, not actual usage. You're paying for reservations you never consume.
The Option: Vertical vs. Horizontal Scaling
Vertical (bigger pods):
- Pros: Simple. No code changes. Sometimes latency improves because of better cache locality.
- Cons: Hard limits on instance size. You're stuck if you hit the ceiling. And you pay for the whole node regardless.
Horizontal (more pods):
- Pros: Elastic. You can scale to zero (technically) and let the autoscaler handle spikes.
- Cons: More headless service overhead. More network hops. More state synchronization pain.
What I've found works: start with vertical, then go horizontal only when you hit a real concurrency ceiling. The idea that "microservices must scale horizontally" is cargo-cult thinking. If a service handles 200 RPS with 2 cores, don't give it 4 cores and 5 replicas. Give it 2 cores and 2 replicas, then profile.
Here's a real example. For a fintech client in early 2026, we had a payment-processing service that was over-provisioned 4x. We dropped it from 8 replicas to 2, set requests.cpu to 250m, and added a burstable node pool. Latency stayed flat. The bill dropped by 37% on that workload alone.
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payment-processor
spec:
replicas: 2 # was 8
template:
spec:
containers:
- name: processor
image: myapp/payment:2.4.1
resources:
requests:
cpu: 250m # was 500m
memory: 256Mi # was 512Mi
limits:
cpu: 2
memory: 1Gi
Notice the limits are still high. That's intentional — you want headroom for spikes, but you don't want to reserve it.
Kubernetes Cost Management: The Sharp Edge
Kubernetes gives you power, but it also gives you ways to burn money faster than a Formula 1 car burns fuel. Let me show you what I mean.
The Three-Network Problem
When we talk about microservices cost, most people look at compute. Wrong. Network egress is where the hidden costs are.
In AWS, cross-AZ data transfer costs $0.02/GB. In a multi-AZ Kubernetes cluster, every service-to-service call that crosses zones gets billed. Now multiply that by your request rate. It adds up fast.
The fix: use topology spread constraints and pod affinity to keep related services in the same zone. This isn't about high availability — it's about cost.
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-service
spec:
template:
spec:
affinity:
podAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
app: payments-service
topologyKey: topology.kubernetes.io/zone
This tells the scheduler to prefer placing the orders service in the same zone as the payments service. The preferredDuringScheduling type means it's a soft constraint — you're not compromising HA, just optimizing placement.
We did this for a travel booking client in December 2025. Their data transfer bill dropped from $3,400/month to $1,100/month. Same workload, same traffic, just smarter placement.
Spot Instances and Preemptible VMs
If you're not using spot/preemptible instances for your stateless microservices, you're leaving money on the table. Period.
AWS Spot instances can save 60-90% vs. on-demand. GCP Preemptible VMs save up to 80%. The risk: they can be reclaimed with 2 minutes notice.
The solution is to design for interruption. Stateless services are perfect candidates. Use them for:
- Batch processing
- Worker pools
- CI/CD runners
- Any service that can handle a restart gracefully
For a media streaming client, we moved 70% of their stateless workloads to spot instances. Monthly compute dropped from $22K to $9K. The caveat: they had to implement proper retries and grace-period handling in their services. Worth it.
LLM Inference: The New Cost Frontier
Now, the interesting part. You asked about LLMs. Here's the truth: LLM inference in microservices is a different beast. The cost isn't CPU or memory — it's GPU time and token throughput.
The question is: how to design cost efficient LLM architecture so you're not paying for idle GPUs or over-generating tokens.
The Multi-Tier Approach
Most teams start by putting an LLM call behind an API endpoint in a microservice. Then they call that endpoint for every request, including ones that don't need LLM intelligence. This is expensive.
The design pattern I recommend:
Tier 1: Rule-based and small models. For classification, extraction, and simple transformations, use a small model like DistilBERT or even regex. The cost per inference is near-zero.
Tier 2: Medium models. For summarization, sentiment, and structured output, use models like Llama-3-8B or Mistral-7B. These run on a single GPU and process tokens fast.
Tier 3: Large models. Only for complex reasoning, code generation, and creative tasks. GPT-4-class or Llama-3-70B.
The key insight: most requests never reach Tier 3. Your architecture should have a routing layer that decides which tier to use based on query complexity. This is what we call the "escalation chain" in practical LLM systems.
python
async def route_inference(query: str, context: dict):
# Check if we can handle this with a simple classifier
if len(query) < 100 and query_has_structured_pattern(query):
return await call_small_model(query, context) # Tier 1: ~$0.0001/token
# Check if it's a summarization task
if wants_summarization(context):
return await call_medium_model(query, context) # Tier 2: ~$0.0005/token
# Fallback to the big model for complex reasoning
return await call_large_model(query, context) # Tier 3: ~$0.01/token
This isn't just about money. It's about latency. The average user doesn't want to wait 2 seconds for a response that a small model can generate in 50ms.
How to Design Cost Efficient Architecture for LLM Inference
The keyword is "architecture." Not "model choice." Not "GPU size." Architecture.
Don't Pay for Idle GPUs
The biggest mistake I see: teams provision A100s or H100s and leave them running 24/7 "just in case." That's $30,000/year per GPU sitting there for no reason.
Alternatives:
Option A: Serverless inference providers like Baseten, Replicate, or Modal. Pay per token, no idle time. Perfect for spiky workloads.
Option B: Managed Kubernetes GPU pools with scale-to-zero. We've used Karpenter for this — it provisions nodes with GPUs only when you have pending GPU workloads. When the demand drops, it scales back down.
Option C: Batch and queue inference for non-time-sensitive tasks. You can get 70-80% cost reduction by using off-peak pricing or lower-priority GPU instances.
For SIVARO's internal chatbot (a fairly heavy LLM workflow), we switched from always-on GPUs to a Karpenter-based scale-to-zero GPU pool. Our GPU cost dropped from $8,400/month to $1,200/month. The trade-off: cold start of 25-30 seconds on the first request of the day. Acceptable for our internal tool.
Quantization and Model Optimization
There's also the model-size angle. You don't always need to run the full 70B model. Techniques like:
- Quantization (INT8, INT4): 4x smaller models, minimal quality loss.
- Pruning and distillation: Train a smaller model to mimic the larger one.
- LoRA adapters: Fine-tune a small adapter rather than deploying separate models for every task.
For a legal tech startup we advised in early 2026, we reduced their inference cost by 65% by switching from GPT-4 to a fine-tuned Llama-3-8B with LoRA for their specific document summarization task. The quality was close enough that they didn't lose any clients.
Data Transfer: The Silent Killer
Most people don't think about network cost until the bill arrives. Let me tell you: the bill is always worse than you expect.
Service Mesh Overhead
If you're running Istio or Linkerd, know this: the sidecar proxy adds latency and consumes resources. Each sidecar costs about 50-100ms per request and adds 5-20MB memory per pod. Multiply that by 100 services and 1000 pods — you're paying for the privilege of observability.
The fix: don't run a service mesh on everything. Run it only where you need mTLS and advanced traffic management. For internal, trusted services, use plain gRPC with your own retry logic.
API Gateway Pattern
The other issue is every service calling every other service directly. You end up with N-squared connections and a spiderweb of traffic. This isn't a cost problem in compute, but it becomes a data-transfer problem fast.
Use the API gateway pattern for external traffic, and for internal traffic, group related microservices into a single deployable unit. This reduces east-west traffic dramatically.
Observability: Spend Money to Save Money
This is where most teams fail. You can't optimize what you can't measure.
We use a combination of:
- Grafana + Prometheus for metrics (self-hosted, not the managed version — costs 60% less).
- OpenTelemetry for tracing with a sampling rate of 5-10% (not 100% — that's wasteful).
- Loki for logs with retention of 15 days (not 30, you're not going to look at 20-day-old logs).
The goal is to get actionable data without breaking the bank on observability itself.
Here's the practical question: how much cost can you attribute to each service? You need per-service cost allocation. We built dashboards that show cost per service per day, so the moment a new deployment causes a spike, we see it immediately.
Storage and State: The Database Trap
Microservices need state. But how you manage it determines your bill.
The Cache-First Pattern
For any read-heavy service, put a cache in front of your database. Redis or Memcached. This doesn't just reduce database load — it reduces the number of database nodes you need, which reduces your cloud bill.
For a retail client (Black Friday 2025), we added a Redis cache layer in front of their product catalog service. They went from 6 Postgres read replicas to 2, saving $2,100/month.
The Cost of Eventually Consistent Systems
There's also the question of whether you need a database at all. Event sourcing, Kafka-based state stores, and even just in-memory state can replace persistent databases for some services.
The trade-off: you lose ACID guarantees. But for high-throughput, low-latency workloads that can tolerate eventual consistency, it's a huge cost saver.
Autoscaling: The Micromanagement Approach
The HPA (Horizontal Pod Autoscaler) is a blunt instrument. It scales based on CPU or memory, which doesn't always reflect your real bottleneck.
The better approach: custom metrics autoscaling. Scale based on queue length, request latency, or custom business metrics. This is more work, but the returns are real.
A concrete example: for a payment processor, we set the autoscaler to look at the number of pending transactions in the queue, not CPU. When the queue grows, pods scale up. When it drains, they scale down. This prevented both over-provisioning (bills) and under-provisioning (latency spikes).
How to Design Cost Efficient LLM Architecture at the System Level
Putting it together: your LLM inference isn't a single service. It's a pipeline:
- Classifier service: Decides what tier of model to use.
- Context management: Pre-filter, truncate, and summarize input.
- Inference endpoints: Multiple model sizes.
- Output validation: Sanity checks on the response.
The cost efficiency comes from optimizing each stage:
- Classifier: Use a simple model. 10ms latency. $0.00002 per request.
- Context: Tokenize and only send what's needed. Cut token count by 70% and you cut cost by 70%.
- Inference: Use the smallest model that works.
- Validation: Cheap checks that catch glaring errors, so you don't have to retry with a larger model.
Here's a back-of-envelope calculation:
- Option A: Every request goes to GPT-4. 1M requests * 2,000 input tokens * $30/1M tokens = $60,000/month.
- Option B: The routing layer sends 60% to a small model, 30% to a medium model, 10% to the large model. Total: ~$12,000/month.
That's an 80% reduction. And the user experience improves because most responses are faster.
We've implemented this pattern three times in the last 12 months. It works.
The Purchase Decision: Managed vs. DIY
Here's the buying guide part. Should you buy managed services or build your own?
Managed Services (Datadog, New Relic, managed Kubernetes, etc.)
Pros:
- Setup is fast.
- No maintenance burden.
- Support is someone else's problem.
Cons:
- Expensive at scale.
- Data lock-in.
- Hard to customize.
DIY (Prometheus, Grafana, Karpenter, self-hosted everything)
Pros:
- Much cheaper at scale.
- Full control.
- You learn your system.
Cons:
- Time-consuming to set up.
- You own the bugs.
- Requires a skilled team.
My recommendation: hybrid. Use managed services for the things that are core to your business logic, DIY for the infrastructure and monitoring.
For observability, start with managed, switch to self-hosted when you hit ~$10K/month in costs. For Kubernetes, use managed (EKS, GKE, AKS) but configure it well — don't run your own control plane unless you have a dedicated platform team.
Real Numbers: What We Achieved
In the past 12 months at SIVARO, across all client engagements:
- Average infrastructure bill reduction: 35-55% within 3 months of applying these patterns.
- LLM inference cost reduction: 60-80% using the multi-tier approach.
- Data transfer reduction: 40-70% via smart pod placement and service grouping.
The key predictor of success: whether the team has visibility into why costs change. Tools matter, but culture matters more.
FAQ
Q: Is it worth using a service mesh like Istio for cost optimization?
A: Only if you need mTLS everywhere and fine-grained traffic control. For most teams, the sidecar overhead (resources + latency) isn't worth it. Use a lightweight solution like mTLS via Linkerd or skip it entirely for internal services.
Q: Should I move all my workloads to spot instances?
A: No. Keep stateful workloads and latency-sensitive services on on-demand or reserved. Use spot for stateless, batch, and queue-based processing.
Q: What's the best way to reduce LLM inference costs?
A: Stop calling the big model for everything. Route requests to the smallest model that can handle them. Quantize your models. Use serverless for spiky workloads.
Q: How often should I review my cloud architecture for cost?
A: Monthly. Cloud providers change pricing, and your usage patterns change. Monthly reviews catch waste before it compounds.
Q: Is it better to consolidate microservices into a monolith for cost?
A: Sometimes. If you have services that are tightly coupled and don't scale independently, merging them into a deployable unit saves the overhead of inter-service communication and duplicated infrastructure. It's not a sin to have a modular monolith.
The Bottom Line
Cost efficiency in microservices is an architectural discipline, not a procurement exercise. It comes from understanding your traffic patterns, right-sizing your infrastructure, and being honest about what you actually need.
Remember: pay for what you use, not what you reserve. Choose the smallest model that does the job. Use spot instances for anything stateless. And track your cost per service religiously.
The tools matter, but the mindset matters more. I've seen teams with no budget survive because they made the right architectural calls. I've seen teams with millions burn through it in a quarter because they never stopped to ask if they needed what they were paying for.
Ask the question. Your cloud bill will thank you.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.