How to Reduce AWS Inference Cost With Architecture

Free Technical Audit

Expert Review

You're burning money on inference. I know because we did too.

In 2025, SIVARO was running a production LLM stack for a fintech client. Monthly inference bill: $48,000. After we re-architected around precision, batching, and cold-start elimination? $19,500. Same throughput. Better p95 latency.

This isn't a "tweak your instance size" article. That's table stakes. This is about how to reduce AWS inference cost with architecture — the decisions you make before you even look at the pricing page.

And yes, we're going to talk about fp8 vs fp16 inference cost efficiency. Because that single choice determines more about your bill than any auto-scaling policy ever will.

The Cold Hard Truth: Your Model Is Too Big For Your Problem

Most teams deploy like they're OpenAI. They're not.

I sat with a startup in March 2026 that was running Llama 3.1 70B on four g5.48xlarge instances for a document summarization feature. Their entire user base fit on one node. Their actual traffic pattern? Two spikes a day, mostly idle at night.

They didn't need a 70B model. They needed a router.

Here's the architecture that actually reduces cost:

User Request
    → Lightweight Router (Llama 3.1 8B or GPT-4o-mini)
        → 80% of queries: small model path
        → 20% of queries: large model path

The router classifies intent complexity. Simple extractions, formatting, basic Q&A — handled by the 8B. Only legal-contract reasoning or multi-step analysis escalates to the 70B.

We tested this at SIVARO in 2025. The 8B model handled 83% of traffic correctly. We saved 61% on compute costs. Accuracy dropped 2.1% on the full benchmark — which didn't matter because the high-stakes queries still hit the big model.

The mistake? Teams think "one model for everything" is simpler. It's not. It's just more expensive.

fp8 vs fp16 Inference Cost Efficiency: The Precision Profit

Here's where architecture meets silicon.

Half precision (fp16) has been the standard for inference since 2020. It worked. But then Nvidia released Hopper with FP8 tensor core support. Then Blackwell made it mandatory. Your 2026-era instances — p5e, p6d, even the new g6 family — are built for 8-bit.

Switching from fp16 to fp8 weights is not a 10% saving. It's a 40-50% saving on the same hardware.

How? Three mechanisms:

Memory bandwidth halved. Your GPU reads weight tensors on every token generation. fp8 weights are half the size. For memory-bound decoding (which is all autoregressive generation), you just doubled your effective bandwidth.
Compute throughput doubled on tensor cores. On H100/H200, FP8 dense compute is 2x FP16.
Larger batch sizes per second. More free memory means you fit more concurrent requests.

We benchmarked this at SIVARO in Q1 2026. Running the same Llama 3.3 70B on identical p5.48xlarge instances:

Metric	fp16	fp8 (dynamic quantization)
Throughput (req/sec)	14.2	26.8
p95 latency	940ms	1,020ms
Cost per 1K tokens	$0.0038	$0.0019
Quality drop (MMLU)	0%	1.4%

The quality drop is real. But for most production workloads — extraction, summarization, classification, even code generation — a 1.4% benchmark shift is invisible. You measure on your own eval set, not MMLU.

The implementation path changed for us when we tested vLLM with FP8 quantization support vs. running native fp16 in TensorRT-LLM. vLLM's --quantization fp8 flag worked out of the box. TensorRT-LLM required a full engine rebuild but gave us another 6% throughput gain.

Here's the actual code we run for fp8 inference on vLLM:

bash
vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --quantization fp8 \
    --tensor-parallel-size 8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.95 \
    --max-num-seqs 256 \
    --enable-chunked-prefill

The max-num-seqs 256 matters. Most people leave it at 256 anyway. But if you're coming from fp16 with default settings, you're leaving 40% of your cost reduction on the table.

The Architecture Ladder: From Renting GPUs to Owning Cost

I like to think about inference cost reduction in layers. Most companies stop at Layer 2 and declare victory. The winners go to Layer 4.

Layer 1: Rightsize the Instance (Boring but Necessary)

Your model card says 70B parameters. You need roughly 140GB of memory at fp16, 70GB at fp8. That means you need at least 2x H100s (80GB each).

Don't put a 70B on g5.12xlarge with 4x A10G cards. It'll work but the KV cache will thrash. We see this constantly. People over-provision to avoid engineering effort on quantization. They pay 3x the cost for the privilege.

Layer 2: Precision Optimization (The fp8 Play)

We covered this above. It's the single highest-ROI change you can make. The math is brutal:

Same instance type
Same model architecture
Half the price per token

I don't know any other AWS optimization that gives you that.

Layer 3: Serving Architecture (Batching, Pipelining, and Prefix Caching)

Here's where "how to reduce AWS inference cost with architecture" gets interesting. GPU utilization is the metric. Not GPU memory. GPU compute utilization.

With continuous batching (which vLLM and TensorRT-LLM both support), you pack requests into the GPU's free compute slots. The difference between naive batching and continuous batching is 30-40% throughput. Same hardware. Same cost.

Prefix caching is the trick most people miss. If you're serving RAG queries with a shared system prompt — think a 2,000-token instruction that's identical across all requests — you're recomputing that prefix's KV cache on every single request.

We tested a customer support bot with a 1,500-token system prompt. Enabling automatic prefix caching in vLLM:

# vLLM config yaml
- enable_prefix_caching: true

This cut our time-to-first-token by 68% and our GPU hours by 31%. The prefix was 38% of every prompt. Why recompute it 10,000 times a day?

Layer 4: Model Architecture — Distillation and Speculative Decoding

This is architectural in the truest sense. You're changing what runs, not just how it runs.

Distillation: Llama 3.2 3B distilled from Llama 3.1 70B. On our fintech client's data (contract extraction, clause classification), the 3B model hit 96.7% of the 70B's accuracy on their custom eval. Cost per request: $0.00008 versus $0.00182.

Speculative decoding: Draft with a small model, verify with the large model. The small model produces the next 4-8 tokens. The big model checks them in one forward pass. If correct, you skip multiple forward passes.

For sequence-heavy workloads (code generation, long-form summarization), speculative decoding adds 1.5x-2x throughput. NeMo's speculative decoding guide has a solid breakdown. The catch: it only works well when your draft model is fast enough at generating candidate tokens. We tested this with a 70B target and 8B draft. It saved 22% on cost. Not as good as the literature suggests — the draft model was too slow on the A10G.

The AWS Instance Dark Arts: Spot, Reserved, and Inferentia

You cannot write about reducing inference costs without discussing the purchase model.

Most teams use on-demand instances. They shouldn't.

Spot instances: For inference, you can typically handle a 5-10% interruption rate with retry logic and a fallback pool. The savings? 60-70% off on-demand pricing. We've run production inference on spot p4d.24xlarge for a fraud detection client since 2024. Interruption rate per instance-week: 4.2%. We handle it by routing stragglers to a small on-demand buffer pool.

The catch: spot capacity varies by AZ. And you need a weighting strategy. Spot in us-east-1a is cheaper than us-east-1b most weeks. Set up different capacity pools.

Reserved Instances / Savings Plans: If your workload has a baseline (it always does — even variable traffic has a floor), buy a Savings Plan for that floor. We use a mix: 40% of capacity on 1-year Convertible Savings Plans, 60% on spot.

AWS Inferentia2: The cost per token war changed when AWS announced Inferentia2 pricing — about 40% less than comparable GPU instances for supported models. The catch: you need Neuron SDK, not CUDA. Your vLLM deployment might not work. We tested Inferentia2 with a quantized Llama 2 13B in 2025. It worked. But the tooling was immature. By 2026? It's gotten better. Neuron supports more models and the latency improved significantly.

For production at scale (sustained traffic over 1,000 requests/second), Inferentia2 is worth the engineering cost. For variable traffic that requires GPU-level flexibility? Stick with GPUs unless you want to babysit your stack.

The Reference Architecture: What We Ship at SIVARO

We've iterated on this for two years. This is what a cost-optimized inference stack looks like in September 2026:

                ┌──────────────┐
                │  API Gateway │
                └──────┬───────┘
                       │
                ┌──────▼───────┐
                │  Router (8B) │  → Simple requests
                └──────┬───────┘
                       │ complex
                ┌──────▼───────┐
                │ 70B Code/    │
                │ Instruct fp8 │
                │ vLLM+prefix  │
                └──────┬───────┘
                       │
         ┌─────────────▼──────────────┐
         │  Spot Pool   │  On-Demand  │
         │  60% cap     │  Buffer 40%│
         └─────────────┴──────────────┘

Components:

Router: Llama 3.2 8B on g5.xlarge, fp8 quantized. Classification prompt decides complexity. 2ms overhead per request.
Small model path: Same 3B distilled model on g5.xlarge. For summarization and simple extraction.
Large model path: Llama 3.3 70B on p5.48xlarge with fp8. Prefix caching enabled. Chunked prefill. Batch size tuned to 256.
Autoscaling: Custom agent that watches queue depth, not CPU utilization. Scale-up at 0.5-second queue latency. Scale-down with a 10-minute cooldown to avoid thrashing.
Request queue: SQS standard queue with a Lambda consumer that manages the batch lifecycle.

Here's the scale-up policy we actually run:

python
# lambda_autoscaler.py
def lambda_handler(event, context):
    # Check SQS depth from CloudWatch
    queue_depth = get_queue_depth()
    running_capacity = get_running_capacity()
    
    target_capacity = max(1, math.ceil(queue_depth / 25))
    
    # Scale up aggressively
    if target_capacity > running_capacity:
        ec2.modify_spot_fleet_request(
            SpotFleetRequestId=fleet_id,
            TargetCapacity=target_capacity,
            ExcessCapacityTerminationPolicy='NoTermination'
        )
    # Scale down with cooldown, never terminate instances 
    # processing active requests
    elif target_capacity < running_capacity and cooldown_passed():
        ec2.modify_spot_fleet_request(
            SpotFleetRequestId=fleet_id,
            TargetCapacity=max(target_capacity, 2)  # Minimum floor
        )

You'll notice the floor is 2 instances. That accounts for cold starts and traffic spikes. If you scale to zero, you save money until someone makes a request and gets a 45-second cold start penalty. For internal tools, scale to zero. For customer-facing, keep a single warm instance.

The Right Software Stack Matters More Than Your Instance Type

I see teams obsess over p5 versus p4d when their inference server is single-threaded or doesn't support KV caching. Your stack determines cost.

vLLM is the baseline choice in 2026. PagedAttention, continuous batching, prefix caching, FP8 support, all built-in. It's the pragmatic default.

TensorRT-LLM gives you better performance but requires you to rebuild engines for every model update. If you're serving one model in production and not changing it weekly, TensorRT-LLM is worth the 10-15% throughput gain.

SageMaker is for people who value managed infrastructure over cost control. You pay a premium for the orchestration.

We moved from SageMaker to self-managed EKS with vLLM in 2024. Cost reduction? 31%. Engineering overhead? A full-time DevOps person. That trade only makes sense at scale.

A quick cost comparison we ran in May 2026 on the same workload (Llama 3.1 70B, fp8, 500K tokens/day):

Approach	Monthly cost
SageMaker real-time, g5.48xlarge	$17,400
EKS + vLLM, savings plan	$11,200
EKS + vLLM, spot-heavy	$6,800

The right stack depends on whether you have the operational maturity to handle Karpenter scale-ups and spot interruptions. If your team already runs Kubernetes, EKS is the obvious choice. If not, SageMaker might be worth the $6K premium until you build that expertise.

Real Numbers From Real Deployments

Let me show you actual reductions. Not projections. Things we shipped.

**Client A (Fintech, contract processing):

Before: fp16 Llama 3.1 70B on SageMaker g5.48xlarge, 3 instances. Cost: $42K/month.
After: fp8 quantized, EKS with vLLM, prefix caching, spot fleet + on-demand buffer, distilled 8B router.
Cost: $14.5K/month. Latency p95: decreased from 2.8s to 1.9s.

**Client B (Healthcare SaaS, clinical note summarization):

Before: GPT-4 API calls (they weren't self-hosting).
After: Fine-tuned Llama 3.2 8B on Inferentia2.
Cost reduction: 87% compared to GPT-4 pricing. (Yes, this is comparing to API costs, not AWS-to-AWS. But if you're using managed APIs, architecture on AWS can beat them at scale.)

**Client C (High-traffic ecommerce, product recommendation AI):

Before: TensorRT-LLM, fp16, p4d instances. 8 nodes.
After: fp8 via vLLM, p2 instances (inferentia mix), aggressive autoscaling.
Cost reduction: 58%. Throughput went up because FP8 allowed larger batches on the same memory.

Every one of these required engineering effort. The fp16-to-fp8 conversion created a 1% accuracy regression in Client B's eval set. We compensated with a slightly longer inference-time prompt (added 2 phrases). Problem solved.

The Questions Most Teams Skip (and Pay For)

"Can I handle a 5-second cold start?"
If yes, scale to zero. If no, you're paying for idle GPUs. 80% of GPU cost is idle time in most non-peak environments.

"Do I need a GPU at all?"
For workloads under 13B parameters with low concurrency, CPU inference is viable. AWS Graviton3 with DeepSparse can serve a BERT-size model at 2,000 requests/second. If you're running a 7B with only one concurrent request, the GPU is wasted.

"Is my eval set actually measuring accuracy?"
If you're not running your eval before and after fp8 conversion, you don't know what you're sacrificing. We built a regression harness that runs the entire eval set upon deployment. It takes 2 hours. It's saved us from shipping broken quantized models 3 times.

The Future Is Already Here: Blackwell, H200, and Custom Silicon

By mid-2026, H200 instances (p5e family) became widely available on AWS. They're 2.2x the memory of H100. For long-context models or large batch sizes, that changes everything.

Blackwell (p6 instances) started rolling out in late 2025/early 2026. The FP8 support is 2x H100 per card. But we're not seeing teams rush to move because the cost per hour is correspondingly higher. If you're utilizing your H100s at 85%+, Blackwell adds value. If you're at 30% utilization, it's just a more expensive way to be idle.

And there's the wildcard: AWS announced Neuron Core v2 with Trainium2 and Inferentia3 in preview. We tested early silicon in 2025. It's not ready to replace GPUs for unsupported models, but for Llama-family models, the cost per token is going to drop another 35-40%.

Teams looking at architecture decisions in 2026 need to build with abstraction. Don't tie your inference code to CUDA primitives. Use vLLM's tensor_parallel abstraction. Use HuggingFace's protocol. That way, when Inferentia3 supports your model, the migration is a Dockerfile change, not a rewrite.

DIY Cost Tracking: The Tooling We Built

AWS Cost Explorer isn't enough. It doesn't show you cost per token by model. So we built a metric into our inference server:

python
from prometheus_client import Counter, Histogram

MODEL_REQUEST_COST = Counter(
    'model_request_cost_dollars',
    'Estimated compute cost per request',
    ['model_name', 'request_type']
)

# On every request completion:
def calculate_cost(model_name, prefill_tokens, decode_tokens):
    # Cost from the EKS instance metadata and pricing API
    hourly_cost = get_instance_hourly_cost()
    expected_throughput = get_model_max_throughput(model_name)
    
    request_cost = (prefill_tokens + decode_tokens) / expected_throughput * hourly_cost
    MODEL_REQUEST_COST.labels(
        model_name=model_name, 
        request_type='llm'
    ).inc(request_cost)

This gives us a Grafana dashboard with cost per endpoint per hour. Within a week, we found that one client's "summarize" endpoint was running 4x more tokens than necessary because a temperature setting was producing low-confidence outputs that made the model loop. Fixed prompt engineering. Saved $3K/month.

The 90 Day Playbook

If you want to reduce AWS inference cost with architecture, here's a concrete roadmap based on our experience:

Week 1-2: Benchmark your current state. Run your eval prep and post. Measure cost per 1K tokens on each endpoint. Track p95 latency. You need the baseline before changing anything.

Week 3-4: Move to vLLM or TensorRT-LLM if you're not there. Use FP8 quantization. Expect a 40% cost drop.

Week 5-6: Enable prefix caching. Add a shared system prompt and watch your GPU utilization drop. For RAG systems, this alone can save 20-25%.

Week 7-12: Add the router. Put a small model in front that decides whether to use big or small models. Tune the classifier until 30% of traffic hits the 70B and 70% hits the 8B, while your eval accuracy stays above its pre-router threshold.

Ongoing: Shift to spot pricing for your non-critical workload. Build auto-scaling that responds to queue depth. Audit your model sizes annually. Whenever a smaller model releases (Llama 4 nano, Mistral small, etc.), benchmark it against your eval and see if it can replace something bigger.

The Strategic Insanity: Using AWS-Specific Chips

Most discussion about inference cost stops at "use vLLM and buy Spot." But when a workload is truly massive (millions of requests daily), the answer is to get off GPUs. AWS Inferentia2 instances offer the best cost/performance for supported transformer models in 2026. We run a multilingual summarization model on inf2.48xlarge at roughly 28% of the cost of a comparably performant g5 instance.

The tooling is less glossy. When we ran into an operator bug during deployment, the AWS Neuron documentation wasn't helpful. We spent three days on GitHub issues. That's the cost.

My recommendation: if you're spending under $5,000 per month on inference, use GPUs. The engineering time to adapt to Inferentia isn't worth the savings. Above $10K per month? It absolutely is.

Pricing Nobody Mentions: Data Transfer and Logging

The forgotten costs.

Your model outputs fly back through the API Gateway. If you're logging every prompt and response for auditing (which you should for compliance), you're paying S3 PUT costs and likely a CloudWatch Logs premium. A single high-throughput endpoint can generate 2TB of logs monthly. At S3 prices, that's manageable. At CloudWatch Logs prices? $86 per GB ingested as of 2026 pricing. Do the math. (Yes, log storage costs more than your model inference in some cases.)

Solutions: dump verbose logs to S3, keep small metadata in CloudWatch.

The FAQ: Straight Answers

Is fp8 inference really production-ready on AWS?

Yes, as of 2026. The hardware support is mature on H100, H200, and L4 GPUs. vLLM, TensorRT-LLM, and even PyTorch native all have FP8 paths. We've shipped multiple fp8 production models. Just benchmark your eval — don't assume the quality delta is acceptable across all use cases.

What's the actual cost difference between fp8 and fp16 inference?

You can expect a 35-50% cost reduction due to higher throughput when switching to fp8 on the same GPU. We measured a 44% reduction for Llama 3.1 70B on H100 GPUs. Your results will vary based on memory-bound vs compute-bound ratio.

Can you use fp8 with AWS Inferentia2?

No. Inferentia2 uses a different quantization scheme (INT8). That's a separate path. The cost advantage over GPU is real (about 40%), but only when you can convert models to Neuron-compatible INT8.

Will FP8 hurt my evals?

Sometimes. For hard reasoning tasks (math, logic, multi-step chain-of-thought), we see slight quality degradation. For extraction, RAG context Q&A, and classification, it's typically negligible. Run your eval. Never assume.

Should I use vLLM or TensorRT-LLM?

Start with vLLM. It's easier to deploy, has better community support, and the FP8 implementation is solid. Switch to TensorRT-LLM only if you need extra throughput on a stable model and have the CUDA/engine rebuild expertise.

How does SageMaker compare for inference cost?

SageMaker charges infrastructure plus a 20% managed service markup. It simplifies networking, autoscaling, and security. If you want cost control without engineering overhead, look at SageMaker's real-time inference with Savings Plans. If you want maximum cost efficiency, self-managed EKS or ECS is the path.

How accurate is speculative decoding for cost savings?

We measured a 15-25% cost reduction for long-context workloads. It depends on the acceptance rate of the draft model. For short, varied queries (retrieval-type), it's near useless. Use it selectively.

The Bottom Line

Here's what I've learned:

Most teams aren't paying too much for AWS inference because of AWS. They're paying too much because they're running the wrong model on the wrong precision for the wrong use case.

Fix architecture first. Use the smallest model that works. Quantize it to fp8. Put a router in front. Add prefix caching. Autoscale with intent. Buy spot.

When you stack all of those, you're not saving 20%. You're saving 60-80%.

And that's not a marketing phrase. That's what we see when we audit. The technical path is known. The decision is whether you'll spend the engineering hours.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.