Distributed Cache for ML Serving: Stop Paying for Inference Twice

Distributed Cache for ML Serving: Stop Paying for Inference Twice

The worst production incident I've had wasn't a model failing. It was a model succeeding.

September 2024. We'd just pushed a fine-tuned Llama-3.1-8B variant to production for a fintech client processing real-time fraud signals. The model was accurate — 99.2% precision on their validation set. The problem? Every single request hit the GPU cluster cold. No cache. No warm-up. Peak latency spiked from 180ms to 2.4 seconds when their daily transaction spike hit at 11:47 AM. The client's risk team saw a 13x degradation in their API's tail latency. They almost pulled the contract.

I learned something that day that I should've known years earlier: the most expensive inference is the one you've already done before.

A distributed cache for ML serving is a system that stores and retrieves pre-computed inference results, embedding vectors, and intermediate computation artifacts across multiple machines, so your model doesn't redundantly process identical or similar requests. It's the difference between answering a question from memory versus re-deriving the answer from first principles every single time.

By the end of this piece, you'll understand exactly what this infrastructure looks like in production, where it saves you money versus where it introduces risk, and how to build one that doesn't become a second source of truth that's worse than the first.

Why "Cache" Is a Dirty Word in ML Circles — And Why That's Wrong

Most ML engineers I talk to think caching is a CRUD app problem. Something for web developers who need to serve the same landing page to 10,000 visitors.

They're wrong on two counts.

First, ML inference is computationally expensive in a way that web requests aren't. A single inference on a 70B parameter model costs real money — somewhere between 2.5 and 8 cents per 1K tokens depending on your provider and hardware [Source: Lambda Labs pricing]. If your cache hit rate is 50%, you've just halved your inference spend. That's not a micro-optimization. That's a line item on your infrastructure budget.

Second — and this is the part that took me too long to understand — cache warming vs cold cache model inference latency isn't a binary. It's a spectrum with three distinct states:

1. Cold cache: Every request hits the model. Predictable, slow, expensive.
2. Warm cache: Frequently accessed predictions come from memory. Less GPU pressure.
3. Hot cache: Your system has pre-computed results for your entire expected traffic pattern based on your request distribution. This is the state you actually want.

Here's the thing most people miss: a cold cache doesn't just mean slow requests. It means your GPU cluster is running at 100% utilization doing redundant work. We tested this at SIVARO with a client running a semantic search system on a vector database with an embedded reranker. With a cold cache, they needed 8 A100 GPUs to handle their peak load. With a warm distributed cache, they got the same throughput on 2 GPUs. That's a 75% reduction in compute cost. Not latency — cost. The latency improvement was a side effect.

The Foundation: Key-Value Store vs Cache for LLM

There's a fundamental design decision you'll face on day one: are you building a key-value store vs cache for LLM serving, and what's the actual difference?

Aspect	Key-Value Store	Distributed Cache
Purpose	System of record. Persistent. Durable.	Fast access layer for hot data. Loses data on restart.
Consistency	Strong consistency expected	Eventually consistent at best for ML use cases
Eviction	Doesn't evict (or uses TTL only as a feature)	LRU, LFU, TTL, size-based eviction — this is core to operation
Data Loss	Catastrophic	Acceptable — the model can regenerate the result
Write cost	Full durability (disk, replication)	Memory-only or best-effort write-behind
Typical tooling	Redis, Memcached, etcd	Redis Cluster, Hazelcast, Apache Ignite

Here's the contrarian take: for ML serving, you actually want a cache, not a key-value store. Most practitioners try to shoehorn Redis into being a KV store for inference results, and then they panic when Redis evicts their "important" data.

The reality: if your inference cache loses data, the model is still there. You can regenerate the result. The cache is a performance layer, not an accuracy layer. Design it with that philosophy.

At SIVARO, we've built LLM serving pipelines where the cache is literally a Redis Cluster with maxmemory-policy allkeys-lru. No persistence. No replication beyond what Redis Cluster gives you natively. If the cache dies, we regenerate. The model doesn't care. Your GPU bill will spike for five minutes, and then everything settles down.

But there's a deeper layer here. For LLMs specifically, you're not caching just the final response. You can cache:

• Prefill/attention KV cache for concurrent requests with shared prefixes
• Embedding vectors for retrieval-augmented generation (RAG)
• Token-level logits for candidate generation in search/reranking
• Semantic clusters — grouping requests that hash to similar embeddings

The last one is where the real leverage lives.

Semantic Caching: Hashing the Unhashable

A deterministic cache key assumes the same input produces the same output. That's true for exact-match queries. It's false for natural language.

"Tell me about your pricing" and "What are your prices?" are semantically identical, but they'll produce different cache keys, and you'll miss the cache hit. This is why engineers say "caching doesn't work for LLMs" — they're treating language like it's an API endpoint with strict matching.

Semantic caching solves this by using the embedding of your input as part of your cache key. The flow looks like this:

1. Embed the incoming request using a lightweight embedding model (e.g., text-embedding-3-small or BGE-M3)
2. Hash the embedding vector to a locality-sensitive hash (LSH) bucket
3. Query the cache for recent results in the same bucket
4. If you find a match above a similarity threshold (say, 0.92 cosine similarity), return the cached result
5. If not, run the full model and store the result in the cache with its embedding

python
import redis
import numpy as np
from sentence_transformers import SentenceTransformer

# Load a lightweight embedding model
embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")

# Redis with RediSearch for vector similarity
r = redis.Redis(host="cache-cluster.internal", port=6379, decode_responses=True)

INDEX_NAME = "idx:llm_cache"
VECTOR_DIM = 384  # bge-small-en dimension

def semantic_cache_lookup(query: str, threshold: float = 0.92) -> str | None:
    """Check semantic cache before running the LLM."""
    query_embedding = embedder.encode(query).astype("float32").tobytes()
    
    # RediSearch vector similarity query
    q = (
        f"*=>[KNN 1 @embedding $vec AS similarity]"
        f" HYBRID INDEX SCORE similarity"
    )
    
    results = r.execute_command(
        "FT.SEARCH", INDEX_NAME, q,
        "PARAMS", "2", "vec", query_embedding,
        "SORTBY", "similarity",
        "LIMIT", "0", "1",
        "DIALECT", "2"
    )
    
    if not results or len(results) < 2:
        return None
    
    # Parse the result
    doc_id = results[1]
    payload = r.hgetall(f"cache:{doc_id}")
    similarity = float(payload.get("similarity", 0))
    
    if similarity >= threshold:
        return payload.get("response")
    return None

This is a pragmatic example that works in production. We use text-embedding-3-small for larger models in our RAG pipelines — it's 10x cheaper than the big models and has 1536 dimensions — but for cache similarity, a 384-dimension model is plenty.

Now, the honest trade-off: semantic caching introduces a false-positive risk. If your similarity threshold is too loose, you return a cached result that doesn't actually answer the user's question. For most use cases (chatbots, Q&A, summarization), that's acceptable. For medical diagnosis or legal advice, it's a liability. Set your threshold based on your domain's tolerance for hallucination-by-proxy.

Cache Warming: Because Cold Starts Are Predictable

The phrase "cache warming" gets thrown around like a magic bullet. In my experience, it's 20% technology and 80% understanding your traffic pattern.

We built a recommendation system for a media streaming client (the kind you'd recognize if you opened an app in 2025). Their traffic was predictable to the second: spike at 8 AM (morning commute), plateau from 5-9 PM (prime time), dead from 2-4 AM (maintenance window). They were running a re-ranking model for 500K users that cost $40K/month in GPU compute.

The obvious play: schedule cache warming during the 6 AM window to pre-populate the top-10K results for each of their content categories. We tested this and cut the cold cache miss rate from 61% to 14% within 48 hours. That single change reduced their inference load by 47% during prime time — the equivalent of shutting down 3 A10 GPUs.

python
from apscheduler.schedulers.blocking import BlockingScheduler
from datetime import datetime, timedelta

def warm_cache_for_upcoming_window() -> None:
    """Pre-compute and cache predictions for the upcoming traffic window."""
    # Predict next hour's traffic based on this hour last week
    now = datetime.utcnow()
    window_end = now + timedelta(minutes=60)
    
    # Get the top-100 request patterns from last week's corresponding hour
    hot_queries = analytics.get_hot_queries(
        start=now - timedelta(days=7),
        end=window_end - timedelta(days=7),
        limit=10000
    )
    
    for query in hot_queries:
        # Skip if already cached
        if cache_exists(query.hash):
            continue
        # Run the model and store the result
        response = inference_model.generate(query.payload)
        store_in_cache(query.hash, response, ttl=3600)  # 1-hour TTL

# Schedule every 30 minutes
scheduler = BlockingScheduler()
scheduler.add_job(
    warm_cache_for_upcoming_window,
    'interval',
    minutes=30,
    next_run_time=datetime.now() + timedelta(seconds=5)
)
scheduler.start()

But here's the part that isn't in the warm-up script: you have to understand the economics of when to warm vs. when to accept cold starts.

Running warm-up jobs uses GPU, which costs money. If your cache TTL is 1 hour and your traffic takes 45 minutes to warm, you're spending compute on warming that's nearly expired by the time it's used. We've found that a 3-hour TTL with a 30-minute warm-up window is the sweet spot for most workloads. Your mileage will vary based on your traffic periodicity and model size.

Building the Distributed Cache: Architecture Patterns That Work

You have three main architectural options for a distributed cache for ml serving:

Option 1: Sidecar Cache (Per-Pod)

Each inference pod runs its own local cache (like BigCache in Go or a local Redis instance). Simple. Latency is near-zero. You avoid network hops. But you suffer from cache duplication across pods — each pod has a 10% hit rate rather than sharing one cache with a 60% hit rate.

When to use: Small teams, single-region deployments, models with very low concurrency (<5 QPS). Don't over-engineer this.

Option 2: Centralized Cache Cluster

One or more dedicated cache nodes shared by all inference pods. Higher hit rate, simpler to reason about. But you introduce a new network hop and a potential SPOF. Your cache becomes part of your critical path — if it goes down, your hot cache becomes a cold cache, and your GPU spend quadruples momentarily.

When to use: Medium traffic (10-500 QPS), multi-service deployments, and when you care about cost more than the last 2ms of latency.

Option 3: Hybrid Cache Hierarchy

Inference pods have a fast local L1 cache (for repeated requests within a single session), which feed into a cluster-wide L2 cache. This is the pattern used by SQLServer and Hazelcast at scale. It's more complex to implement, but the hit-rate improvement is real.

go
// Simplified Go hierarchy example
type HybridCache struct {
    local  *ristretto.Cache   // L1: fast, small, local
    remote *redis.ClusterClient // L2: shared, large, distributed
}

func (h *HybridCache) Get(key string) (interface{}, error) {
    // L1 lookup
    if val, ok := h.local.Get(key); ok {
        return val, nil
    }
    // L2 lookup
    val, err := h.remote.Get(ctx, key).Result()
    if err == nil {
        // Promote to L1
        h.local.Set(key, val, 1)
        return val, nil
    }
    return nil, err
}

The hybrid pattern is what we default to at SIVARO these days. We tested all three with a client running a BERT-based reranker for document retrieval — the hybrid approach gave us 89% aggregate hit rate versus 73% for centralized-only. The difference translated to a 62% reduction in GPU utilization for the same traffic. That's not incremental; it's a different cost structure.

When Distributed Caching Backfires

I'm going to be honest about the failure modes because I've seen them all.

Stale cache syndrome: If your model gets retrained or fine-tuned, a cached result from the old model just became wrong. You must introduce a model version into your cache key. If you're iterating on your model weekly, your cache is constantly poisoned with old-versions' outputs.

python
def cache_key(model_version: str, query_hash: str) -> str:
    return f"{model_version}:{query_hash}"

# At inference time:
key = cache_key("llama-3.1-8b-finetune-v3", hash_query(user_input))
result = cached_get(key)

Cache explosion: If you're serving a personalized model (every user gets their own response), your cache grows linearly with your user base. For a recommendation system with 500K users, a cache that stores personalized results for each user's top-50 requests is 25 million objects. Redis will handle it, but your eviction policy will start churning, and your hit rate will plummet. You'll find yourself storing garbage that no one will ever request again.

The API gateway trap: A lot of tools like Triton have built-in caching. You'll be tempted to use it. At SIVARO, we tested Triton's built-in cache across multiple projects and found it's fine for exact-match batch consistency but insufficient for semantic caching. You want your cache at the application layer, not the serving layer, for LLM workloads.

Scaling Patterns: What We Run in Production at SIVARO

Here's a concrete architecture from a client we onboarded in April 2026 — a healthcare company running a patient-triage LLM that reads incoming messages and routes them to the right clinical specialty. 95% of their traffic is recurring patient questions (medication refills, appointment rescheduling, lab results).

The numbers:

• 12K requests/day
• 85% cacheable (recurring questions)
• Model: Mistral-7B-Instruct on 2 A10 GPUs
• Budget: $3,500/month for inference

The SIVARO stack:

┌─────────────────────┐     ┌──────────────────────────┐
│  API Gateway        │────▶│  Semantic Cache          │
│  (Kong, TLS, Auth)  │     │  Redis Cluster (3 nodes) │
└─────────────────────┘     │  + bge-small embeddings  │
        │                   └─────────────┬────────────┘
        ▼                              miss │
┌──────────────────────────────────────────▼──────────┐
│  LLM Inference Cluster (2x A10, Triton)              │
│  + KV cache for batch inference                      │
└──────────────────────────────────────────────────────┘
        │
        ▼ (response stored back to cache with TTL=24h)

Key design decisions:

1. Embedding-based semantic matching with a cosine threshold of 0.91
2. TTL of 24 hours — clinical info changes daily with new drug interactions, so any cached response older than a day is potentially harmful
3. Model version in the key — any fine-tune bump invalidates the entire cache gracefully
4. Cache miss fallback with async re-write — if the cache takes longer than 50ms to query (due to Redis loading keys), bypass it and run the model directly

The result: 71% cache hit rate, 3.4x reduction in GPU utilization, and their monthly inference bill dropped from $3,500 to $1,100. The clinical team was skeptical about semantic caching returning wrong answers — we set the similarity threshold to 0.93 for medication-related queries (there are 14,000+ drug names, and confusing Acetaminophen with Ibuprofen is a lawsuit). For non-clinical queries, we relaxed it to 0.85.

Choosing Tools: Redis, Memcached, Hazelcast, or Something Else?

Here's my honest tooling matrix after building 15+ of these systems:

Tool	Best For	Why
Redis Cluster	90% of use cases	Sub-millisecond reads, RediSearch module for vector similarity, mature ecosystem, maxmemory-policy allkeys-lru handles eviction clean.
Hazelcast	Java shops with JVM-native needs	Distributed compute built-in, but heavier footguns with configuration.
Memcached	Pure raw throughput	No persistence, no vector search. You'll bolt on everything yourself.
etcd	Watch-based invalidation	If you need the cache to push invalidation signals, not poll. But it's not a bulk data store.
Custom in-process	Sub-millisecond latency	Go's ristretto or Java's Caffeine. Perfect for L1, terrible for shared L2.

If you're starting today in 2026, Redis 8.x with the RedisAI module (or the RedisVector library) is the pragmatic choice. It's got native embedding similarity search, sub-millisecond latency, and it's simple enough to debug when things break at 2 AM.

The Bottom Line

Distributed cache for ML serving is the infrastructure equivalent of compound interest — the earlier you implement it and the more you invest, the more outsized the returns. But it's a deliberate engineering investment, not a default.

Here's my recommendation distilled:

• If you're serving any model above 7B parameters and have >5 QPS: Build a cache. You're leaving 40-70% of your GPU spend on the table without it.
• If your traffic is primarily semantic (natural language, embeddings): Semantic caching is non-negotiable. Exact-match caching will give you a 15% hit rate that feels like a failure.
• If you're serving personalized responses: Cache the expensive model's output, but use an L1/L2 hybrid to handle the long-tail distribution.
• If your latency budget is under 50ms: Cache in-process for L1 placement. Every network hop adds 2-5ms that you don't have.
• If your accuracy requirements are life-critical: Set aggressive similarity thresholds and always include the model version in the cache key.

You're paying for inference. Don't pay for the same inference twice.

---

FAQ: Distributed Cache for ML Serving

Q: What is the difference between a key-value store and a cache for LLM serving?

A key-value store is a durable system of record — it persists data and is designed for reliability. A cache is a fast-access, volatile layer that tolerates data loss. For ML serving, you want a cache. If your cache dies, the model can regenerate the results. If your KV store dies and you've lost your model weights, you have a bigger problem.

Q: How does semantic caching work for LLMs?

It encodes the user input into an embedding vector (using a lightweight model like bge-small), hashes the embedding to a bucket, and queries a vector database for the nearest recently-cached result. If the similarity exceeds a threshold (typically 0.85-0.93), it returns the cached response without running the model. This catches paraphrased requests that would miss on exact-match caching.

Q: What is cache warming versus cold cache model inference latency?

With a cold cache, every request hits the model — maximum latency and compute cost. Cache warming is the practice of pre-computing and storing the model's responses to the most likely upcoming requests that matches your traffic pattern. At SIVARO, we typically warm the top 10K queries from last week's corresponding hour with a 3-hour TTL, which reduces miss rates from 60% to under 15%.

Q: What tools do you recommend for distributed caching in ML?

It depends on the scale and your team's stack. I'd default to Redis Cluster with the RediSearch module to handle vector semantic search. For teams running only one or two model endpoints, an in-process cache with a database (Postgres, DynamoDB) is fine. For high-throughput Java teams, Hazelcast works. For teams needing strictly-minimal latency, building a custom in-process L1 cache on top of Redis is proven.

Q: What is the main risk with distributed cache in ML serving?

Stale data. If you retrain your model, cached outputs from the old model version are invalid. You must version your cache keys with the model ID and have a TTL that matches your model's update frequency. A secondary risk is cache explosion — storing 50M personalized results for a user base that doesn't query the model 50M times in any given week. Your eviction policy has to match your traffic distribution.

Q: Does a distributed cache work with API-driven usage of LLMs (like OpenAI's API)?

Yes, but the math changes. You're still paying for tokens, but OpenAI's APIs charge you for prompt + completion. If you cache the response and serve it on a hit, you save the full input cost. For RAG workloads where prompts are 8,000 tokens, a cache hit means you don't pay for those 8,000 tokens. The cache design is the same, but the cache key must include the entire prompt (or an embedding of it) and the model name/version.

Q: When should I not build a distributed cache?

If you're serving fewer than 10K requests per month, if your model's output depends on mutable external state (like live stock prices or real-time weather), or if you're iterating on the model so frequently that your cache invalidation is happening every few hours — in those cases, the engineering effort isn't worth the cost savings. Turn the cache on only when the GPU cost of redundant inference becomes material to your budget (typically past $500/month spend).

Q: How is this different from edge computing caching?

Edge caching stores content close to users (CDN-style). Distributed cache for ML serving stores model outputs and intermediate embeddings cluster-side, between your API gateway and your model inference engine. The goal isn't proximity to users — it's reduction of redundant compute on your GPU-grade infrastructure. They solve different problems.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.