SIVARO
How to Evict Stale Embeddings from Cache
I spent three days in March debugging a recommendation system that was serving embeddings from February. The cache was working perfectly. That was the problem.
Here's the thing nobody tells you about embedding caches: they don't fail loudly. They serve stale vectors with perfect confidence, and your recall metrics slowly rot while you wonder why user engagement is slipping. By the time you notice, you've been recommending "new" movies to people based on a model that hasn't existed for six weeks.
Let me define the problem clearly before we get into the mechanics.
An embedding cache stores vector representations of your data—user profiles, documents, images, whatever—so you don't have to recompute them every time you need them. When you know how to cache llm embeddings properly, you can cut inference costs by 60-80% and drop p99 latency from 80ms to under 5ms. We've measured this at SIVARO across dozens of production systems. But the caching is only as good as your invalidation strategy.
Because here's the hard truth: embeddings are not static. You retrain your model, and every vector in that cache is now garbage. Your users interact with your system, and their semantic representations drift. Your document corpus grows, and old vectors don't reflect new context relationships.
Most teams build a distributed cache for ml serving—Redis, FAISS, pgvector, whatever—set a TTL, and call it done. Then they wonder why their system feels "off" three weeks later.
It's not off. It's stale. And you need a real eviction strategy.
What "Stale" Actually Means for Embeddings
Let's be precise about this. A stale embedding is any vector that no longer accurately represents the semantic content of the entity it claims to represent.
There are three distinct flavors of staleness, and they require different treatments.
Model version staleness. You fine-tune your embedding model on new data Tuesday night. Every vector computed with Monday's model is now in a different latent space. Cosine similarity between old and new vectors is meaningless—they're not comparable. This is the most destructive kind of staleness because it corrupts all downstream operations, not just individual lookups.
Source data staleness. The user's behavior has changed, but their cached embedding hasn't updated to reflect it. A document got rewritten, but its vector still describes the old text. This is gradual, per-item corruption.
Semantic drift. The relationships in your latent space shift over time as new data comes in. Even without explicit retraining, the "correct" embedding for a given item is a moving target. This is the sneakiest one—your cache returns vectors that were perfectly accurate when computed, but are increasingly wrong relative to the current distribution of your data.
I've seen teams conflate all three and try one eviction policy for everything. That's a mistake. You need separate mechanisms because the failure modes are different.
The Naive Approach: TTLs and Why They Fail
Setting a time-to-live on embeddings is the default. Redis expires the key, your query pipeline recomputes, everyone moves on. It's simple, it's predictable, and it's mostly wrong.
The problem is that TTLs are blind. They don't know when your model version changed. They don't know which embeddings are actually degrading. They just count seconds.
I tested this at a fintech client in early 2025. We set a 24-hour TTL on user embeddings for a fraud detection system. The theory was fine—recompute user vectors daily, keep them fresh. What actually happened: the embedding model was retrained on an emergency basis after a detection regression, and the TTL kept serving old vectors for up to 24 hours because it had no concept of "model version."
Twenty-four hours of degraded fraud detection. In production. With real money at stake.
A TTL is fine as a backstop, but it should never be your primary invalidation mechanism. It's the "I don't know when this changes, so let's guess" strategy. You can do better.
Versioning: The Foundation You Need
The first thing to implement is explicit model versioning on every cache entry. This is non-negotiable.
Every embedding in your cache should carry metadata: which model version produced it, when it was computed, and what the hash of the source data was at computation time.
python
@dataclass
class CachedEmbedding:
vector: list[float]
model_version: str
computed_at: datetime
source_hash: str
hit_count: int
When you deploy a new embedding model, you don't need to kill the cache. You need to version the cache. Split it by model version—each version gets its own namespace or partition.
python
def get_embedding(cache: DistributedCache, item_id: str, model_version: str) -> list[float]:
key = f"emb:{model_version}:{item_id}"
cached = cache.get(key)
if cached is not None:
return cached.vector
vector = compute_embedding(item_id, model_version)
cache.set(key, CachedEmbedding(
vector=vector,
model_version=model_version,
computed_at=datetime.utcnow(),
source_hash=hash_source(item_id),
hit_count=0
))
return vector
This way, when you roll out model v4.2 while v4.1 is still serving traffic, you don't have a thrash where every request recomputes embeddings. Old requests hit the v4.1 partition, new requests hit v4.2, and you migrate traffic gradually.
When do you evacuate the old version? That's where the next piece comes in.
Dual-Write and Shadow Eviction
Here's the pattern I've settled on after testing several approaches with clients building recommender systems and RAG pipelines.
When you deploy a new model version, you don't eagerly recompute everything. That's wasteful—you might recompute 10 million embeddings and only serve 2 million of them. Instead, do dual-writes on new requests and shadow recomputation on cache hits.
python
def get_embedding_with_shadow(cache, item_id, active_version, shadow_version):
key = f"emb:{active_version}:{item_id}"
cached = cache.get(key)
if cached is None:
vector = compute_embedding(item_id, active_version)
cache.set(key, vector)
return vector
# Shadow: recompute with new version asynchronously
if should_shadow_recompute(cached):
async_recompute(item_id, shadow_version)
return cached.vector
The should_shadow_recompute check is a gating mechanism. You don't want to shadow-recompute every single hit—that defeats the purpose of caching. Instead, sample based on hit rate. Embeddings that are frequently accessed are worth fresh computation. Cold embeddings can wait.
This gives you a graceful transition. Hot items move to the new version within hours. Cold items migrate lazily. When the migration rate drops below a threshold—say, 95% of served embeddings are already on the new version—you evacuate the old partition entirely.
I ran this pattern at an e-commerce client in Q1 2026. We transitioned their product search embeddings to a new model and saw zero cache-miss spikes. The old partition was fully drained in 47 hours, and we deleted it without any service disruption.
Source-Aware Invalidation
Model versioning handles the "new model deployed" case. But what about individual items whose source data changes?
A document gets edited. A user's profile fields change. A product's description is rewritten. If their embeddings are cached, you're serving vectors that describe content that no longer exists.
The fix is source-hash tracking. Store a hash of the source data alongside the embedding. On each cache access (or on a periodic sweep), compare the stored hash with the current source hash.
python
def get_or_refresh(cache, item_id, source_data):
current_hash = hash_data(source_data)
cached = cache.get(f"emb:{item_id}")
if cached is None or cached.source_hash != current_hash:
vector = compute_embedding(source_data)
cache.set(f"emb:{item_id}", CachedEmbedding(
vector=vector,
source_hash=current_hash,
computed_at=datetime.utcnow()
))
return vector
return cached.vector
This is cheaper than it sounds. You're not recomputing embeddings on every request—you're just hashing the source data, which is microseconds. The hash comparison lets you know immediately whether the cached embedding is still valid.
But hashing on every request adds overhead you might not want. The alternative is event-driven invalidation: your system already emits events when documents change, when user profiles update, when products are modified. Subscribe to those events and invalidate the relevant cache keys.
python
# Event consumer for document updates
def on_document_updated(event):
doc_id = event.document_id
cache.delete(f"emb:doc:{doc_id}")
# Optionally recompute immediately
if event.high_priority:
vector = compute_embedding(fetch_document(doc_id))
cache.set(f"emb:doc:{doc_id}", CachedEmbedding(
vector=vector,
source_hash=hash_data(fetch_document(doc_id)),
computed_at=datetime.utcnow()
))
We built this into a legal-tech client's document retrieval system in mid-2025. They were editing contracts continuously, and their RAG pipeline was returning embeddings for pre-edit versions of clauses. Event-driven invalidation fixed it. Accuracy went from 71% to 94% on retrieval benchmarks.
Semantic Drift Detection
This is the hardest problem. Model version and source data staleness are discrete—you know when they happen. Semantic drift is continuous. Your embedding space is a moving map, and you're using an old edition.
The detection strategy I've converged on is distributional monitoring. Track the distribution of embeddings being served from cache versus embeddings being freshly computed. If they start diverging significantly, your cache is serving stale representations.
python
def monitor_drift(fresh_embeddings, cached_embeddings, threshold=0.1):
for dim in range(embedding_dim):
fresh_mean = np.mean([e[dim] for e in fresh_embeddings])
cached_mean = np.mean([e[dim] for e in cached_embeddings])
fresh_std = np.std([e[dim] for e in fresh_embeddings])
z_score = abs(fresh_mean - cached_mean) / (fresh_std + 1e-8)
if z_score > threshold:
alert(f"Semantic drift detected in dimension {dim}: z={z_score}")
This is a crude but effective early warning system. In practice, you'll want a few samples of fresh embeddings per partition per hour—enough to establish a baseline without hammering your compute.
At a healthcare AI client in late 2025, this caught drift from changes in their patient-triage model. The model wasn't retrained—their data distribution shifted as they onboarded new hospitals with different demographics. Embeddings computed two months prior were measurably off, and the drift monitor caught it three weeks before any human would have noticed downstream quality issues.
Distributed Cache Design for ML Serving
Now let's talk about the infrastructure side. You can't just run Redis for this. The access patterns are different from typical web caching.
For a distributed cache for ml serving, I've landed on a two-tier architecture:
Tier 1: Hot cache. In-memory, local to each serving node. Redis or even just a Python dict if you're single-node. Stores the hottest 5% of embeddings. Fast p99 access—under 1ms.
Tier 2: Warm cache. Distributed, shared across nodes. FAISS or pgvector for approximate nearest neighbor search, or Redis with vector extensions. Holds the long tail.
python
def get_embedding_multi_tier(hot_cache, warm_cache, item_id):
# Try hot cache first
hot_result = hot_cache.get(item_id)
if hot_result is not None:
return hot_result
# Fall back to warm cache
warm_result = warm_cache.get(item_id)
if warm_result is not None:
# Promote to hot cache
hot_cache.set(item_id, warm_result)
return warm_result
# Compute from scratch
vector = compute_embedding(item_id)
warm_cache.set(item_id, vector)
hot_cache.set(item_id, vector)
return vector
The eviction policies differ per tier. The hot cache evicts by LRU—you want recently accessed embeddings at hand. The warm cache evicts by staleness—you want the freshest vectors, even if they're less frequently used.
Don't conflate the two. LRU on a stale-correctness problem is how you end up serving garbage faster to more people.
Practical Eviction Policies, Ranked
Based on what I've seen working in production across our clients, here's my honest ranking of eviction strategies:
Best: Model-version partitioned cache with dual-write + shadow recomputation. Handles the most destructive staleness. Requires infrastructure investment but pays off in correctness.
Great: Event-driven invalidation for source changes. Precise, real-time, and cheap when you already have an event bus. The best cost-to-benefit ratio of anything on this list.
Good: TTL with intelligent duration. The "unknown unknowns" backstop. Set it to be longer than your average session but shorter than your model update cadence. For most use cases, 6-12 hours is reasonable.
Sketchy: Pure TTL. Works for demo systems. Falls apart when models retrain or data changes at scale.
Irresponsible: No eviction. I've seen production systems with a year-old embedding cache. The quality was catastrophic, and nobody knew until they ran an audit.
Cost Considerations
There's a reason people avoid aggressive eviction: computing embeddings is expensive. Modern embedding models like OpenAI's text-embedding-3-large or even open-source BGE-M3 take real compute per document. At scale, recomputing 10 million embeddings can cost thousands of dollars in inference time.
I've seen startups burn their entire GPU budget on embedding recomputation after a model upgrade. The fix is to be surgical. Don't recompute everything—recompute what's hot, what's changed, and what's about to go stale.
python
def scheduled_eviction(all_items, cache, model_version, budget_percent=20):
# Compute staleness score for each item
staleness_scores = []
for item in all_items:
score = compute_staleness_score(item, cache)
staleness_scores.append((score, item))
# Sort by staleness and evict the top budget_percent
staleness_scores.sort(reverse=True)
evict_count = int(len(staleness_scores) * budget_percent / 100)
for _, item in staleness_scores[:evict_count]:
cache.delete(f"emb:{model_version}:{item.id}")
This is a budget-aware approach that limits recompute cost while prioritizing the worst offenders. You'll never be perfectly fresh, but you'll be freshest where it matters.
The Monitoring Story
Whatever eviction policy you choose, you need to observe its effectiveness. Here's what I have every SIVARO client instrument:
Cache hit rate. Obvious. But drill down: hit rate by model version, by embedding type, by hot path vs. cold path.
Serving latency percentiles. p50, p95, p99. Watching these shift after an eviction policy change tells you if you're thrashing.
Staleness distribution. For a sample of served embeddings, how old are they? What model version are they from? This is the "health check" for your cache.
Source deviation rate. How often does the source hash mismatch the cached hash? If this is climbing, your cache is falling behind your data changes.
python
from prometheus_client import Histogram, Gauge
stale_embeddings = Gauge('cache_stale_embeddings', 'Embeddings with outdated source hashes')
embedding_age = Histogram('embedding_age_hours', 'Age of served embeddings', buckets=[1, 6, 24, 72, 168, 720])
I'd argue the monitoring matters more than the eviction policy itself. With good visibility, you'll know when your policy is failing before it costs you. Without it, you're flying blind and the first sign of trouble will be a user complaint or a collapsed metric.
When to Skip Caching Entirely
Contrarian take, but hear me out: some embedding workloads shouldn't be cached at all.
If your embedding computation is cheap—say, under 2ms and under $0.00001 per inference—and your traffic is moderate, caching adds complexity without meaningful benefit. The staleness problem disappears when there's no cache to go stale.
I ran the math for a client in 2024. Their embedding calls were 3ms and they were doing 200 requests per second. Caching would save them maybe 300ms of p99 latency and some GPU load. But they would have needed Redis, invalidation logic, and monitoring. All to save resources they had in abundance. Not worth it.
Caching pays for itself when one or more is true: your embedding computation is expensive, your latency requirements are tight, your model updates are infrequent, and your data changes are rare. If all four of those flip on you, caching becomes a liability, not an asset.
FAQ
Q: How do I know when to evict vs. recompute on the fly?
Recompute on the fly for hot paths where you need zero staleness and can afford the latency. Evict-and-recompute for cold paths where the cost of proactive computation isn't justified.
Q: What's the right TTL for embeddings?
Depends entirely on your model update cadence and data change rate. For stable models with slowly changing data, 24-72 hours is fine. For models that update weekly, your TTL should be under your update cadence—otherwise you're serving stale vectors for days.
Q: Is it better to use a smaller model for embeddings to make eviction cheaper?
Sometimes, but only if quality permits. I've seen teams use sentence-transformers/miniLM instead of larger models precisely so recomputation is cheap enough to do eagerly. Trade-off is embedding quality, but if your downstream tasks don't suffer, it's a win.
Q: Does FAISS handle stale embeddings?
No. FAISS is an ANN index, not a cache. If you're using FAISS for similarity search, you need to periodically rebuild or update the index with fresh embeddings. Same with pgvector and other vector stores.
Q: Can I use Redis for embedding caching?
Yes, Redis has vector similarity support since 2024. But it's still just a cache—you own the eviction logic. The patterns in this article work on any key-value store with TTL support.
Q: What happens if I evict an embedding that's referenced by other cached data?
This is why you need reference tracking. If you cache composite objects that embed vectors, you need to cascade invalidations. Worse, if you have a vector index that references embeddings by ID, evicting the source embedding doesn't remove the index entry. You need explicit index maintenance.
Q: How do I handle embeddings that are shared across models?
Use multi-tenancy in your cache keys. Same item, different model version → different keys. Never overwrite one model's embedding with another's. The version prefix in your key is the safety mechanism.
Conclusion
How to evict stale embeddings from cache comes down to three insights: know what version of the model produced each vector, know when the source data changes, and watch the distribution drift. Everything else is implementation details.
You don't need to build a perfect system. You need to build one that fails in the right direction—preferring slight over-computation to serving stale semantics. The cost of recomputation is predictable. The cost of staleness isn't: it shows up as degraded recommendations, broken retrieval, and silent quality collapse months later.
Version your embeddings. Invalidate on source change. Monitor the distribution. Evict the stale ones. Do that and you'll have a cache that's actually serving your users—not one that's quietly serving the ghost of your model's past.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.