The First Principles of Model Routing: Stop Treating LLMs Like Oracles
I still remember the exact moment I realized everything we were doing with LLMs was backward.
January 2025. My team at SIVARO was building a customer-facing document analyzer. We'd trained a custom fine-tune. Cost us six figures. It was supposed to handle everything — contract review, invoice parsing, compliance checks.
It didn't.
The model was great at identifying boilerplate clauses but hallucinated jurisdiction details. It handled contract structure beautifully but couldn't tell you the dollar amount in a purchase order with any reliability. We were forcing one model to be good at everything instead of asking the obvious question:
What is this specific request, and who should handle it?
That's the first principles of model routing. Not a router in the networking sense. Not a load balancer. A decision system that asks: Given this input, which model (or system, or human) has the highest probability of producing the correct output, at acceptable latency, within budget?
Most people think model routing is about cost optimization — pick the cheap model when you can, the expensive one when you must. They're wrong. Cost is the easiest dimension. The hard part is correctness, latency, safety, and traceability — all interacting simultaneously.
Here's what we've learned building routing systems handling 200K events/second across 40+ model endpoints. These aren't theories. These are scars.
Why Routing Is a Distributed Systems Problem — Not an ML Problem
If you've been following the space, you've seen the same graph I have: model capabilities are plateauing at the frontier while model diversity explodes. GPT-4o, Claude 3.5 Opus, Gemini 2.0, Llama 4, Mistral Large, DeepSeek-R1 — each has a different failure mode.
The industry is converging on a dirty secret that no vendor wants to admit: no single model is universally best. Not even close.
This isn't a model problem. It's a routing problem. And routing at scale is a distributed systems problem.
In July 2025, when we at SIVARO started designing our production routing layer, we benchmarked 12 models across 20K prompts. The variance was brutal. Model A scored 94% on structured data extraction but failed 40% of the time on ambiguous queries. Model B crushed creative tasks but hallucinated numbers constantly. The optimal routing policy wasn't "use model A" or "use model B" — it was send each query to whichever model has the highest probability of getting it right.
That's a distributed systems challenge described well in Your Agent is a Distributed System (and fails like one). The failure modes aren't "model accuracy" — they're timeouts, partial failures, race conditions, and coordination overhead.
Think about it. Your routing service has to:
- Inspect the input (latency cost)
- Query a model catalog (network cost)
- Decide which model to use (compute cost)
- Call the chosen model (inference cost)
- Handle the response (timeout/retry logic)
- Log everything for observability (storage cost)
Each step adds latency and failure surface area. Ignore the distributed systems reality of your router and you'll build something that works in demo but shatters at load.
First Principles: The Four Dimensions of Routing Decisions
I've distilled our routing logic down to four orthogonal dimensions. Every routing decision optimizes over these:
1. Capability Suitability (The "Can It?" Dimension)
Does the model have the knowledge and reasoning ability to answer correctly?
This isn't trivial. We tested GPT-4o on legal jurisdiction questions — 87% accuracy. Claude 3.5 Opus on the same set? 79%. But swap the domain to creative brand naming, and those numbers invert.
The trap: Assuming capability is static. Models get updated. New models release. Your routing policy must adapt. We saw a 12% accuracy drop on one routing path when Anthropic silently updated Claude 3.5 Opus in February 2026. Our static ruleset broke overnight.
The fix: Continuous probing. We maintain a held-out eval set of 500 prompts per domain, re-scored daily against every available model. The routing matrix updates automatically.
2. Cost Efficiency (The "Should It?" Dimension)
This is where most people start and stop. It's the easiest dimension to measure.
Token cost is obvious. What's not obvious: the cost of wrong answers. A $0.03 query that produces a hallucinated legal citation costs you $300 in remediation later. Sometimes paying for GPT-4o-inference is cheap.
At SIVARO, we track a metric we call cost-per-correct-output (CPCO). Formula: (total inference cost) / (number of queries with acceptable response). In February 2026, we found our "always use GPT-4o-mini" route had CPCO of $0.08 because it generated wrong answers 18% of the time requiring re-routing. Our "route to Llama 4 for structured, GPT-4o for ambiguous" policy had CPCO of $0.04.
Half the cost. The expensive model saved money.
3. Latency Budget (The "When?" Dimension)
Every system has a latency SLO. If your call center needs responses in under 200ms, you can't route to a 5-second-thinking-time model.
But here's the counterintuitive part: routing itself adds latency. We initially built a router that analyzed each prompt with a lightweight classifier model before routing. The classifier took 150ms. That left 50ms for the actual model call. Impossible.
We reversed the order: route based on fast heuristics first (regex, keyword matching, prompt length), escalate to model-based classification only for ambiguous cases. THE SIGNAL: What matters in distributed systems | #4 calls this "layered decisioning" — each layer trades depth for speed.
Our current system routes 60% of traffic on heuristic rules alone. Latency: 12ms. The remaining 40% gets model-classified and routed. Average end-to-end: 180ms.
4. Safety Alignment (The "Should It Not?" Dimension)
This is the one everyone forgets until disaster strikes.
Different models have different safety profiles. Some refuse medical advice aggressively. Some happily generate financial projections. Some have contextual awareness of jurisdiction and regulation.
In October 2025, a client of ours was using a "cheap route" for customer support that accidentally routed a privacy-violating query to a model with no safety guardrails. The model produced a response that violated GDPR. The client got fined €20M.
We now route more conservatively for regulated domains. Safety-critical queries go to models we've explicitly audited — even if they're more expensive. The routing matrix has a "safety clearance level" dimension that overrides all others when triggered.
The Architecture: What We Actually Built
Here's our production routing architecture. It's not theoretical — it's running today, processing 200K events/sec at SIVARO.
python
# Simplified routing decision engine (production, July 2026)
from dataclasses import dataclass
from enum import Enum
import time
class RouteDimension(Enum):
CAPABILITY = "capability_score"
COST = "cost_per_token"
LATENCY = "p50_latency_ms"
SAFETY = "safety_clearance"
@dataclass
class RoutingDecision:
model_id: str
confidence: float
dimensions_satisfied: list[RouteDimension]
decision_time_ms: float
fallback_chain: list[str]
def route_query(query: str, context: dict) -> RoutingDecision:
start = time.perf_counter()
# Layer 1: Heuristic fast-path (12ms)
heuristics = fast_pattern_classifier(query)
if heuristics.is_unambiguous:
model = heuristics.recommended_model
return RoutingDecision(
model_id=model,
confidence=0.95,
dimensions_satisfied=[RouteDimension.CAPABILITY, RouteDimension.LATENCY],
decision_time_ms=(time.perf_counter() - start) * 1000,
fallback_chain=[model, "default-gpt4o"]
)
# Layer 2: Model-based classification (150ms)
classifier_result = model_classifier.classify(query, context)
model_scores = score_all_models(query, classifier_result.domain)
best_model = max(model_scores, key=lambda m: m.composite_score(
safety=context.get('safety_level', 0),
latency_budget=context.get('latency_ms', 2000),
cost_max=context.get('max_cost_per_query', 0.10)
))
return RoutingDecision(
model_id=best_model.id,
confidence=best_model.confidence,
dimensions_satisfied=[RouteDimension.CAPABILITY, RouteDimension.COST,
RouteDimension.SAFETY, RouteDimension.LATENCY],
decision_time_ms=(time.perf_counter() - start) * 1000,
fallback_chain=[best_model.id, "default-gpt4o", "fallback-claude35"]
)
The key insight here is the fallback chain. Every routing decision includes a chain of models to try if the primary choice fails. This handles the reality that models go down, degrade, or return errors.
Caching Is the Unsexy Hero Nobody Talks About
Everyone wants to talk about routing policies. Nobody wants to talk about caching.
In our production system, 34% of all queries are semantically identical to something we've already answered. For enterprise systems with contract analysis or compliance review, that number hits 52%.
We built a semantic cache that stores (query embedding, response, model_used, confidence). When a new query comes in, we check cache first. If the embedding similarity is above 0.97, we return the cached response.
This isn't a routing decision — it circumvents routing entirely.
Caching for Agentic Java Systems: Internal, Distributed, ... covers this excellently. The caching layer needs to be distributed, consistent, and fast. We use Redis with HNSW indexing for approximate nearest neighbor search. Cache hit latency: 5ms. Cache miss latency: +2ms (negligible compared to inference).
Our cache saves us $18K/month in inference costs. But more importantly, it eliminates routing overhead for repeat queries. When you cache, you skip the entire routing decision tree.
Multi-Agent Systems: Routing Gets Harder
If you think routing is hard for single-query systems, wait until you see multi-agent architectures.
I've been watching the agent space closely. What I see are companies building "teams" of agents that communicate, delegate, and escalate. Sounds elegant. In practice, it's a distributed systems nightmare.
Multi-Agent Systems Have a Distributed Systems Problem nails this. Each agent conversation becomes a series of routing decisions. Agent A routes to Agent B, which routes to a model, which returns results to Agent B, which routes back to Agent A. The routing graph becomes cyclic, feedback loops emerge, and latency compounds.
We tested a three-agent system for document review in April 2026. The average response time? 8.7 seconds. End-to-end routing decisions accounted for 2.3 seconds of that. The agents kept re-routing to each other, generating coordination overhead that dwarfed actual model inference time.
Our fix: We made routing decisions explicit and logged them as a directed acyclic graph (DAG). No agent can route back to an upstream agent. Every routing decision includes a max-depth parameter. If the routing graph exceeds depth 5, it escalates to a human.
json
{
"agent_trace": [
{"node": "classifier", "action": "route", "to": "legal_extractor", "depth": 1},
{"node": "legal_extractor", "action": "route", "to": "claude35", "depth": 2},
{"node": "claude35", "action": "response", "to": "legal_extractor", "depth": 2},
{"node": "legal_extractor", "action": "aggregate", "to": "classifier", "depth": 3},
{"node": "classifier", "action": "complete", "depth": 3}
],
"max_depth_allowed": 5,
"total_routing_latency_ms": 340,
"total_model_latency_ms": 4100
}
The Routing Matrix: Your Single Source of Truth
Here's the concrete artifact you need to build. Call it the routing matrix. It's a table — one row per domain or query type, one column per model, cell value is the quality score (0-1).
| GPT-4o | Claude 3.5 | Llama 4 | Mistral L | DeepSeek-R1
------------|--------|------------|---------|-----------|------------
Legal | 0.91 | 0.88 | 0.72 | 0.65 | 0.83
Medical | 0.87 | 0.91 | 0.68 | 0.71 | 0.79
Creative | 0.78 | 0.83 | 0.90 | 0.88 | 0.76
Structured | 0.94 | 0.79 | 0.92 | 0.95 | 0.81
Ambiguous | 0.96 | 0.93 | 0.74 | 0.69 | 0.88
We update this matrix weekly. Every Monday at 2 AM UTC, our eval pipeline re-scores every model against every domain with 500 held-out queries.
The matrix is deployed as a config file that the routing engine hot-reloads. No code changes needed when a model improves or degrades. Every System is a Log: Avoiding coordination in distributed ... influenced this design — the routing config is a log of model evaluations, not a mutable state that requires consensus.
The Three Routing Strategies That Actually Work
After 18 months of iterating, we've settled on three routing strategies that cover 95% of use cases. Everything else is noise.
Strategy 1: Threshold-Based Routing (Simplest)
Set a quality threshold. Use the cheapest model that meets it.
python
def threshold_route(query: str, min_quality: float = 0.85):
models = sorted(available_models, key=lambda m: m.cost)
for model in models:
score = routing_matrix.get(query.domain, model)
if score >= min_quality:
return model
return fallback_model # Usually GPT-4o
Works great when quality requirements are clear and stable. Fails when domains overlap or thresholds need to change per query.
Strategy 2: Expected Value Routing (Best)
This is what we use in production. Calculate the expected value of each model call: (probability of correct answer * value of correct answer) - (probability of wrong answer * cost of wrong answer) - (cost of model call).
python
def expected_value_route(query: str, context: dict) -> str:
best_model = None
best_ev = -float('inf')
value_of_correct = context.get('value_of_correct', 1.0)
cost_of_wrong = context.get('cost_of_wrong', 0.5)
for model in available_models:
prob_correct = routing_matrix.get(query.domain, model)
prob_wrong = 1 - prob_correct
ev = (prob_correct * value_of_correct) - (prob_wrong * cost_of_wrong) - model.cost
if ev > best_ev:
best_ev = ev
best_model = model
return best_model
This catches the case where an expensive model's higher accuracy justifies the cost. It's also how you avoid the "cheap model paradox" where you save 2 cents per query but lose 10 cents in wrong answers.
Strategy 3: Ensemble Routing (For High-Stakes)
Run multiple models in parallel, compare outputs, pick the most consistent one. We use this only for compliance and legal queries.
python
import asyncio
async def ensemble_route(query: str, models: list[str]) -> str:
responses = await asyncio.gather(
*[call_model(query, model) for model in models]
)
# Simple voting: pick the response closest to the centroid
embeddings = [embed(r) for r in responses]
centroid = average(embeddings)
distances = [cosine_distance(emb, centroid) for emb in embeddings]
best_idx = argmin(distances)
return responses[best_idx]
Cost: 2.5x-4x more expensive.
Benefit: 99.2% accuracy on our compliance benchmark vs 94% for best single model.
When to use: Financial reporting, medical diagnosis, legal filings. Never for content generation or customer support.
The Reality Check: Where Routing Breaks
I've been talking about routing like it's the solution to everything. It's not. Here's where it breaks.
Routing fails when the router itself is wrong. If your query classifier misidentifies the domain, you route to the wrong model. Period. We see this 3-5% of the time. The fix is to include a "confidence threshold" — if the router's confidence is below 0.7, fall back to a general-purpose frontier model.
Routing fails on novelty. When you receive a query type you've never seen before, your routing matrix has no data. Our system defaults to "most expensive, most capable" for any query with <10 historical examples. It's the only safe default.
Routing amplifies latency variance. We saw a case where the router chose Llama 4 (thinking it was fast) but that specific endpoint was experiencing cold starts. The 99th percentile latency jumped from 400ms to 3200ms. We now include endpoint health metrics (p50, p99 latency, error rate) as routing factors.
Distributed systems concepts like circuit breakers and bulkheads apply here. Each model endpoint should have a circuit breaker. If error rates exceed 5% in a window, that endpoint is removed from routing decisions automatically.
FAQ: Model Routing First Principles
Q: When shouldn't I use model routing?
A: When your use case is homogeneous. If you only ever need one capability (e.g., summarization) and one model handles it well enough, routing adds complexity for zero benefit. We see teams implement routing for trivial systems because it sounds sophisticated. Don't.
Q: How often should I update the routing matrix?
A: Weekly minimum. Daily if you can afford the eval compute. Models get updated silently. We caught a Claude 3.5 Opus regression within 48 hours because our Monday eval flagged a 3% accuracy drop on legal queries.
Q: Can I use the routing model itself as the classifier?
A: Yes, but it's circular. If you use GPT-4o to decide whether to call GPT-4o, you're paying GPT-4o cost for every query. Better to use a lightweight classifier (Mistral 7B or even a fine-tuned BERT) for routing decisions.
Q: What about open-source vs proprietary models?
A: We route 60% of traffic to open-source models (Llama 4, Mistral). They're cheaper, but their failure modes are different. Open-source models tend to be worse at safety alignment. Our routing policy routes any query containing protected health information (PHI) or personally identifiable information (PII) to proprietary models with documented safety guarantees.
Q: How do you handle model versioning?
A: Every model endpoint has a version tag. The routing matrix stores scores per (model, version). When a new version deploys, it hits a shadow traffic route for 24 hours where it receives 1% of traffic but scores aren't published. After that, it enters the active matrix.
Q: What's the most common routing mistake?
A: Optimizing for cost before establishing correctness baselines. Every team I've seen do this (including mine, early on) ends up with a system that's cheap but wrong 20% of the time. Fix the correctness floor first, then optimize cost.
Q: Do you route to humans too?
A: Yes. When no model achieves an expected value above 0.5 (on our normalized scale), we escalate to a human operator. This happens for about 0.3% of queries — typically novel edge cases or complex multi-jurisdiction legal questions.
A Final Principle: Routing Is a Product Decision, Not an Infrastructure One
I've watched teams spend months building perfect routing infrastructure only to realize they didn't know what "good" looked like.
The hardest question isn't "which model should handle this?" — it's "what does success look like for this query?" That's a product question. It requires understanding user intent, acceptable error rates, latency expectations, and safety boundaries.
At SIVARO, we now start every new routing problem with a product brief. Not a technical architecture. The product brief defines the dimensions of success. Only then do we build the routing matrix.
The first principles of model routing are simple in theory: match the query to the model with the highest probability of success. In practice, it's a distributed systems problem, a product strategy problem, and an economics problem — all tangled together.
Build your router with that reality in mind, or don't build it at all.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.