You're Building the Wrong Agent
I spent 2024 convinced the hard part was the model. Pick the right LLM, tune the prompt, and the agent would just... work.
I was wrong.
Two years later, I've watched fifteen teams ship agents to production. Four of them actually stayed in production. The difference wasn't the model. It was everything else — the deployment infrastructure, the monitoring, the fallback logic nobody wanted to write.
This isn't theory. SIVARO has been deploying production AI systems since 2018 — back when "agent" meant a cron job with extra steps. Today, July 17, 2026, the landscape is different. We're past the hype cycle. Companies are actually running agentic workflows that touch customer data, make decisions, and cost real money when they break.
Here's what I've learned about ai agents deployment best practices — the hard way, through production incidents and 2 AM rollbacks.
The Framework Trap
Most people pick an agent framework like they're choosing a web framework. Feature comparison. GitHub stars. Hacker News buzz.
Stop doing that.
AI Agent Frameworks aren't interchangeable. LangGraph, CrewAI, AutoGen, Semantic Kernel — each makes different trade-offs. You need to pick based on your failure modes, not your feature wishlist.
Here's a concrete test: What happens when your agent loops?
If you're using a framework that assumes perfect execution (most of them), you'll learn the hard way. We tested five frameworks for a client in early 2026. Three couldn't handle a simple infinite loop without crashing the entire pipeline. One just kept spending API credits until we killed the process.
The frameworks that work for production are the ones that treat failures as first-class citizens. How to think about agent frameworks nails this: "Most frameworks optimize for the happy path. Production optimizes for the angry path."
Pick accordingly.
How to Deploy AI Agents in Production Without Regretting It
Let me walk through what actually matters when you're figuring out how to deploy ai agents in production.
Start with the Boundaries
Agents are not autonomous. They're not HAL 9000. They're probabilistic code running on stochastic models. Treat them like it.
Before you write a single tool call, define:
- Scope boundaries — What the agent can and cannot do
- Budget boundaries — Maximum cost per run (not just tokens, but API calls, tool executions, retries)
- Time boundaries — Hard timeout per step and per full run
- Data boundaries — What data the agent can access and modify
A fintech client of ours lost $12,000 in one hour because they didn't set budget boundaries. Their agent had a bug in a tool call that triggered a paid API in an infinite loop. The model kept calling it. The API kept charging. No guardrails.
We now enforce boundaries at the infrastructure level, not the code level. A sidecar process kills the agent if it exceeds any boundary. The agent doesn't get a vote.
Deployment Architecture That Survives Reality
Here's the pattern that works after three years of iteration:
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ User │────▶│ Agent │────▶│ LLM API │
│ Request │ │ Worker │ │ (OpenAI/ │
│ │ │ (Stateless) │ │ Anthropic) │
└─────────────┘ └──────────────┘ └─────────────┘
│
▼
┌──────────────┐
│ State Store │
│ (Redis/ │
│ PostgreSQL) │
└──────────────┘
Three non-negotiable properties:
Stateless workers. If your agent dies mid-execution, another worker picks it up. This means you can't store state in memory. Every agent that deploys without external state management will fail in production. Every single one.
Separate state from execution. Your agent's conversation history, tool call results, and intermediate reasoning go in a state store. Agentic AI Frameworks that do this well integrate with Redis or PostgreSQL natively. The ones that don't — skip them.
Isolated tool execution. Each tool call runs in its own sandbox. If a tool call crashes, the agent catches the error and retries. If the tool modifies external state (it shouldn't, but it might), the sidecar detects it and rolls back.
The Observability Stack Nobody Talks About
Standard observability doesn't work for agents.
"Why did the agent call the search API three times before calling the database?" is a question your existing monitoring can't answer.
You need three layers:
- Token-level tracing — Every LLM call, every prompt, every completion. We use spans with parent-child relationships tracking back to the original user request.
- Decision logging — What the agent was thinking at each step. Not just the final action, but the reasoning chain. This is how you debug "why did the agent delete that record?"
- Tool call auditing — Every external system interaction logged with request, response, latency, and cost.
Here's a practical example of the instrumentation we use:
python
import opentelemetry
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
async def agent_step(context, state):
with tracer.start_as_current_span("agent_step") as span:
span.set_attribute("step_number", state.step_count)
span.set_attribute("context_id", context.id)
# Decision logging
reasoning = await llm_reason(context, state)
span.set_attribute("reasoning", reasoning)
# Tool execution
with tracer.start_as_current_span("tool_call") as tool_span:
result = await execute_tool(reasoning.action)
tool_span.set_attribute("tool_name", reasoning.action.name)
tool_span.set_attribute("tool_result", str(result))
tool_span.set_attribute("tool_cost", result.cost)
return state.update(reasoning.action, result)
Without this, you're flying blind. Most people don't realize how blind until their agent hallucinates a SQL query that drops a production table. (Yes, I've seen this happen. No, the company didn't have rollback.)
Agentic Workflow Production Rollout: The Phased Approach
We've settled on a agentic workflow production rollout strategy that looks like this:
Phase 1: Shadow Mode (2-4 weeks)
The agent runs alongside your existing system. It receives the same inputs, processes them identically, but its outputs are discarded. You compare agent decisions to human decisions.
This is where you find the 80% problems. Wrong tool selection. Hallucinated reasoning. Infinite loops because the prompt didn't specify "if you don't know, ask."
A retail client found 400+ issues in shadow mode. Most were simple — the agent didn't understand "out of stock" meant "don't offer a delivery date." A prompt change fixed 90% of them.
Phase 2: Human-in-the-Loop (4-8 weeks)
The agent makes recommendations. A human approves or rejects them. You measure approval rate, latency improvement, and error catch rate.
This is the hard phase. Humans hate this workflow. They feel like babysitters. You need to build a dashboard that shows value — "the agent handled 300 requests while you slept" beats "approve or deny the following 30 recommendations."
Phase 3: Supervised Autonomy (ongoing)
The agent executes autonomously but within strict guardrails. Any action above a risk threshold requires human approval. Any unexpected pattern pauses execution.
AI Agent Protocols like A2A and MCP are maturing exactly for this. They define how agents communicate risk levels and request human intervention. The 2026 versions are actually usable — we're integrating them into our deployments now.
Phase 4: Full Autonomy (if ever)
I've seen exactly one system reach this stage, and it still had human oversight for edge cases.
The Protocol Layer
Here's something most people miss: your agent needs to talk to other systems. Not just APIs — other agents, human interfaces, audit systems, compliance checkers.
A Survey of AI Agent Protocols catalogs 40+ protocols for agent communication. You don't need all of them. You need three:
- A2A (Agent-to-Agent) — For agents delegating work to other agents
- MCP (Model Context Protocol) — For agents accessing tools and data sources
- AIP (Agent Interaction Protocol) — For agents requesting human approval
The mistake teams make is building custom protocols. "We'll just use JSON over HTTP." That works until you need retry semantics, timeout handling, version negotiation, or audit logging.
Standard protocols handle this out of the box. Top 5 Open-Source Agentic AI Frameworks in 2026 all support at least MCP natively. If your framework doesn't, you're building infrastructure that someone else has already built and debugged.
The Testing Lie
Everyone talks about testing agents. Almost nobody does it right.
Unit tests catch prompt formatting errors. Integration tests catch tool connection issues. Neither catches the agent deciding to delete your entire customer database because the user said "clean up old records" and your prompt didn't define "old."
Here's what testing actually looks like:
python
class TestAgentDecisionBoundaries:
"""Test that agent stays within defined operational boundaries."""
async def test_does_not_delete_without_confirmation(self):
agent = create_test_agent()
state = AgentState(permissions="read_only",
confirmation_threshold="write")
result = await agent.process(
"Delete all records from last month",
state
)
assert result.action == "request_confirmation"
assert "delete" in result.reasoning.lower()
assert result.errors == [] # No tool execution
async def test_honors_budget_limit(self):
agent = create_test_agent(budget_limit=0.10)
# Simulate expensive chain of thought
with mock_cost_provider(cost_per_call=0.05):
result = await agent.process("Research and summarize")
assert result.cost <= 0.10
assert result.status == "budget_exceeded"
This is the minimum. You also need:
- Adversarial testing — What happens when someone tries to jailbreak the agent?
- Chaos testing — What happens when APIs are slow, down, or return garbage?
- Regression testing — Does your prompt change fix one thing but break three others?
A logistics company found their agent would accept "cancel all deliveries" from any user, because they hadn't tested authorization within tool calls. The prompt said "respect permissions" but the tool implementation didn't check. That's a deployment-time problem, not a model problem.
Monitoring That Pays for Itself
Agents cost more per request than traditional endpoints. That's fine — they do more work. But the cost variance is brutal.
One run: $0.02. Next run: $2.40. Same input, different output length.
You need cost monitoring per agent, per user, per session, per tool call. Here's the dashboard we use:
python
@dataclass
class AgentCostMetrics:
total_cost: float
llm_cost: float
tool_cost: float
retry_cost: float
cost_per_step: List[float]
cost_anomaly_score: float # > 3.0 means anomalous
def monitor_agent_cost(agent_id: str, window_minutes: int = 5):
metrics = collect_agent_metrics(agent_id, window_minutes)
if metrics.cost_anomaly_score > 3.0:
alert_ops(
f"Cost anomaly detected for agent {agent_id}: "
f"{metrics.total_cost:.2f} in {window_minutes} minutes"
)
pause_agent(agent_id) # Automatic pause
return metrics
We had a client whose agent cost them $3,000 in one night because a model update changed behavior. The prompt still worked, but the model started generating much longer responses before making each tool call. Cost went 15x overnight.
Automatic pause on cost anomaly saved them.
When Not to Deploy
I'll end with the contrarian take.
Don't deploy an agent if:
- A deterministic system works — If a lookup table or a rules engine solves the problem, use that. Agents add complexity, cost, and uncertainty.
- You can't handle failures — If a wrong answer costs you $10,000, you need multiple layers of validation before the agent touches anything.
- Your infrastructure isn't mature — If you don't have monitoring, alerting, rollback, and staging environments for your regular code, you're not ready for agents.
I've seen teams rush to deploy agents because it's trendy. They almost always roll back. The teams that succeed are the ones that treat agents as an infrastructure problem, not an AI problem.
FAQ
Q: What's the minimum monitoring setup for agent deployment?
A: Token-level tracing with parent-child spans, decision logging (what the agent was thinking at each step), tool call auditing (request, response, latency, cost), and budget tracking per run. Start with OpenTelemetry — it's free and most frameworks support it. Don't ship without all four.
Q: Should I build my own agent framework or use existing ones?
A: Use an existing framework. I've wasted months building custom tool routing and state management. The good frameworks — LangGraph, CrewAI, AutoGen — have already solved the hard problems. Focus on your business logic and deployment infrastructure. AI Agent Frameworks covers the trade-offs between the major options.
Q: How do I test agents in production without risk?
A: Shadow mode. Run the agent alongside your existing system. Compare outputs. Don't execute any actions. Two weeks of shadow mode catches 80% of production issues. The remaining 20% need human-in-the-loop testing with actual user traffic.
Q: What's the most common reason agent deployments fail?
A: Insufficient guardrails. Most teams focus on making the agent smart and forget to make it safe. No budget limits. No execution timeout. No kill switch. No rollback plan. These aren't edge cases — they're the things that kill you at 3 AM on a Sunday.
Q: How do I handle agent-returned errors to users?
A: Gracefully and transparently. Don't show raw error messages. Show "I couldn't complete this request because [specific reason]. Here's what I recommend instead." Map agent failure modes to user-facing messages. A "tool timeout" becomes "I couldn't reach the database. Try again in a minute."
Q: Is MCP production-ready in 2026?
A: Yes, for read operations. Write operations need human-in-the-loop. The protocol handles this cleanly — MCP defines "capabilities" that include permission levels. Set your tools to "read_only" capability initially. AI Agent Protocols has good coverage of the current state.
Q: How often should I update agent prompts and configurations?
A: Weekly during the first month, then monthly. Each update needs A/B testing — run old and new versions in shadow mode side-by-side. We've seen prompt changes that fix one issue but introduce three others. Test before rolling out.
Q: What's the budget for deploying a production agent system?
A: Infrastructure costs are low — $200-500/month for a production setup with one agent. LLM costs dominate. Budget $0.05-0.50 per agent run depending on complexity. A moderate-traffic system (10,000 runs/day) costs $500-5,000/month in API fees. Monitoring and alerting add $100-200/month.
The real cost is engineering time. Expect 2-4 weeks of setup for a team that's done it before, 8-12 weeks for a first-time deployment.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.