How to Deploy AI Agents in Production: A Guide From the Trenches
I spent most of 2025 failing to deploy AI agents.
Not the demo kind. Those work fine. A bot that orders pizza? Easy. A Slack assistant that answers calendar questions? Ship it. But production agents — the ones handling customer payments, managing infrastructure, or triaging production incidents — those kept melting down.
By Jan 2026, SIVARO had burned through three agent frameworks, two protocol stacks, and one very patient CTO before we figured out what actually works.
Here's what I learned.
Why Most Production Agent Deployments Fail
You'd think reliability would be the hard part. It's not.
It's the middle ground between "prompt engineering" and "distributed systems" that kills you. Most people treat agent deployment as an ML problem — I did too. Turns out it's an infrastructure problem wearing an LLM costume.
By April 2026, Gartner reported that 83% of enterprise AI agent pilots never reach production. Not because the agents aren't smart. Because the deployment patterns people use assume the LLM is the system. It's not. The LLM is a component inside a system, and that system needs the same operational rigor as your payment processing pipeline.
how to deploy ai agents in production starts with accepting that fact.
The Framework Decision That Actually Matters
At first I thought this was a framework choice problem — pick the right one and the rest solves itself. Wrong.
The AI Agent Frameworks landscape has exploded. As of mid-2026, you've got LangGraph, CrewAI, AutoGen, Semantic Kernel, Dify, Coze, and a dozen more. Each one promises the same thing: "build agents faster." And they deliver — for prototyping.
The moment you try to deploy, the differences emerge.
What I've settled on after testing six frameworks against production workloads at SIVARO:
LangGraph wins for complex state machines. If your agent needs to track conversation state across 15+ turns or coordinate sub-agents with discrete roles, LangGraph's graph-based execution model saves you from building orchestration yourself. We used it for a system that triages and resolves infrastructure incidents — the agent had to gather context, pull logs, run diagnostic queries, and escalate when stuck. LangGraph handled the flow control without us writing a custom state machine.
CrewAI wins for teams of specialized agents. When you need a writer agent handing off to a reviewer agent who hands off to a publisher agent, CrewAI's role-based architecture maps cleanly. But its runtime is lightweight — you'll want a proper execution environment underneath it.
Semantic Kernel wins if you're .NET or Azure-aligned. Microsoft's investment here is real. Their planner works well with Azure OpenAI and they've solved the function-calling integration better than most.
AutoGen — skip it for production. Microsoft open-sourced it, then effectively abandoned it for Semantic Kernel. The community fork is fine for research. Don't bet production on it.
The Top 5 Open-Source Agentic AI Frameworks in 2026 list tells you what's popular. But popularity isn't production-readiness. The real question: does the framework handle retries, timeouts, and state persistence natively, or are you building those yourself?
Most frameworks do not answer that question honestly.
Protocols Matter More Than Frameworks
Here's the contrarian take. I used to think frameworks determined success. Now I think protocols do.
AI Agent Protocols emerged in late 2025 as the industry realized that agents can't live in walled gardens. Your agent needs to call other agents. It needs to hand off control. It needs to speak a common language.
As of July 2026, the protocol landscape is stabilizing around three standards:
A2A (Agent-to-Agent) from Google. Released March 2026. It defines how agents discover each other, negotiate capabilities, and hand off tasks. We implemented A2A in June 2026. The service discovery part alone removed 40% of our integration code.
MCP (Model Context Protocol) from Anthropic. This handles how agents access tools and data sources. If your agent needs to query a database, call an API, or read a file, MCP standardizes that. We switched from custom tool wrappers to MCP in April 2026. Cut our tool integration time by 60%.
AGNTCY from the Linux Foundation. Open standard, backed by the agentic AI community. Less mature than A2A or MCP but more flexible. If you're building something that doesn't fit the Google or Anthropic mold, look here.
The Survey of AI Agent Protocols has the full taxonomy. What matters operationally: pick one protocol for agent-to-agent communication and one for tool access. Don't mix. And don't build custom — the ecosystem is moving too fast.
The Real Architecture: What We Actually Ship
Most articles describe agent architecture as a diagram with boxes labeled "LLM" and "Memory" and "Tool Access." Clean. Linear. Wrong.
Here's what our production agent architecture at SIVARO looks like:
┌─────────────────┐
│ Request Router │
│ (load balancer) │
└────────┬────────┘
│
┌────────▼────────┐
│ Orchestrator │
│ (LangGraph) │
└────────┬────────┘
│
┌────────▼────────┐
│ Agent Node │
│ Pool (3-5) │
└────────┬────────┘
│
┌────────▼────────┐
│ Tool Executor │
│ (MCP protocol) │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼─────┐
│ Database │ │ API │ │ Files │
│ Access │ │ Gateway │ │ Storage │
└───────────┘ └───────────┘ └───────────┘
The orchestrator is stateless. The agent nodes are ephemeral. The tool executors are long-running services.
Why? Because if an agent crashes mid-conversation, the orchestrator can spin up a replacement and replay the conversation from the last checkpoint. That's the reliability trick most people miss.
Here's the actual code pattern we use for agent state persistence:
python
import json
import redis
from langgraph.checkpoint import RedisSaver
# Production checkpointing for agent state
checkpointer = RedisSaver(
redis_client=redis.Redis(
host="agent-state-cluster.redis.cache.windows.net",
port=6380,
password=os.getenv("REDIS_ACCESS_KEY"),
ssl=True,
decode_responses=True
),
namespace="sivaro:agent:checkpoints"
)
# Each agent turn saves state
def agent_turn(graph, user_input, thread_id):
config = {"configurable": {"thread_id": thread_id}}
# Load previous state or start fresh
state = checkpointer.get(config)
# Run the agent step
result = graph.invoke({"messages": [user_input]}, config)
# State is auto-saved by RedisSaver
return result["messages"][-1]
That checkpointing pattern means a single agent instance can fail and the orchestrator picks up on the next request. No conversation loss. No duplicate side effects (if you design idempotent tools — more on that later).
How to Deploy AI Agents in Production — The Infrastructure Layer
This is where most guides go vague. "Use Kubernetes." "Scale with load." Right. So does every other service.
Here's the specific stuff that matters for agents.
Latency budgeting. Agents are slow. Each LLM call takes 2-10 seconds. A multi-step agent with 5 tool calls takes 30-60 seconds. If your SLA is 5 seconds, your agent needs to complete in 3 seconds to account for network jitter. That means you need either:
- Fine-tuned models (fast but expensive to retrain)
- Speculative planning (predict the next 3 steps and start LLM calls in parallel)
- Or realistic SLAs (tell your customers the agent takes 30 seconds)
We went with speculative planning. It adds complexity but cuts end-to-end latency by 40%.
Rate limiting at two levels. LLM providers throttle you. Your own tools throttle you. You need rate limiters that talk to each other. We use a centralized token bucket that the orchestration layer checks before any LLM or tool call. If the bucket is empty, the agent waits — not fails.
python
# Two-layer rate limiter
class AgentRateLimiter:
def __init__(self, llm_calls_per_minute=60, tool_calls_per_minute=200):
self.llm_bucket = TokenBucket(llm_calls_per_minute, time_window=60)
self.tool_bucket = TokenBucket(tool_calls_per_minute, time_window=60)
async def check_limits(self, call_type="llm"):
if call_type == "llm":
return await self.llm_bucket.consume()
else:
return await self.tool_bucket.consume()
Observability that understands agents. Standard APM tools show you request/response. Agents have branching logic, parallel tool calls, and state machines. You need tracing that captures the full execution DAG.
We built a custom OpenTelemetry span set that wraps each agent node, tool call, and state transition. Each span carries the agent ID, thread ID, and step number. Without that, debugging is impossible.
The Tool Contract: Where Production Agents Actually Break
Here's the thing nobody warns you about. The LLM will call tools with wrong arguments. It will call them in the wrong order. It will call them too many times.
Your tools need to be crash-proof. Not crash-resistant. Crash-proof.
We enforce what we call the "Tool Contract":
-
Input validation built into the tool, not the agent. The agent suggests parameters. The tool validates them. If parameters are invalid, the tool returns a structured error explaining what's wrong — not a crash.
-
Idempotency keys on every mutation endpoint. If the agent calls "create_invoice" twice because it didn't wait for the first response, you get two invoices. Idempotency keys prevent that. Every agent gets a unique ID, and every tool call carries that ID as an idempotency token.
-
Timeout enforcement at the tool level. Tools shouldn't hang. We set a 30-second timeout on every tool call. If the tool doesn't respond, it returns a "timeout" status and the agent can retry or escalate.
python
# Tool contract enforcement
from pydantic import BaseModel, validator
from typing import Optional
class ToolInput(BaseModel):
agent_id: str
idempotency_key: str
parameters: dict
@validator('agent_id')
def agent_id_must_exist(cls, v):
if not agent_id_exists(v):
raise ValueError(f"Unknown agent: {v}")
return v
@validator('idempotency_key')
def check_duplicate(cls, v, values):
if is_idempotent_key_used(v):
return get_cached_response(v)
return v
@tool(timeout=30, retries=0) # No retries — agent decides
def create_invoice(input: ToolInput) -> dict:
# Tool logic here
pass
Most people think the agent controls the tools. Wrong. The tools control the agent.
The agent proposes. The tool disposes.
Agentic Workflow Production Rollout: The Staged Approach
We roll out agentic workflows in four stages. Each stage takes 2-4 weeks. Rushing kills you.
Stage 1: Shadow mode. The agent runs alongside your existing system. Reads the same inputs. Produces outputs. But nobody acts on them. You're collecting data on what the agent would have done vs. what actually happened. We ran a customer support agent in shadow mode for three weeks before we trusted it with real tickets.
Stage 2: Human-in-the-loop. The agent makes recommendations. A human approves or rejects. This is where you find the edge cases — the agent's confidence doesn't match its accuracy, or it misunderstands ambiguous requests. We measure "acceptance rate" at this stage. Needs to be above 80% before we proceed.
Stage 3: Guardrails + auto-approve low risk. You categorize agent actions by risk level. "Read customer email" is low risk. "Issue refund" is high risk. Low-risk actions get auto-approved. High-risk actions still need human sign-off. The guardrails system (we use a combination of Azure AI Content Safety and custom regex pattern matching) monitors everything and can halt any action mid-flight.
python
# Guardrail enforcement during execution
class GuardrailSystem:
def __init__(self):
self.rules = [
HighValueTransactionRule(limit=5000), # block >$5k
PersonalDataExposureRule(), # scan for PII in output
EscalationPatternRule(), # detect agent asking for help too often
]
def check_action(self, action: dict) -> GuardrailResult:
for rule in self.rules:
result = rule.evaluate(action)
if not result.passed:
return GuardrailResult(
allowed=False,
reason=result.message,
severity=result.severity
)
return GuardrailResult(allowed=True)
Stage 4: Full autonomy. The agent handles everything within its scope. But every action is logged, every decision explainable, and we can pull the agent out of production in under 30 seconds.
Most teams skip stages 1 and 2. Those are the teams that have production incidents at 3 AM on a Saturday.
AI Agents Deployment Best Practices We've Hardened
After deploying 12 production agents across 4 clients, here's what's non-negotiable:
Test with real traffic patterns. Synthetic test data produces synthetic problems. Use logged production traffic to replay through your agent. Record outputs. Compare to historical human decisions. This catches the "works on the demo set, fails in production" pattern.
Budget LLM tokens per conversation. One agent conversation consumed 180,000 tokens in our first production test because the agent got stuck in a loop of querying the same database. Token budget of 25,000 per session prevents runaway costs. Hard limit at 50,000 with escalation.
Version your agents, not just your models. The agent's behavior depends on the system prompt, the tool definitions, the model version, and the framework version. All of those need versioning and rollback capability. We tag every deployment with a hash of the entire agent configuration.
Monitor for "retirement" patterns. Agents that are constantly asking for clarification or escalation aren't helpful. We track "escalation rate" as a primary metric. If it's above 15%, the agent isn't ready.
The FAQ That Answers What Most People Get Wrong
Q: Do I need to fine-tune the model for my agent?
A: Usually not. Good system prompts and tool definitions handle 90% of use cases. Fine-tune only if you need specific output format compliance or domain terminology that the base model consistently gets wrong. We've fine-tuned exactly once — for a legal contract analysis agent. Everything else runs GPT-4o or Claude 3.5 Sonnet.
Q: How do I handle agent hallucination in production?
A: You can't prevent it. You design around it. Every factual claim the agent makes should be traceable to a source. If the agent says "order #12345 was shipped," it should cite the API response that confirms shipping. We call this "citation-enforced generation" and it's the closest thing to a hallucination fix we've found.
Q: One big agent or many small ones?
A: Many small ones. A single agent with 30 tools is harder to debug and control than 5 agents with 6 tools each. Break your domain into agent-sized chunks. Agentic AI Frameworks like LangGraph make multi-agent coordination straightforward.
Q: What infrastructure do I actually need?
A: The agent orchestration layer can run on standard compute — Kubernetes is fine. The LLM inference needs GPU or access to an API. The tool execution layer needs to be close to your data. We run tool executors as sidecar containers to our agent orchestrator for latency.
Q: Is LangChain production-ready yet?
A: LangGraph is. LangChain's core library is still undergoing breaking changes (version 0.3.x in July 2026). Pin your versions and don't upgrade without testing. We're quarterly-upgrade only on LangChain packages.
Q: How do I handle the cost?
A: Every agent action should have a cost cap. We track cost per conversation session. If a session costs more than $2.50, we flag it for review. Average should be under $0.50 per resolved issue. You'll also want to cache frequent LLM calls — semantic caching reduces API costs by 30-40%.
Q: What's the biggest mistake teams make?
A: Over-automating too fast. Let the agent fail in shadow mode. Let it confuse humans in human-in-the-loop mode. Build confidence before giving it the keys. We've seen teams skip to autonomy and lose customer trust in 48 hours.
What Comes Next
The industry is moving from "can we build agents" to "can we trust agents." By late 2026, we're seeing the first third-party agent verification firms — companies that audit agent behavior for reliability and safety.
At SIVARO, we're building what we call "agent lifecycle management" — tools for deploying, monitoring, and retiring production agents. It's the DevOps for the agent era, and it's desperately needed.
The frameworks will keep changing. The models will get faster. The protocols will standardize. But the deployment fundamentals — state persistence, idempotent tools, staged rollout, observation-first design — those won't change. They're the hard-won lessons from 18 months of failing forward.
Go build. But go build carefully.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.