The a2a Protocol Production Deployment Example: What Worked (and What Didn't)
I spent June 2026 deploying an agent-to-agent (a2a) protocol stack in production for a logistics client. Three teams, eight weeks, two near-disasters. Here's what actually matters.
The Agent Transfer Protocol (a2a) isn't theoretical anymore. Google released the spec in April 2025. By January 2026, we had four vendors shipping compatible agents. By June, my team at SIVARO pushed a multi-agent system live for a freight forwarder processing 12,000 bookings daily.
This is that story. No fluff. No vendor pitches. Just the decisions, the mistakes, and the patterns that survived production.
What a2a Actually Solves (and Doesn't)
Most people think a2a is about "agent interoperability." That's technically true but practically useless. Here's what it really does:
It gives you a standard way for one agent to say "I need this task done" and another agent to say "Here's the result" — without hard-coding the handshake logic between every pair.
Before a2a, we built bespoke bridges. Agent A calls Agent B via REST. Agent B calls Agent C via gRPC. Agent C calls Agent D via WebSocket. Every integration was a snowflake. Every failure required tracing through three different transport layers.
a2a standardizes the agent card (discovery), the task lifecycle (creation, update, completion), and the message format. Nothing more. Nothing less.
What a2a doesn't do: handle authentication, define your data model, manage state, or guarantee delivery. You still own all of that.
The Architecture We Deployed
Here's the production architecture that's been running since June 1, 2026:
mermaid
graph TD
A[Client API Gateway] --> B[Orchestrator Agent]
B --> C[Booking Agent]
B --> D[Pricing Agent]
B --> E[Compliance Agent]
C --> F[Legacy ERP via a2a Bridge]
D --> G[External Rate API]
E --> H[Sanctions Database]
B --> I[Monitoring Agent]
I --> J[Prometheus + Grafana]
I --> K[PagerDuty]
The orchestrator receives a booking request. It fans out tasks to three specialist agents via a2a. Each specialist reports back. The orchestrator merges results and responds.
We chose LangGraph as our framework because it supported a2a natively by Q4 2025 (AI Agent Frameworks: Choosing the Right Foundation for ...). LangChain's blog on agent frameworks confirmed our suspicion: "the framework should disappear once you're in production" (How to think about agent frameworks). LangGraph doesn't get in the way. It just runs.
The Agent Card: Your First Production Headache
Every a2a agent publishes an agent card — a JSON document describing what it does, what inputs it expects, and what outputs it produces.
This seems trivial. It's not. Here's our production agent card for the Booking Agent:
json
{
"agentId": "booking-agent-v2-1",
"name": "Booking Agent",
"description": "Handles ocean freight booking creation and status tracking",
"capabilities": [
{
"name": "create_booking",
"input": {
"type": "object",
"properties": {
"reference": {"type": "string", "required": true},
"containers": {"type": "array", "items": {"$ref": "#/definitions/container"}, "required": true},
"destination_port": {"type": "string", "required": true},
"incoterm": {"type": "string", "enum": ["FOB", "CIF", "EXW"]}
}
},
"output": {
"type": "object",
"properties": {
"booking_number": {"type": "string"},
"status": {"type": "string"}
}
}
}
],
"endpoints": {
"taskEndpoint": "https://booking-agent.internal.sivaro.io/a2a/task",
"agentCardEndpoint": "https://booking-agent.internal.sivaro.io/a2a/card"
},
"version": "2.1.0",
"rateLimit": 100
}
Three lessons from this:
First, version your cards. We didn't version v1. When the Pricing Agent updated its input schema, the Orchestrator sent malformed requests for 47 minutes before we caught it. Add version to every card, increment on every schema change.
Second, limit the description. I wrote a paragraph explaining the business logic. Useless. Agents don't read prose. Stick to machine-parseable constraints — enums, required fields, regex patterns.
Third, keep the card simple. The A Survey of AI Agent Protocols paper from arXiv covers 47 different protocol features. We use 6. The rest are unused. Don't build features you won't use.
Task Lifecycle: The Part Where Things Break
a2a defines four task states: pending, running, completed, failed. That's it. Simple, right?
Wrong.
The problem is what happens between states. Our deployment taught us three hard lessons:
Timeout cascading. The Orchestrator sends a booking to the Booking Agent. The Booking Agent calls the ERP, which takes 30 seconds. But the Orchestrator's timeout is 15 seconds. The Orchestrator retries, creating a duplicate booking. We lost $12,000 in shipping fees before we caught it.
Fix: Make timeout configurable per task type, and propagate it in the task request payload.
Idempotency is not optional. Every a2a task request must include an idempotency_key. We skipped this in our first sprint. Bad idea. When the network blips (which it does, roughly 0.3% of requests in our environment), the client retries and you get two BookingsAgent tasks for one user request.
Here's our production retry logic:
python
import hashlib
import uuid
from datetime import datetime
class TaskClient:
def __init__(self, agent_endpoint: str):
self.endpoint = agent_endpoint
def create_task(self, workflow_id: str, intent: dict):
# Generate deterministic idempotency key from workflow context
key_content = f"{workflow_id}:{intent['capability']}:{datetime.utcnow().hour}"
idempotency_key = hashlib.sha256(key_content.encode()).hexdigest()[:32]
payload = {
"idempotency_key": idempotency_key,
"capability": intent["capability"],
"input": intent["input"],
"timeout_seconds": 30
}
return self._post(f"{self.endpoint}/a2a/task", payload)
State synchronization. a2a is asynchronous by design. The Orchestrator sends a task, gets a task_id, and then polls for status. But what if the Orchestrator crashes between sending and polling?
We solved this with a task state table in PostgreSQL. Every sent task gets a row. A background worker reconciles tasks that are pending for more than 60 seconds. This caught 3 orphaned tasks in the first week.
Observability: The Thing Nobody Talks About
Everyone talks about building agents. Nobody talks about watching them fail.
Our a2a deployment generates 250GB of logs daily. Without structured observability, you're blind. Here's our production setup:
- All a2a messages include a trace_id header (W3C trace context)
- Every task lifecycle event logs: trace_id, agent_id, task_id, state, duration_ms
- Metrics labeled by agent_id, capability, state, error_code
- Alert when task failure rate exceeds 2% over 5 minutes
- Dashboard for end-to-end latency by workflow type
The top 10 agentic AI frameworks for 2026 list from Instaclustr mentions observability as a key differentiator. I'd go further: it's the deciding factor. We tested CrewAI and AutoGen before settling on LangGraph. Both lacked production-grade tracing. LangGraph had OpenTelemetry integration.
Our one non-negotiable: every agent must log every a2a message it sends and receives. Not just errors. Every message. You'll thank me when you're debugging a "but the data disappeared" ticket at 2 AM.
Here's the logging middleware we inject into every agent:
python
import time
import traceback
from opentelemetry import trace
class A2ALoggingMiddleware:
def __init__(self, logger):
self.logger = logger
self.tracer = trace.get_tracer(__name__)
def handle_task(self, task_request):
start = time.time()
trace_id = trace.format_trace_id(trace.get_current_span().get_span_context().trace_id)
self.logger.info({
"event": "a2a_task_received",
"trace_id": trace_id,
"task_id": task_request.get("idempotency_key"),
"capability": task_request.get("capability"),
"timestamp": time.time()
})
try:
result = self._process(task_request)
duration = (time.time() - start) * 1000
self.logger.info({
"event": "a2a_task_completed",
"trace_id": trace_id,
"task_id": task_request.get("idempotency_key"),
"duration_ms": duration,
"status": "completed"
})
return result
except Exception as e:
duration = (time.time() - start) * 1000
self.logger.error({
"event": "a2a_task_failed",
"trace_id": trace_id,
"task_id": task_request.get("idempotency_key"),
"duration_ms": duration,
"error": str(e),
"traceback": traceback.format_exc()
})
raise
Error Handling: The Ugly Truth
a2a's error model is minimal. You get a failed state and an error message string. That's it.
In practice, you need structured errors. Here's our error taxonomy:
json
{
"errors": {
"temporary": {
"TIMEOUT": {"retryable": true, "message": "Agent did not respond within timeout"},
"RATE_LIMITED": {"retryable": true, "message": "Agent is rate limited"},
"INTERNAL_ERROR": {"retryable": true, "message": "Agent encountered internal error"},
"TEMP_UNAVAILABLE": {"retryable": true, "message": "Agent temporarily unavailable"}
},
"permanent": {
"INVALID_INPUT": {"retryable": false, "message": "Input does not match schema"},
"UNSUPPORTED_CAPABILITY": {"retryable": false, "message": "Capability not supported"},
"UNAUTHORIZED": {"retryable": false, "message": "Authentication failed"}
}
}
}
Why this matters: our first deployment retried everything 3 times. Permanent errors (like invalid input) consumed 66% of retry slots. We were burning resources on errors that would never succeed.
Now: retry on temporary only, with exponential backoff (1s, 4s, 16s). Reject immediately on permanent. Saves 40% of agent CPU cycles.
Security: The Elephant
a2a says nothing about authentication. It's a protocol for task exchange, not for security.
We use mTLS between agents. Every agent has a client certificate signed by our internal CA. The Orchestrator verifies the target agent's certificate before sending any task.
Why not API keys? Because API keys leak. A developer committed a Pricing Agent API key to a public GitHub repository in January. We rotated keys for 14 agents in 48 hours. mTLS doesn't have that problem — certificates can be revoked individually.
Here's our mTLS handshake for a2a:
python
import ssl
import requests
def create_agent_session():
ssl_ctx = ssl.create_default_context()
ssl_ctx.load_cert_chain(
certfile="/etc/ssl/certs/agent-cert.pem",
keyfile="/etc/ssl/private/agent-key.pem"
)
ssl_ctx.load_verify_locations(cafile="/etc/ssl/certs/ca-cert.pem")
ssl_ctx.verify_mode = ssl.CERT_REQUIRED
session = requests.Session()
session.verify = "/etc/ssl/certs/ca-cert.pem"
session.cert = ("/etc/ssl/certs/agent-cert.pem", "/etc/ssl/private/agent-key.pem")
return session
Performance Under Load
Here's what we're seeing in production (July 2026):
| Metric | Value |
|---|---|
| Peak throughput | 2,400 tasks/minute |
| P50 latency | 1.2 seconds |
| P99 latency | 8.7 seconds |
| Task failure rate | 1.8% |
| Agent availability | 99.92% |
The bottleneck? The legacy ERP. The a2a layer adds 50-150ms of overhead per task. The ERP adds 2-5 seconds.
Contrarian take: a2a is fast enough. The protocol overhead is noise. Your bottleneck will be your slowest downstream system, not the protocol.
Monitoring Agent Health
AI agents fail differently than microservices. A microservice either returns 200 or 500. An agent might return 200 with garbage data.
We built a health-check endpoint that every agent exposes:
GET /a2a/health
Returns:
json
{
"status": "healthy",
"uptime_seconds": 123456,
"last_successful_task": "2026-07-19T10:23:15Z",
"tasks_processed": 8472,
"error_rate_1h": 0.015,
"model_latency_p99_ms": 2500,
"queue_depth": 3
}
Our monitoring agent pings this every 30 seconds. If error_rate_1h > 0.05 or queue_depth > 50, it pages the on-call engineer.
We learned this the hard way. Our Pricing Agent's ML model started returning confidence scores below 0.3 for 40% of requests — but it never crashed. Without health metrics, we wouldn't have noticed for hours.
The Deployment Pipeline
Here's our ai agent deployment pipeline tutorial in practice:
We use GitOps with ArgoCD. Every agent is a container. The agentCard.json is part of the repository. CI runs:
- Unit tests (mock all a2a endpoints)
- Integration tests (run 10 agents locally, test full workflows)
- Schema validation (is the agent card valid a2a?)
- Load tests (simulate 500 concurrent tasks)
Deployment is canary: 10% of traffic for 15 minutes, then 50%, then 100%. If error rate spikes, rollback is automatic.
We also run a shadow deployment. Every production task is duplicated to a staging environment. We compare results. If staging differs from production, we investigate. This caught a schema mismatch in the Compliance Agent before it hit production.
a2a Protocol Production Deployment Example: The Checklist
If you're doing this tomorrow, here's your checklist:
- Define agent cards first — before writing any code. Get schema agreement between teams.
- Implement idempotency — use deterministic keys from workflow context.
- Add structured errors — distinguish temporary from permanent failures.
- Set up observability — trace_id on every message, structured logs, latency metrics.
- Write health endpoints — expose model-level and queue-level metrics.
- Deploy with canaries — don't switch all traffic at once.
- Run shadow tests — duplicate production traffic to staging.
- Document timeouts — per-capability, not a single global timeout.
- Use mTLS — not API keys.
- Test failure modes — what happens when the Pricing Agent is down for 5 minutes?
FAQ
Q: Do I need a2a if I'm running a single agent?
No. a2a adds complexity. Use it when you have 3+ agents that need to talk to each other.
Q: a2a vs MCP (Model Context Protocol)?
MCP is for agent-to-tool communication. a2a is for agent-to-agent. You'll likely need both. We use MCP for tool calling and a2a for inter-agent workflows.
Q: How do you handle agent discovery at scale?
We use a service registry (Consul) with the agent card URL. The Orchestrator discovers agents by capability. Simple. Doesn't need a blockchain or a mesh.
Q: What about WebSocket vs HTTP for a2a?
HTTP for short tasks (under 30 seconds). WebSocket for long-running tasks. We use HTTP for 95% of tasks.
Q: Can a2a handle streaming responses?
Not natively. The spec is request-response. For streaming, we added a separate event stream channel that the client polls. Works fine.
Q: How do you test a2a agents?
Unit tests with mock a2a servers. Integration tests with real containers. Shadow testing in production. Chaos engineering once a week — we randomly kill agents and verify recovery.
Q: What's the biggest mistake teams make?
Over-engineering. They try to build a universal agent platform with routing, caching, retry, monitoring, and governance — all before running a single agent in production. Start with one workflow. Add complexity when you have data.
Q: When should I NOT use a2a?
When your agents are tightly coupled and always co-deployed. Use gRPC direct calls. a2a's async indirection adds cost for no benefit.
What's Next
The AI Agent Protocols: 10 Modern Standards Shaping the ... article from SSONetwork predicts federated agent discovery by Q1 2027. I think they're right. The manual configuration of agent cards doesn't scale past 50 agents.
We're also watching the open-source agentic AI frameworks list from AI Multiple. LangGraph and CrewAI are adding a2a-native features each release. By Q4 2026, I expect framework integration to be seamless. It's not there yet.
Final Take
a2a isn't revolutionary. It's a simple protocol for a simple problem: two agents need to exchange a task.
The hard parts — idempotency, observability, security, error handling — are all outside the protocol. You build those yourself. The protocol just gives you a consistent envelope to put them in.
That's worth it. We went from 14 bespoke integrations to 1 protocol. Our mean time to recovery dropped from 6 hours to 45 minutes. Not because a2a is magical, but because having a standard meant we could build better tooling around it.
A2a production deployment example isn't about the protocol. It's about everything else you need to make agents reliable. Focus there.
—
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.