A2A Protocol Production Deployment: Our Battle-Tested Guide

Free Technical Audit

Expert Review

What I'm about to share cost us six months of production pain.

In February 2025, SIVARO got a call from a logistics company. They'd built an agent system using the A2A protocol from Google's initial spec. Looked great in staging. In production, it fell apart inside 48 hours. Agents talking to agents? Worked beautifully. Until it didn't.

The A2A (Agent-to-Agent) protocol lets autonomous agents communicate directly — no central orchestrator, no brittle API layer. Think of it as HTTP for agents. Each agent publishes its capabilities, discovers others, and negotiates task execution on the fly. But going from demo to production is where most teams hit a wall.

I'm Nishaant Dixit, founder of SIVARO. We've deployed A2A-based systems for three clients now, handling 200K+ events per second in production. This guide walks through exactly how we do it — the architecture, the monitoring, the failure modes you won't read in the spec.

You'll learn:

The four-layer production architecture we use
Exactly how to configure agent discovery and trust
What observability looks like when agents are making decisions in milliseconds
The three failure patterns that will kill your deployment

Let's get into it.

Why A2A Blew Up in 2025

Most people think agent protocols are a technology problem. They're wrong.

The real bottleneck was coordination. Every agent framework before A2A (AI Agent Frameworks: Choosing the Right Foundation for ...) assumed you'd build your own message bus. Google changed that with A2A in April 2024. By late 2025, it's the closest thing to a standard we have.

The AI Agent Protocols: 10 Modern Standards Shaping the ... article nails the comparison — but the short version is: A2A handles discovery, task negotiation, and multi-modal responses natively. Your agent doesn't need to know where a weather service lives. It just broadcasts "who's got temperature data?" and gets a response.

That's the promise. Here's the reality.

Our Production Architecture: Four Layers

We tested three architectures before settling on this one. The naive approach (every agent directly connected) fails at about 50 agents. The hub-and-spoke (central message broker) bottlenecks at scale. Our four-layer pattern handles 500+ agent meshes without drama.

Layer 1: Agent Registry + Discovery

POST /agents/register
{
  "agent_id": "weather-service-v3",
  "capabilities": ["temperature:fetch", "forecast:7day", "alerts:weather"],
  "agent_card": {
    "name": "Weather Service",
    "description": "Provides weather data for any US zip code",
    "url": "https://agents.sivaro.io/weather/v3/a2a",
    "version": "3.2.1",
    "authentication": {
      "scheme": "oauth2",
      "audience": "https://agents.sivaro.io"
    }
  },
  "metadata": {
    "owner": "weather-team",
    "p99_latency_ms": 250,
    "max_concurrent": 100,
    "maintenance_window": "sunday-0200-0400-utc"
  }
}

This is the first thing most teams get wrong. They register capabilities as free text. Don't. Use a controlled vocabulary. We maintain a canonical list of ~200 capability types. If your agent can't describe itself using those types, it doesn't register.

Why? Because discovery queries fail silently. An agent asks "who offers temperature data?" and gets no match because one agent says "temp-service" and another says "weather-fetch-temperature." You won't catch this until something breaks at 3 AM.

Our registry exposes a simple query endpoint:

GET /agents/discover?capability=temperature:fetch&latency_max=300&version>=3.0.0

Returns the three best matches based on latency history and availability.

Layer 2: Message Transport (with Backpressure)

A2A specifies a message format but not transport. We use gRPC for agent-to-agent communication with a WebSocket fallback for browsers. Here's our actual message envelope:

{
  "message_id": "msg-20260719-abc123",
  "source_agent": "pipeline-orchestrator-v2",
  "target_agent": "data-enrichment-service-v4",
  "message_type": "TASK_OFFER",
  "conversation_id": "conv-xyz789",
  "timestamp": "2026-07-19T14:30:00Z",
  "priority": 3,
  "ttl_ms": 5000,
  "payload": {
    "task_id": "task-001",
    "input_schema": {
      "type": "object",
      "properties": {
        "raw_event": {"type": "object"},
        "enrichment_type": {"type": "string", "enum": ["geo", "user_profile", "device_info"]}
      }
    },
    "callback": "https://pipeline-orchestrator-v2.sivaro.io/callback"
  }
}

The TTL field? That's your backpressure mechanism. If an agent can't process within the TTL, the sender retries with a different agent. We don't do infinite retries — three attempts, then fail to a dead-letter topic.

Layer 3: Task Negotiation Protocol

This is where A2A shines and where most implementations break.

An agent doesn't just fire-and-forget. It offers a task, the receiving agent responds with acceptance, rejection, or a counter-offer. Our negotiation looks like:

// Agent A offers
TASK_OFFER -> Agent B

// Agent B responds
TASK_ACCEPT {
  "estimated_completion_ms": 1200,
  "confidence": 0.95,
  "terms": {
    "max_cost": "0.002 USD per task",
    "data_retention": "deleted after 24 hours"
  }
}

// Agent A acknowledges
TASK_START {
  "acknowledged_terms": true
}

// Agent B streams progress
TASK_PROGRESS {
  "status": "processing",
  "percent_complete": 42,
  "intermediate_result": { ... }
}

// Agent B completes
TASK_COMPLETE {
  "result": { ... },
  "metrics": {
    "compute_ms": 1100,
    "input_tokens": 450,
    "output_tokens": 120
  }
}

Sounds clean. It's not. The negotiation adds 200-400ms per interaction. For latency-sensitive pipelines, that's death. We optimized by caching accepted terms. If Agent A has accepted Agent B's terms in the last 5 minutes, we skip negotiation and go straight to TASK_START.

Layer 4: Observability Pipeline

This is the part nobody talks about. How to think about agent frameworks mentions observability — but what they don't say is that A2A creates distributed traces that span agent boundaries. You can't use standard request tracing because agents make autonomous decisions.

We built a sidecar pattern. Every A2A message goes through a sidecar process that injects tracing context:

// Sidecar injection
const traceContext = {
  trace_id: "trace-001",
  parent_span_id: "span-abc",
  span_id: "span-def",
  agent_decisions: [
    {action: "selected_weather_service", confidence: 0.92, latency_ms: 45},
    {action: "rejected_geo_enrichment", reason: "threshold_not_met"}
  ]
}

message.headers["x-trace-context"] = JSON.stringify(traceContext)

This lets us reconstruct agent decision trees. When a pipeline fails, we can see exactly which agent chose what and why. We feed this into ai agent observability production dashboards.

The Three Failure Patterns You'll See

Pattern 1: The Silent Timeout Chain

Agent A calls Agent B. Agent B calls Agent C. Agent C is slow. Agent B doesn't tell Agent A. Agent A's TTL expires. Agent A retries with Agent D. Agent D calls Agent B again. Now you have two parallel executions of the same task.

Fix: Every agent must send a heartbeat within 30% of the TTL. If no heartbeat, the sender assumes failure. We enforce this at the transport layer — it's not negotiable.

Pattern 2: Capability Drift

Your weather service updates from v3 to v4. The registry knows. But Agent A cached the discovery result 15 minutes ago. It keeps calling v3. v3 is now decommissioned. Calls fail.

Fix: Short TTLs on discovery cache (max 60 seconds). Plus a push notification when any agent's capability changes. We use a Redis pub/sub channel for this. Every agent subscribes to its capability changes and re-registers.

Pattern 3: The Confidence Cascade

Agent B is 95% confident in its result. Agent C is 97% confident. The pipeline's final output is a product of these confidences — that's 92%. Looks fine. But Agent D is only 40% confident, and Agent E (which depends on D) doesn't output confidence at all.

We saw this kill a production recommendation system. The output looked good but was based on garbage.

Fix: Every agent must report confidence. If any agent in the chain drops below 0.5, the pipeline halts and a human reviews. We use a weighted confidence score — downstream agents inherit the minimum confidence of their dependencies.

A2A Protocol Production Deployment Example: Step by Step

Here's the exact deployment script we use internally. It provisions three agents, registers them, and sets up monitoring.

bash
# Clone our A2A agent template
git clone https://github.com/sivaro/agent-template.git ./my-agent

# Install dependencies
cd my-agent
npm install
npm run build

# Configure agent
cat > config.yaml << 'EOF'
agent:
  id: "event-processor-v1"
  transport: "grpc"
  registry_url: "https://registry.sivaro.io"
  capabilities:
    - "event:normalize"
    - "event:deduplicate"
  constraints:
    max_concurrent_tasks: 50
    p99_latency_ms: 200
    memory_mb: 512

observability:
  sidecar_url: "http://localhost:9091"
  trace_sample_rate: 0.1
  
authentication:
  method: "oauth2"
  client_id: "${A2A_CLIENT_ID}"
  client_secret: "${A2A_CLIENT_SECRET}"
EOF

# Register with discovery service
curl -X POST https://registry.sivaro.io/agents/register   -H "Authorization: Bearer ${A2A_TOKEN}"   -d @- << 'REGJSON'
{
  "agent_id": "event-processor-v1",
  "capabilities": ["event:normalize", "event:deduplicate"],
  "url": "grpc://event-processor-v1.sivaro.io:50051"
}
REGJSON

# Start the agent
npx sivaro-agent --config config.yaml

The sivaro-agent binary handles message routing, health checks, and sidecar communication. We open-sourced this last month — [link to our repo]. It's the same thing we run in production.

Production Checklist We Use

Before any A2A deployment goes live, we verify:

[ ] Every agent has a declared p99 latency (not p50 — that's lying)
[ ] Discovery cache TTL ≤ 60 seconds
[ ] Push notifications for capability changes
[ ] Sidecar tracing ≥ 10% sample rate (100% for first week)
[ ] Dead-letter queue for failed messages
[ ] Confidence score requirement (≥ 0.5 to pass)
[ ] Heartbeat interval ≤ 30% of message TTL
[ ] Circuit breaker: max 5 consecutive failures before agent isolation
[ ] Load test at 3x expected peak throughput

We missed the circuit breaker on our first deployment. A misconfigured weather agent took down the entire pipeline. The circuit breaker isolates failing agents — they're still registered but won't receive new tasks. We added it mid-flight.

Observability: What You Actually Need

Most ai agent observability production guides tell you to monitor latency and error rates. That's table stakes. What you actually need:

Agent decision traces — Which agent chose what, and why
Confidence distribution — Is the network getting uncertain over time
Capability coverage — Are all capabilities represented by at least 2 agents (for redundancy)
Negotiation overhead — How much time is spent on protocol vs actual work

We use a custom dashboard built on ClickHouse. It ingests sidecar traces and shows real-time agent health:

sql
-- Sample query: agent health with decision confidence
SELECT 
  agent_id,
  avg(confidence) as avg_confidence,
  count(*) as task_count,
  sum(case when status = 'FAILED' then 1 else 0 end) / count(*) as failure_rate,
  quantile(0.99)(negotiation_time_ms) as p99_negotiation,
  quantile(0.99)(work_time_ms) as p99_work
FROM agent_traces
WHERE timestamp > now() - INTERVAL 1 HOUR
GROUP BY agent_id
ORDER BY failure_rate DESC

Dealing With Legacy Systems

Not everyone runs pure A2A. We have clients with ten-year-old REST APIs they can't rewrite. The A Survey of AI Agent Protocols paper covers this — but in practice, we built adapters.

An A2A-to-REST adapter wraps the legacy API as an A2A agent:

python
class RestToA2AAdapter:
    def __init__(self, rest_url, capability_mapping):
        self.rest_url = rest_url
        self.capability_mapping = capability_mapping  # dict: capability -> endpoint
        
    async def handle_task_offer(self, task):
        # Convert A2A task to REST call
        endpoint = self.capability_mapping.get(task.capability)
        if not endpoint:
            return TaskReject(reason="unsupported_capability")
        
        # Translate payload
        rest_payload = self.translate_to_rest(task.payload)
        
        # Call legacy API
        async with aiohttp.ClientSession() as session:
            async with session.post(
                f"{self.rest_url}/{endpoint}",
                json=rest_payload,
                timeout=aiohttp.ClientTimeout(total=10)
            ) as response:
                result = await response.json()
        
        # Return as A2A response
        return TaskComplete(
            result=self.translate_to_a2a(result),
            confidence=0.95,
            metrics={"compute_ms": response.elapsed * 1000}
        )

The adapter adds ~50ms latency. That's fine. It also handles the legacy system's weird error codes — mapping HTTP 500 to A2A TASK_FAILED, 429 to TASK_REJECT with "backpressure" reason.

The Anti-Patterns (What We Learned To Stop Doing)

Long-lived connections — We tried persistent gRPC streams. They worked until agents restarted without notice. Now every message is a new connection with auth.
Implicit trust — A2A doesn't authenticate agents by default. We added mutual TLS between all agent endpoints. The registry issues short-lived certificates (24 hours). Infra teams hate this until a compromised agent tries to inject tasks.
Shared state — Don't. Let agents pass state through messages. We saw a team use Redis pub/sub for agent coordination. It worked until a Redis cluster went down and agents had inconsistent views. Messages only.
Bidirectional dependencies — Agent A calls B, B calls A. Circular dependency creates deadlock. We enforce directed acyclic graphs at registration time. The registry rejects registration if it creates a cycle.

What We're Building Now

The protocol is stabilizing, but the ecosystem is still maturing. Top 5 Open-Source Agentic AI Frameworks in 2026 shows the landscape — we're contributing to the open-source agent runtime.

Our focus: predetermined cost ceilings. Every agent negotiation currently has variable compute cost. For production billing, you need to know the maximum cost before accepting a task. We're prototyping a "cost card" in the agent registration — similar to bandwidth pricing in cloud services.

The Agentic AI Frameworks: Top 10 Options in 2026 list flags this as a gap. No framework handles cost guarantees well. We're building it because our clients can't go to production without knowing their infrastructure bill.

Final Take

A2A protocol production deployment isn't about the protocol itself. It's about the infrastructure around it — discovery, observability, backpressure, failure isolation. The protocol handles the "how" of agent communication. You handle the "what happens when it breaks."

We've seen teams spend months on agent logic and two days on operations. Then wonder why it crashes. Don't be that team.

Build the registry first. Wire up the sidecar second. Agent logic is the easy part.

FAQ

Should I use A2A or MCP (Model Context Protocol)?

A2A handles agent-to-agent communication. MCP handles agent-to-tool integration. They're complementary. We use both — MCP for vector database queries, A2A for multi-agent workflows. AI Agent Protocols: 10 Modern Standards Shaping the ... explains the distinction better than I can.

Can I use A2A with non-AI services?

Yes. We have agents that wrap a PostgreSQL instance — they receive query tasks, execute SQL, return results. The protocol doesn't care about the agent's internals. Just the interface.

How many agents can one A2A mesh support?

We've tested up to 1,500 agents in a single mesh. The bottleneck is the registry — it needs to handle discovery queries at scale. Our registry handles 10K queries per second on a single node. Scale horizontally for more.

What's the latency overhead of A2A?

Negotiation adds 200-400ms per interaction. With cached terms, it drops to 50-100ms. The actual message transport (gRPC) adds single-digit milliseconds. For latency-sensitive pipelines, cache aggressively and minimize negotiation hops.

Do I need Kubernetes for A2A production?

No. We've deployed on bare metal and serverless (AWS Lambda via adapters). Kubernetes makes agent scaling easier but isn't required. The registry needs to be always-on — that's the only hard requirement.

How do you handle agent versioning?

Each agent card includes a version field. The registry supports version constraints in discovery queries. We run two versions of critical agents simultaneously and redirect traffic based on version ranges. Every Friday, we deprecate the oldest version.

Is A2A secure for financial services?

With mutual TLS and short-lived certificates, yes. We have a client processing payment reconciliations with A2A. The protocol itself doesn't handle encryption — that's transport-layer responsibility. Use gRPC with TLS, don't expose agents to the public internet, and implement rate limiting at the registry level.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.