Is Apache Kafka Still Used? A 2026 Field Report

July 7, 2026

I got a call last Tuesday from a founder who's building what he calls "the next-gen event platform." He's 27, raised $12M, and he told me Kafka is dead. "Nobody uses Kafka anymore. It's all Redpanda and WarpStream now."

I didn't laugh at him. I've been wrong before. But I've also been running SIVARO since 2018, building data infrastructure for clients processing 200K events per second. I've watched Kafka survive four "Kafka killers." I've seen it lose ground in exactly one use case (low-latency streaming under 10ms). And I've watched it quietly become the backbone of half the production AI systems I encounter.

So here's the real question: is apache kafka still used? The answer isn't yes or no. It's "it depends on whether you're building a real system or a demo."

Let me show you what I mean.

What Kafka Actually Is (And Isn't)

Apache Kafka started at LinkedIn in 2011 as a distributed commit log. Jay Kreps, Neha Narkhede, and Jun Rao needed something that could handle the firehose of LinkedIn's activity data—page views, clicks, searches—without losing messages or grinding to a halt.

Here's what Kafka is:

A distributed log that stores ordered sequences of records
A pub-sub message bus with persistent storage
A stream processing platform (via Kafka Streams and ksqlDB)
A system that trades latency for durability and throughput

Here's what Kafka isn't:

A real-time message queue like RabbitMQ (Kafka's latency floor is ~2-5ms, RabbitMQ can do microseconds)
A database (it doesn't support random reads or complex queries)
An ETL tool out of the box (though people use it as one)

Is apache kafka an etl? Technically no. Practically? Half the data pipelines I see are "Kafka → something else" patterns. Companies use Kafka as the transport layer, then bolt on Kafka Connect for source/sink connectors. It's not ETL in the traditional sense—it's more like ELT with attitude.

The Kafkaesque Question

Let me pause and address the elephant in the room.

You've probably heard the term "Kafkaesque." It comes from Franz Kafka, the early 20th-century writer whose work features bureaucratic nightmares, absurd systems, and protagonists trapped in incomprehensible machinery. Franz Kafka (1883-1924) - PMC documents his life—a man who worked as an insurance clerk by day and wrote surreal fiction by night.

There's a joke in our industry: managing a Kafka cluster is the most Kafkaesque experience possible. You think you've configured the replication factor correctly. Then a broker goes down. Your consumers start lagging. The logs are unreadable. You're trapped in a system you don't fully understand, and no one can explain why the partitions aren't rebalancing.

The irony isn't lost on me.

What is kafka's most famous story? asks this question on existential literature forums. Most people say The Metamorphosis—a traveling salesman wakes up as a giant insect and spends the rest of the story trying to maintain normalcy. That's every Kafka admin I've ever met. Your cluster has become a cockroach, and you're still trying to get the offset commits working.

But Kafka's story that resonates most with infrastructure engineers? The Trial. A man is arrested for a crime no one will explain. He spends the entire book trying to navigate an opaque judicial system. That's debugging consumer group rebalancing on a high-traffic cluster at 2 AM.

Let's be honest: Kafka's complexity isn't accidental. It's the price of what it does well.

The 2026 State of Kafka

Here's the data point that matters:

Confluent reported $890M in revenue in 2025. That's up 28% year-over-year. They have 4,500+ customers.

Redpanda, the main competitor, has raised $170M total. They're growing fast. But they're not replacing Kafka in the enterprise—they're winning the developers-who-hate-ZooKeeper market.

Uber still runs Kafka. So do Netflix, Spotify, LinkedIn, and every major bank I've worked with. In 2025, I audited a system for a European payments company: 12 Kafka clusters, 2,400 topics, 180 TB/day throughput. They'd been running it since 2019. They weren't migrating off it.

Is apache kafka still used? In production, at scale, globally? Yes. Absolutely.

But the interesting question is how it's being used, and where it's losing ground.

Where Kafka Still Dominates

1. Event Sourcing and CQRS

Kafka's append-only log is perfect for event sourcing. You write events, you never delete them, you replay them when needed. This pattern is everywhere now—financial systems, inventory management, audit trails.

I built a system for a logistics company in 2024. They needed to track every state change for 50,000 shipments per hour. Kafka's log compaction meant we could keep the full history and still serve current state. No database could do that efficiently.

2. Data Pipelines and Microservices Integration

This is Kafka's bread and butter. You have Service A producing events, Service B consuming them, and you need exactly-once semantics and ordering guarantees.

Most people think they need Kafka for this. They're wrong if they have fewer than 10 microservices or fewer than 5,000 events/second. RabbitMQ or NATS will serve you better. But once you cross that threshold, Kafka's partitioning model and consumer groups become the difference between a system that works and a system that collapses.

3. Log Aggregation and Metrics

The original use case. In 2026, every observability platform worth using ingests through Kafka. Datadog? Kafka on the backend. New Relic? Kafka. Grafana Loki? Kafka underneath.

I set up a monitoring stack for a SaaS company last year: Fluent Bit → Kafka → ClickHouse. We were ingesting 300 GB of logs daily. Kafka buffered the spikes. ClickHouse handled the queries. Without Kafka in the middle, we'd have lost data every time ClickHouse got overloaded (which was often).

4. Production AI Systems

This is the new frontier. Everyone's building AI agents, RAG pipelines, and real-time inference systems. Kafka is the nervous system.

Here's a pattern I'm seeing more often:

User submits a request → API gateway → Kafka topic
Multiple AI services consume from that topic (one for intent classification, one for entity extraction, one for response generation)
Results get published to output topics
A consumer aggregates and returns the response

Kafka's durability means you don't lose user requests when an AI service crashes. Its partitioning means you can scale each AI service independently. Its replayability means you can reprocess failed requests.

I'm working with a company that processes 200,000 AI inference requests per hour through Kafka. They chose it over Redis Streams because they needed guaranteed delivery and long-term retention. Redis would have lost data on broker failure. Kafka doesn't.

Where Kafka Is Losing

1. Sub-10ms Latency Use Cases

Kafka's minimum latency is around 2-5ms under ideal conditions. Real-world production clusters? 10-20ms typical.

If you need microsecond latency—real-time trading, gaming leaderboards, live streaming control signals—Kafka isn't your tool. Use NATS, Redpanda, or raw TCP sockets.

Redpanda has been eating Kafka's lunch here. They rewrote the Kafka protocol in C++, removed ZooKeeper, and got latency down to single-digit milliseconds. I've tested it. It's faster. But it also uses more memory per partition and has fewer operational tools.

2. Small Teams or Single-Service Applications

Do not run Kafka for a side project. Do not run it for your startup's MVP. Do not run it for a team of three.

I've seen this mistake dozens of times. Someone reads "event-driven architecture" on a blog, spins up Kafka, and spends two weeks configuring ZooKeeper authentication. Meanwhile, they could have shipped with RabbitMQ in an afternoon.

Kafka is a platform. Platforms require dedicated ops. If you don't have someone who understands consumer group rebalancing, partition leader election, and log compaction, you're going to have a bad time.

3. Simple Pub-Sub

If you need "send message to one consumer, get it once, move on," Kafka is overkill. Its strength is complex routing, replayability, and long-term storage. If you don't need those, use Redis Pub/Sub or RabbitMQ.

I've seen teams adopt Kafka for a simple notification system. They ended up with a five-broker cluster, 20 topics, and a consumer group configuration that took two weeks to debug. The alternative? A Redis list with a single consumer. Five lines of code.

The Alternatives (Honest Comparison)

Redpanda

API-compatible with Kafka (you can use Kafka clients)
No ZooKeeper (built-in Raft consensus)
Better latency, lower operational overhead
More expensive per GB stored (in-memory performance comes at a cost)
Fewer ecosystem integrations (Confluent Schema Registry doesn't work natively)

Verdict: Best for latency-sensitive workloads and teams that hate managing ZooKeeper. Not a drop-in replacement if you use Kafka Streams or ksqlDB extensively.

WarpStream

Kafka protocol compatible, but runs on object storage (S3, GCS)
No local disks. No ZooKeeper. No brokers.
Infinite storage, elastic scaling
Higher latency (S3 API calls are slow)
You pay per query, not per node

Verdict: Best for infrequent access patterns and cost-sensitive analytics workloads. Terrible for real-time streaming.

Pulsar

Separate compute and storage layers
Native multi-tenancy
Better geo-replication than Kafka
Higher complexity than Kafka

Verdict: Niche. I've seen it used well in large telecommunications companies. Almost nowhere else.

NATS

Sub-millisecond latency
Simple, lightweight
At-most-once delivery by default
No persistent storage

Verdict: Great for IoT, messaging, and real-time control. Not a Kafka replacement.

Real Code: Three Patterns I Use Daily

Pattern 1: Kafka as Event Store

python
from confluent_kafka import Producer, Consumer, KafkaError
import json

# Producer: Write events to the event store
producer = Producer({
    'bootstrap.servers': 'localhost:9092',
    'acks': 'all',  # Wait for all replicas to acknowledge
    'enable.idempotence': True  # Exactly-once semantics
})

def write_event(user_id, event_type, data):
    event = {
        'user_id': user_id,
        'event_type': event_type,
        'data': data,
        'timestamp': int(time.time() * 1000)
    }
    producer.produce(
        topic='user-events',
        key=str(user_id),
        value=json.dumps(event),
        callback=lambda err, msg: 
            print(f'Failed: {err}') if err else None
    )
    producer.flush()

Pattern 2: Kafka Streams for Real-Time Aggregation

java
StreamsBuilder builder = new StreamsBuilder();

KStream<String, Transaction> transactions = 
    builder.stream("transactions", 
        Consumed.with(Serdes.String(), transactionSerde));

KGroupedStream<String, Transaction> groupedByUser = 
    transactions.groupByKey();

// 1-minute tumbling window: total spending per user
KTable<Windowed<String>, Double> spendingTotals = 
    groupedByUser
        .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
        .aggregate(
            () -> 0.0,
            (key, transaction, total) -> total + transaction.amount,
            Materialized.with(Serdes.String(), Serdes.Double())
        );

// Alert on high-value transactions
transactions
    .filter((key, tx) -> tx.amount > 10000)
    .to("high-value-alerts");

Pattern 3: Kafka Connect for ETL (Yes, People Call It ETL)

yaml
# PostgreSQL → Kafka → S3 pipeline via Kafka Connect
{
  "name": "postgres-orders-sink",
  "config": {
    "connector.class": "io.confluent.connect.s3.S3SinkConnector",
    "tasks.max": "4",
    "topics": "orders",
    "s3.bucket.name": "data-lake-orders",
    "s3.region": "us-east-1",
    "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
    "partitioner.class": "io.confluent.connect.storage.partitioner.DailyPartitioner",
    "flush.size": "10000",
    "rotate.interval.ms": "60000"
  }
}

This runs on every cluster I manage. Kafka Connect is janky (error handling is terrible), but it works reliably once configured.

The Hard Truth (From Experience)

I've been running Kafka in production since 2019. Here's what I wish someone told me:

Kafka's reliability is a lie if you don't configure it right.

Default settings are terrible. acks=1 by default loses data on broker failure. auto.offset.reset=latest means new consumers start at the end of the topic, missing everything before. enable.auto.commit=true means your consumer can commit offsets before processing finishes.

Every billion-dollar Kafka outage I've seen or heard about came from misconfiguration, not from Kafka itself.

ZooKeeper is still the bottleneck.

Kafka removed ZooKeeper in KRaft mode (KIP-500). It's production-ready in 3.x versions. But migration is painful. I've migrated three clusters. Two went smoothly. One had a split-brain scenario that took 18 hours to resolve.

If you're starting fresh in 2026, use KRaft. If you have an existing ZooKeeper-based cluster, budget two months for migration and test it thoroughly.

You will need at least three brokers for production.

One broker for development. Three for "production." Five if you need disaster recovery across regions. More if you're doing 100K+ events/second per broker.

I've seen startups run Kafka on two brokers. It always fails eventually. The replication just doesn't work.

The 2026 Verdict

Is apache kafka still used? Yes. Extensively. In every major company and most mid-size ones doing serious data work.

But its role is narrowing. It's no longer the default choice for everything. It's the right choice for:

Systems processing >10K events/second
Architectures requiring replayability and exactly-once semantics
Enterprise environments where reliability matters more than developer convenience
AI pipelines that need durable, scalable event ingestion

It's the wrong choice for:

Prototypes and MVPs
Sub-10ms latency applications
Small teams without dedicated infrastructure support

The Kafkaesque irony? Kafka itself has become the bureaucratic machinery it was named after. It's powerful. It's reliable. And it's absolutely miserable to manage without a dedicated team.

But that doesn't make it obsolete. It makes it mature. And in 2026, mature infrastructure is worth more than shiny alternatives.

Pick your tool based on your problem, not your ideology.

Frequently Asked Questions

Is Apache Kafka still relevant in 2026?

Yes. Kafka remains the most widely adopted distributed streaming platform in production. Confluent's $890M revenue in 2025 and continued adoption by companies processing >100K events/second confirm it's not going anywhere. It's losing ground in low-latency niches to Redpanda, but dominating in data pipelines, event sourcing, and AI infrastructure.

Is Apache Kafka an ETL tool?

No. Kafka is a distributed log and messaging system. But it's commonly used as the transport layer in ETL pipelines via Kafka Connect. The pattern is: source database → Kafka (transport) → sink target (data lake, warehouse, search index). It's more accurate to call it ELT with streaming—Kafka stores the raw data, and downstream systems transform it.

What is Kafka's most famous story?

The most famous story by Franz Kafka (the author, not the software) is The Metamorphosis, where a traveling salesman wakes up as a giant insect. The parallel with Kafka the software is unintentional but apt—managing a Kafka cluster often feels like waking up as something incomprehensible.

What's replacing Kafka?

Nothing replaces Kafka entirely. Redpanda replaces it for latency-sensitive workloads (removing ZooKeeper, adding C++ performance). WarpStream replaces it for cost-sensitive analytics (running on object storage). Pulsar replaces it for multi-tenant enterprise deployments. But none have Kafka's ecosystem maturity, tooling, and community.

Can I use Kafka for real-time AI inference?

Yes, and increasingly you should. Kafka's durability and partitioning make it ideal for production AI pipelines. A common pattern in 2026: user requests → Kafka topic → multiple AI services consuming in parallel → results aggregated to response topic. Kafka ensures no requests are lost and each service can scale independently.

How many brokers do I need?

Minimum one for development. Minimum three for production (for replication). Minimum five for multi-zone disaster recovery. Every broker failure in a two-broker cluster will cause data loss. I've seen it happen. Don't cut corners here.

What's the biggest mistake teams make with Kafka?

Misconfiguration. Default settings prioritize convenience over reliability. Most outages come from acks=1, auto.offset.reset=latest, or auto-commit enabled. Spend time understanding your consumer group configuration. Test failure scenarios. Your cluster will fail eventually—design for that.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.