Kafka vs Pulsar Comparison: What 5 Years of Production Hell Taught Me
January 2021. I was at a client – a fintech startup processing 50,000 transactions per second – trying to convince them that Kafka was the obvious choice for their event streaming layer. They had a small ops team. No dedicated SRE. I told them "Kafka is mature, everyone uses it, you'll be fine."
By March 2023, they had suffered three partition rebalances that took down their entire real-time fraud detection pipeline. Two of those were during peak hours. The CTO asked me if I'd ever read Franz Kafka. He meant the writer. He was being ironic. The absurdity of existence, the bureaucratic nightmare – he felt that the tech was living up to its namesake (Franz Kafka).
That's when I got serious about Pulsar.
I'm Nishaant Dixit. I run SIVARO, a product engineering shop that builds data infrastructure for companies pushing 200K events per second. I've spent the last five years knee-deep in both Kafka and Pulsar deployments. This article is what I wish someone had told me in 2020.
You're not choosing between two message queues. You're choosing between two philosophies of distributed systems – and one of them has a steeper learning curve that can swallow your team whole.
Here's what we'll cover:
- The architectural decisions that matter (not the ones vendors tell you about)
- A complete kafka vs pulsar comparison from someone who's burned his hands on both
- How to build a Kafka producer in Python – with the pitfalls I see engineers make daily
- What a Kafka consumer group actually does (and why it breaks when you least expect it)
- When Pulsar saves your ass, and when it just adds complexity
Let's go.
The Absurdity of Kafka (Yeah, I'm Going There)
Most people think Kafka's complexity is a feature. It's not – it's a consequence of a design that was never intended to be a general-purpose messaging system.
Kafka was built at LinkedIn for log aggregation. It worked beautifully for that. Then everyone started using it for everything: event sourcing, stream processing, command-queues, pub-sub. The platform grew into a monster.
Here's the thing nobody tells you about Kafka's architecture: the coupling between compute and storage.
In Kafka, each broker holds both the data and the processing responsibility for the partitions assigned to it. Want to add more consumers? Hope you have enough brokers to handle the data load and the consumer groups. Want to rebalance? We'll just freeze all consumption for a few seconds while we move partitions around. Hope your latency-sensitive consumers can handle that.
I've seen this go sideways at least a dozen times. One client – a gaming analytics company – had a consumer group with 200 partitions. Every time they deployed new code, the rebalance took 30+ seconds. Their dashboard went blank. Users complained. They blamed the engineers. The engineers blamed the rebalance. It was pure Kafkaesque – a bureaucratic, opaque process with no clear exit (Franz Kafka & Kafkaesque).
And yet, Kafka is still the default. Why? Ecosystem. Documentation. And because "nobody got fired for buying Kafka."
But if you're building a system that needs to scale beyond a few dozen brokers, or you need true multi-tenancy without custom infrastructure, Kafka starts to creak.
Confluent has improved things with KRaft (removing ZooKeeper) and tiered storage. But the fundamental coupling remains. As of July 2026, KRaft is production-ready in Confluent Platform 7.9 – but it doesn't fix the rebalance problem. The existential dread of EventListener still haunts Java developers everywhere (The Absurdity of Existence: Franz Kafka and Albert Camus).
Pulsar's Architecture: Not Just Another Messaging System
Here's where Pulsar flips the script.
Pulsar separates compute from storage. Clean. Explicit. Non-negotiable.
You have a layer of brokers that handle produce/consume logic – stateless, easy to scale horizontally. Then you have a separate storage layer built on Apache BookKeeper (bookies) that holds the actual data. The two don't have to scale together. Need more throughput? Add brokers. Need more storage? Add bookies.
This isn't just academic. Last year at SIVARO, we migrated a logistics client from a 50-broker Kafka cluster to a 12-broker Pulsar cluster with 6 bookies. Same throughput (150K msg/sec). Better latency P99. And we stopped worrying about disk space on individual brokers – with BookKeeper, data is distributed evenly across all bookies via segment-oriented storage. No hot spots. No manual rebalancing.
Pulsar also has native multi-tenancy. You can have hundreds of tenants sharing the same cluster with strict resource isolation (namespaces, quotas, auth). Kafka's quotas exist but don't really isolate – one noisy consumer can still degrade neighbors. I've seen it. You haven't lived until you've debugged why your production topic's lag spiked because someone in another team was reading from a different topic with auto.offset.reset=earliest on a rebalance.
And then there's geo-replication. Pulsar does it at the cluster level using built-in replication mechanisms. Kafka requires MirrorMaker or Confluent Replicator – both extra operational burden.
But here's the catch: Pulsar's operational complexity is different, not lower. You need to run ZooKeeper for BookKeeper metadata anyway. You need to understand segment storage, ledger recovery, and the bookie's GC behavior. The learning curve for operators is real.
At first I thought this was a branding problem – turns out it was a documentation problem. In 2026, Pulsar's docs are better, but still not Kafka-level. You'll spend more time on Stack Overflow. That's a trade-off.
kafka vs pulsar comparison: The Real Differences
Let me cut through the vendor hype. Here's the table that matters.
| Aspect | Kafka | Pulsar |
|---|---|---|
| Partition rebalance impact | High – all consumers blocked during rebalance | Low – consumers hold subscriptions, not partitions |
| Exactly-once semantics | Only in Kafka Streams or with EOS config (limited) | Built-in for all consumers (effective-read-once) |
| Geo-replication | MirrorMaker (complex) or paid replicator | Native built on BookKeeper |
| Multiple consumers per topic | Single consumer group per partition (one active consumer per partition) | Multiple consumers (shared, key-shared, failover) – one partition can be consumed by many |
| Data retention | Configurable per topic, but tied to disk on a single broker | Infinite retention through tiered storage (S3, GCS) |
| Exactly-once for producers | Requires idempotent producer config + acks=all |
Native deduplication at the producer level via message ID |
| Operational maturity | Very mature, lots of tooling | Good but smaller community – fewer monitoring tools |
The biggest practical difference? Consumer elasticity.
In Kafka, if you have 10 partitions and 20 consumers, only 10 will be active. The rest sit idle. To scale consumption, you need more partitions – which means repartitioning, which is a migration.
In Pulsar, you can have 20 consumers reading from 10 partitions (shared subscription). Messages are distributed across consumers. No wasted resources. This is huge when you have variable load or bursty workloads.
I've seen a Pulsar cluster absorb a 10x traffic spike without any rebalance. Kafka? We would have had to pre-provision partitions, pay for brokers we didn't use, and still risk rebalance chaos.
But here's the flip side: order guarantees in shared subscriptions require key-shared subscriptions, which add overhead. If you need strict order per partition across multiple consumers, Kafka's "one consumer per partition" is simpler.
Kafka Consumer Group Explained (With the Bit Nobody Tells You)
This gets butchered so often. Let me fix it.
A Kafka consumer group is a set of consumers that cooperate to consume from a set of partitions. Each partition is assigned to exactly one consumer in the group. That's it.
How it works under the hood:
- Consumers join the group by sending a
JoinGrouprequest to the group coordinator (one of the brokers). - The coordinator picks a group leader (first consumer that joined, or random).
- The leader gets the full list of members and partitions and runs a partition assignment strategy (range, round-robin, sticky, cooperative sticky).
- The leader sends the assignment back to the coordinator, which distributes it to each consumer.
Now the part that breaks: rebalancing.
Every time a consumer joins, leaves, or a partition is added, the entire group rebalances. All consumers revoke their partitions. Stopping consumption. Then reassign. This is fine for batch workloads. Terrible for streaming apps.
The code example:
python
from kafka import KafkaConsumer
import json
def consume():
consumer = KafkaConsumer(
'events',
bootstrap_servers=['localhost:9092'],
group_id='fraud-detection-group',
enable_auto_commit=True,
auto_offset_reset='earliest',
# This is the default – but it's the cause of rebalance pain
# Use cooperative rebalance to reduce impact
partition_assignment_strategy='cooperative-sticky'
)
print("Consumer started. Waiting for messages...")
for message in consumer:
data = json.loads(message.value)
# simulate processing
print(f"Partition: {message.partition}, Offset: {message.offset}, Key: {message.key}")
# Don't call consumer.commit() manually if auto_commit is True
The fix I always recommend: cooperative sticky rebalancer (introduced in Kafka 2.4). Instead of stopping all consumers, it rebalances incrementally – only consumers that need to change partitions stop. This cut our rebalance time from 30 seconds to under 2 seconds.
But even cooperative rebalancing doesn't help if you have hundreds of partitions. Each rebalance still involves a JoinGroup round trip. For real-time systems, Pulsar's approach (no rebalance at the partition level) wins.
https://kafka.apache.org/documentation/#consumercooperation – read this before deploying. Seriously.
How to Build a Kafka Producer in Python
Here's a pattern I use in production. The key is idempotence=True and acks=1 (or all if you need exactly-once, but that adds latency). Don't use acks=0 unless you're okay losing data.
python
from kafka import KafkaProducer
import json
def create_producer():
producer = KafkaProducer(
bootstrap_servers=['broker1:9092', 'broker2:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
key_serializer=lambda k: k.encode('utf-8'),
acks=1, # Wait for leader ack
retries=5, # Retry on transient errors
max_in_flight_requests_per_connection=5,
enable_idempotence=True, # Prevent duplicates on retry
linger_ms=10, # Batch for efficiency – tune this
batch_size=65536 # 64KB default, increase for larger batches
)
return producer
def send_event(producer, topic, key, value):
future = producer.send(topic, key=key, value=value)
# Block for result – or use async callbacks in high-throughput
record_metadata = future.get(timeout=10)
print(f"Sent to topic {record_metadata.topic}, partition {record_metadata.partition}, offset {record_metadata.offset}")
if __name__ == "__main__":
p = create_producer()
send_event(p, "user-events", "user-123", {"action": "login", "timestamp": "2026-07-21T10:00:00Z"})
p.flush()
p.close()
Common mistakes I see:
- No idempotence. Without it, a retry after a timeout can duplicate the message. Yes, it's possible even with
acks=allif the broker crashes after committing the offset. Turn on idempotence. It's free in modern Kafka versions. - Wrong linger_ms. Setting it to 0 means no batching – high CPU, low throughput. But setting it too high adds latency. For real-time alerting, I use
linger_ms=5. For bulk ingestion,linger_ms=50. - Using synchronous
send().get()inside a tight loop. That becomes serial. Use asyncio orcallbackfor parallelism.
If you're building a producer for high throughput (say 100K+ msg/sec), use a custom partitioner. The default murmur2 hash is fine, but if you want to route based on some business key, implement your own.
python
class CustomPartitioner:
def __init__(self):
self.partition_count = None
def __call__(self, key, all_partitions, available_partitions):
if self.partition_count != len(all_partitions):
self.partition_count = len(all_partitions)
if key is None:
return all_partitions.random()
# Simple hash – but you could look up a partition mapping
import hashlib
return int(hashlib.md5(key.encode()).hexdigest(), 16) % len(all_partitions)
When Pulsar Wins (and When It Doesn't)
Win #1: Multi-tenant SaaS platforms. We built a customer event pipeline for a B2B analytics company. Each tenant had its own namespace. Quotas per namespace. No noisy neighbor problem. In Kafka, we would have had to run separate clusters per customer tier – or build a complex proxy layer.
Win #2: Variable throughput with predictable latency. Pulsar's segmentation means you can have huge backlogs without affecting new writes. BookKeeper writes to current segment while older segments are flushed. Kafka's single active segment per partition can cause write stalls if the broker's page cache is full.
Win #3: You need geo-replication without paying for Confluent. We have a client with clusters in US, EU, and APAC. Pulsar's built-in replication across clusters is straightforward to configure. Kafka requires MirrorMaker 2 (which works, but adds operational monitoring, checkpoints, offset sync issues).
When Pulsar doesn't win:
- Your team is already deep in Kafka. Migration costs are real. Don't switch just because Pulsar looks shinier.
- You need the ecosystem. Kafka Connect has hundreds of connectors, KSQL, Streams API. Pulsar Functions and Pulsar IO are improving but not as broad.
- Small deployments (1-3 brokers). Pulsar adds BookKeeper overhead. For 10K msg/sec, Kafka with 3 brokers is simpler.
- You need strict ordering across multiple consumers per partition. In shared subscription mode, Pulsar distributes messages across consumers – order is only guaranteed for key-shared. If you absolutely need global order per partition, Kafka's one-consumer-per-partition model is cleaner.
The Kafkaesque Trap
I'm stealing a line from philosophy: "The Kafkaesque is a condition where the system itself becomes incomprehensible to the people inside it" (Franz Kafka - Existential Primer).
That's exactly what happens to teams who over-engineer their Kafka deployment.
You start with 3 partitions. Then 10. Then 50. Then you add schema registry, Kafka Streams, connectors from 7 different vendors. Your server.properties file is 400 lines. Nobody remembers why offsets.topic.replication.factor is set to 3. The last person who understood the consumer rebalance strategy left the company.
At some point, the system serves you, not the other way around.
Pulsar can fall into the same trap – you can overcomplicate with multiple clusters, tiered storage, and custom subscriptions. But its architectural separation makes it easier to reason about.
If you're reading Kafka's biography (The Best 5 Books to Read), you'll notice a pattern: his characters are trapped by bureaucracy they can't escape. Don't let your event streaming platform become that.
FAQ: Kafka vs Pulsar Comparison (Real Questions from Clients)
Q: Should I switch from Kafka to Pulsar in 2026?
A: Only if you have a specific pain – unpredictable rebalance, multi-tenancy, or geo-replication. Otherwise, Kafka is fine. Migration costs 2-3 months of engineering time.
Q: Is Pulsar production-ready?
A: Yes. Version 3.3 is stable. We've run it in production since 2023. But you need ops experience. Don't let a junior team manage the bookies.
Q: How do I decide which one to start with?
A: New project, small team? Start with Kafka (easier to hire for). Large team, complex scaling requirements? Pulsar's architecture will save you later.
Q: What about performance – Kafka vs Pulsar latency?
A: At low throughput (<50K msg/sec), they're comparable. At high throughput, Pulsar's decoupled architecture can give better P99 latency because bookies handle storage independently. But benchmark your own workload.
Q: Does Pulsar support exactly-once?
A: Yes, for both produce (message deduplication) and consume (effectively-read-once by default). Kafka requires extra config and is only available in clients that support EOS.
Q: Can I use Kafka Streams with Pulsar?
A: No, Kafka Streams is tied to Kafka consumer internals. Pulsar has its own stream processing (Pulsar Functions, Flink integration). If you need Kafka Streams' DSL, stick with Kafka.
Q: What's the biggest gotcha with Pulsar?
A: BookKeeper's GC can cause latency spikes if you have high throughput with large message sizes (>100KB). Monitor bookie's org.apache.bookkeeper.bookie.Bookie metrics – if GC time exceeds 1 second, tune the read-ahead cache or add more bookies.
Q: How do I handle consumer rebalancing in Kafka?
A: Use cooperative sticky rebalancer, limit partition count per consumer to <50, and avoid auto.offset.reset=earliest in production. For critical apps, implement a custom rebalance listener that drains in-flight work before revoking.
Final Thought
I've lost count of how many times I've seen a perfectly good Kafka deployment turned into a nightmare by over-customization. And I've also seen Pulsar adoption fail because the ops team didn't understand BookKeeper's internals.
There's no single right answer.
What matters is that you pick a system that matches your team's skill set and your scaling trajectory. If you're processing 50K events per second and expect to double every year, Pulsar's separation of storage and compute will save you money and headaches. If you're a startup with 5K events per second and a Kafka-savvy team, don't overthink it.
The kafka vs pulsar comparison isn't about who has more features. It's about who has fewer surprises in production.
I've been surprised enough.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.