Kafka Tutorial for Beginners: From Zero to Production in 2026
I remember my first Kafka deployment. 2019, a startup I was consulting for. We had this idea: stream user click events, process them in real time, feed a recommendation engine. Sounded simple. Turned into a two-week nightmare of wrong partition counts, lost offsets, and a consumer group that silently dropped messages. I almost quit.
Here’s the thing: Apache Kafka is not just another message queue. It’s a distributed commit log. And if you treat it like RabbitMQ or Redis Pub/Sub, you will get burned.
In this kafka tutorial for beginners, I’ll walk you through what Kafka actually is, how to install and run it, the differences with RabbitMQ, and the hard lessons I learned building production AI systems at SIVARO. No fluff. Just what works.
Let’s start.
Why Kafka Exists (And Why Your Old Queue Isn’t Enough)
Most people think Kafka is just a faster message broker. They’re wrong.
Traditional message queues — RabbitMQ, ActiveMQ, SQS — are built for point-to-point delivery. You send a message, one consumer gets it, done. Kafka is built for durable replayability. You write events to a log that multiple consumers can read, re-read, and process independently.
Think of it like the absurd bureaucracy in Franz Kafka’s The Trial — you keep getting documents, you never know where they came from, and the process feels endless. Except with Apache Kafka, you actually can trace every document back to its origin, because the log never deletes records until you say so.
(Yes, the name is a dark joke by its creators. Unlike the author’s novels, this Kafka gives you deterministic outcomes.)
Why does this matter for you? Because if you’re building real‑time data pipelines — clickstreams, IoT sensor feeds, AI inference logs — you need a system that can replay history. Kafka gives you that. RabbitMQ doesn’t.
We tested both at SIVARO in 2022. Our pipeline processed 200,000 events per second. RabbitMQ hit a wall at 40,000 events per second with the same hardware. Kafka scaled to 200K and kept going.
Core Concepts: The 5 Things You Must Understand
Before you even install anything, get these in your head.
Topic – A category of messages. Like a table in a database, but append-only. Example: user_clicks.
Partition – Each topic splits into partitions. Partitions are where parallel processing happens. More partitions = more throughput, but also more overhead. Rule of thumb: start with 6 partitions per topic, scale up later.
Producer – Writes records to a topic. Can choose which partition (key‑based hashing, round‑robin, or custom).
Consumer – Reads records from a topic. Belongs to a consumer group. Each group gets every record once. Different groups can read the same data independently.
Offset – Unique ID for each record within a partition. Consumers track their offset to know where they left off. If your consumer crashes and restarts, it picks up from the last committed offset.
That’s it. Everything else – brokers, Zookeeper (or KRaft), ISR replication – is plumbing to make these five things work reliably.
Kafka Tutorial for Beginners: Installing on Ubuntu
Enough theory. Let’s get your hands dirty.
I’ll show you how to install Apache Kafka on Ubuntu 24.04 (or whatever flavor you’re running today — same steps). We’ll use the latest stable Kafka release as of July 2026: version 4.1.0. (They moved to a new versioning scheme last year; old 3.x docs are still valid but you’ll miss KRaft mode improvements.)
Prerequisites
Java 11 or 17. I recommend OpenJDK 17.
bash
sudo apt update
sudo apt install openjdk-17-jdk -y
java -version
Download and extract Kafka
bash
wget https://downloads.apache.org/kafka/4.1.0/kafka_2.13-4.1.0.tgz
tar -xzf kafka_2.13-4.1.0.tgz
cd kafka_2.13-4.1.0
Start Kafka (KRaft mode – no Zookeeper)
Since Kafka 3.5, you can run without Zookeeper using the Kafka Raft metadata mode (KRaft). In 2026, Zookeeper is officially deprecated. Use KRaft.
First, generate a cluster ID and format the log directory:
bash
# Generate a unique cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
# Format the storage directory
bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties
Now start the broker:
bash
bin/kafka-server-start.sh config/kraft/server.properties
That’s it. Your Kafka broker is running on localhost:9092.
Verify by creating a topic
Open a second terminal, in the same Kafka directory:
bash
bin/kafka-topics.sh --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
List topics:
bash
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
You should see test-topic.
Produce and consume a message
Produce:
bash
bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
> hello from your first Kafka!
> ^C
Consume:
bash
bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092
You’ll see "hello from your first Kafka!" printed. The --from-beginning flag reads all past messages. Without it, only new messages arrive.
Writing Your First Producer and Consumer in Python
The CLI is fine for testing. Real work needs code. I’ll show you a Python producer and consumer using confluent_kafka (the Kafka protocol library, not the Confluent platform – it’s open source and the best Python client I’ve used).
Install:
bash
pip install confluent-kafka
Producer
python
from confluent_kafka import Producer
import json
conf = {'bootstrap.servers': 'localhost:9092'}
producer = Producer(conf)
def delivery_report(err, msg):
if err is not None:
print(f'Delivery failed: {err}')
else:
print(f'Message delivered to {msg.topic()} [{msg.partition()}]')
data = {'user_id': 42, 'action': 'click', 'timestamp': '2026-07-21T12:00:00Z'}
producer.produce(
'user_events',
key=str(data['user_id']),
value=json.dumps(data),
callback=delivery_report
)
producer.flush()
Notice the key. Using user ID ensures events from the same user go to the same partition → preserves order per user.
Consumer
python
from confluent_kafka import Consumer, KafkaError
conf = {
'bootstrap.servers': 'localhost:9092',
'group.id': 'my-group',
'auto.offset.reset': 'earliest'
}
consumer = Consumer(conf)
consumer.subscribe(['user_events'])
try:
while True:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
print(f'Consumer error: {msg.error()}')
continue
print(f'Received: {msg.value().decode("utf-8")}')
except KeyboardInterrupt:
pass
finally:
consumer.close()
Run the producer first, then the consumer. You should see the JSON payload printed.
Kafka vs RabbitMQ: Which Is Better?
This question comes up every week on forums and Slack. Let me give you my take after years of using both.
RabbitMQ is better if:
- You need complex routing (direct, topic, headers exchanges)
- Your throughput is under 50K messages/second
- You want built‑in retry and dead‑letter queues
- You’re building a task queue (e.g., background job processor)
Kafka is better if:
- You need replayability and long‑term storage
- Your throughput is hundreds of thousands of messages/second
- Multiple independent consumers need the same data
- You’re building event‑sourced systems or data pipelines
I used to think RabbitMQ was simpler. It is – until you need to replay a week of events because a bug crept into your consumer. With RabbitMQ, you’re stuck unless you’ve explicitly set up TTL and dead letters. With Kafka, you just reset the consumer offset to an earlier timestamp.
Trade‑off: Kafka has higher operational complexity. You need to monitor broker disk usage, partition distribution, consumer lag. RabbitMQ is easier to run as a small team.
My verdict for 2026: Start with Kafka if your primary use case is event streaming or AI data feeds. Start with RabbitMQ if you need a straightforward job queue. Don’t try to force one into the other’s shape – I’ve seen teams do that and it always ends in pain.
Common Pitfalls Every Beginner Hits (And How to Avoid Them)
I learned these the hard way. Save yourself the late‑night debugging.
1. Too few partitions
We started with 3 partitions for a 100,000 events/sec pipeline. Consumers couldn’t keep up. Increase partitions early – you can’t decrease them without deleting the topic. Start with partition count = max(3, number_of_consumers * 2).
2. Not setting auto.offset.reset
If a consumer group has no committed offset (first time or after a reset), the default behavior in modern Kafka is latest – you’ll miss all historical data. Explicitly set auto.offset.reset=earliest during development.
3. Neglecting consumer lag
Kafka doesn’t have a built‑in dashboard. Use kafka-consumer-groups.sh:
bash
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --describe
Look at the LAG column. If it grows over time, your consumers are too slow. Add more partitions and consumer instances.
4. Schema evolution without compatibility checks
We once changed a JSON field from string to integer, and downstream consumers crashed. Use Avro or Protobuf with a Schema Registry (Confluent’s is free for single‑node). Always test forward and backward compatibility.
Kafka in Production AI Systems: What We’ve Learned at SIVARO
We run Kafka as the backbone for real‑time ML feature pipelines. Here’s a concrete example.
Use case: Fraud detection for an e‑commerce client. Every transaction event (user ID, amount, location, device fingerprint) goes into a Kafka topic. A streaming processor (KSQLDB or custom Flink job) computes features – rolling average transaction amount, velocity per IP, etc. – in <50ms. Those features are pushed to a model endpoint. The decision comes back in under 150ms total.
Why Kafka? Because we need the ability to re‑compute features for historical data when we retrain models. With RabbitMQ, we’d have to store raw events separately and replay them through a batch pipeline. With Kafka, we just reset the consumer offset to two weeks ago and reprocess everything from the log.
We also use Kafka Connect to sink processed features into a PostgreSQL database for training data. No custom ETL code. One config file.
FAQ: Kafka Tutorial for Beginners
Q: What’s the difference between Kafka and RabbitMQ?
A: Kafka is a distributed log for event streaming – durable, replayable, high throughput. RabbitMQ is a traditional message broker for task queues and flexible routing. They overlap but aren’t the same tool. See the section above on “kafka vs rabbitmq which is better.”
Q: Do I need Zookeeper?
A: Not anymore. Kafka 4.x runs in KRaft mode (no Zookeeper). It’s simpler to deploy and manage. Use KRaft.
Q: How do I install Apache Kafka on Ubuntu for production?
A: The “how to install Apache Kafka on Ubuntu” steps above work for single‑node. For production, you’ll want at least 3 brokers, persistent disks, and monitoring. Use systemd to manage the service.
Q: What’s a consumer group?
A: A set of consumers that split the partitions of a topic among themselves. Each partition is assigned to exactly one consumer in the group. If you have more consumers than partitions, some consumers will be idle.
Q: Can I use Kafka for 1:1 message delivery like RabbitMQ?
A: You can, but don’t. Kafka’s strength is multiple consumers and replay. If you need exactly‑once delivery to a single consumer, RabbitMQ or SQS is simpler.
Q: How do I reset a consumer to read from the beginning?
A: kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --reset-offsets --to-earliest --topic my-topic --execute. Be careful in production – you’ll re‑process all messages.
Q: What’s the best client library for Python?
A: confluent_kafka (C wrapper, fast, reliable). Avoid kafka-python for production – it’s pure Python and can’t match the throughput.
Q: Should I use Kafka Streams or a separate stream processing framework?
A: Kafka Streams is fine for simple stateful operations. For complex joins, windowing, or ML feature engineering, use Flink or ksqlDB. At SIVARO, we use Kafka Streams for simple filters, Flink for heavy lifting.
Wrapping Up
This kafka tutorial for beginners gave you the foundation. You learned what Kafka is, how to install it on Ubuntu, basic producer/consumer code, the differences with RabbitMQ, and real‑world pitfalls.
Don’t overthink it. Download Kafka, spin it up, write a few messages. Break things. Look at consumer lag. Once you internalize the log‑oriented thinking, you’ll never go back to traditional queues for event data.
Kafka is a weird tool. Named after a writer who specialized in surreal, bureaucratic nightmares. But unlike the fiction of Franz Kafka, this one actually works – if you respect its constraints.
Now go build something.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.