Kafka Mastery Path¶
Most people meet Kafka through a confusing analogy — "it's like a queue, but not really" — and never get the one idea that actually explains everything else: a Kafka topic is a durable, append-only log, split into partitions, and every consumer just remembers its own position in that log. Once that's real to you, a lot of things stop being magic: why two consumers in the same group never read the same partition, why "exactly once" needs a producer ID and a transaction coordinator and isn't free, why a compacted topic can still have old keys if you haven't read past a tombstone's retention window, and why a consumer that stops committing offsets isn't broken, it's just rewinding every time it restarts.
This course was written and run against a real, local Apache Kafka 4.4.0 broker in KRaft mode (no ZooKeeper — Kafka removed it entirely as of version 4.0) on an ordinary laptop with Java 17+. No cloud account, no managed Kafka service, and no Docker Compose cluster were needed for almost everything here: real topics, real producers and consumers, a real cgroup-free single-broker cluster, real Kafka Connect (including a genuine classpath gotcha this course hit and explains), and a real Kafka Streams application doing real stateful aggregation.
It assumes basic programming and command-line comfort. No prior messaging-system or distributed-systems experience is assumed, though if you want Java itself in depth first, see the Java Mastery Path.
How the program is organized¶
| Level | Focus | Modules | What you can do after |
|---|---|---|---|
| 1 · The Log, Topics & Partitions | What Kafka is, brokers and KRaft, topics/partitions, producers, consumer groups, replication, delivery semantics, retention | 10 | Run a real cluster and reason correctly about ordering and delivery |
| 2 · Operating a Cluster | Producer/consumer tuning, exactly-once, monitoring, log internals, security, reassignment, quotas | 10 | Configure and diagnose a cluster, not just use one |
| 3 · Kafka Connect & Kafka Streams | Connect architecture and the REST API, Streams topology, state stores, joins/windows, schema evolution | 10 | Build real ingestion pipelines and stream-processing apps |
| 4 · Production Kafka & Capstone | Performance, capacity planning, failure modes, disaster recovery, event-driven patterns, Kubernetes | 10 | Plan, operate, and troubleshoot Kafka in production |
Each level has 9 teaching modules plus a hands-on project (module 10), for 40 lessons total.
How to use this site¶
- Run it yourself. Download Kafka from kafka.apache.org, format storage with
kafka-storage.sh, and start a broker — no ZooKeeper, no account, no cloud bill. Java 17 or newer is all you need. - Read the "what was run" note at the top of each lesson. The overwhelming majority of this course's command output is real, observed output from a real broker; anything that genuinely couldn't be run locally (multi-datacenter replication at scale, a managed-service console) says so plainly.
- Do the "predict, then run" exercises. Kafka has several behaviors (rebalancing, compaction timing, exactly-once overhead) that are easy to mis-predict from the docs alone and genuinely clarifying to watch happen.
- Do the projects. Each one is a real, runnable pipeline or application, not a diagram.
More from the Mastery Path series¶
Free, structured, module-wise training across 97 other languages, platforms and disciplines — expand a category to browse: