Kafka Explained: The Event Backbone of Big Systems
Introduction When one service needs to tell many others that something happened — an order placed, a payment completed, a sensor reading taken — calling each service directly gets fragile fast. Apache Kafka, created at LinkedIn in 2011 and now an Apache project, solves this with an event log: services publish events to named topics, and any number of consumers read them at their own pace. It is the backbone behind fraud detection, analytics pipelines, and real-time feeds at companies far larger than yours — and understanding it explains how modern systems stay loosely coupled at scale. Topics, partitions, producers, and consumers Kafka's model has four pieces: Topics — named, append-only logs of events, like orders or payments . Events are never edited or deleted in place; new events append. Partitions — each topic splits into partitions, ordered logs that spread across servers, so a topic scales beyond one machine. Events with the same key land in the same partition, pres...