Building a Distributed Log
A hands-on guide to building a Kafka-inspired, replicated distributed log from scratch — not just as an exercise, but to understand why Kafka is designed the way it is. Every design decision is motivated by a real production concern.
Start reading →What you'll build
- An append-only log with segments, sparse offset index, and memory-mapped I/O
- A binary wire protocol and length-prefixed TCP transport layer
- Cluster membership and failure detection with HashiCorp Serf
- Distributed metadata consensus with HashiCorp Raft
- Pull-based replication with ISR tracking and high-watermark advancement
- Producer and consumer client libraries with leader discovery and ack modes