← All guides
GoKafkaRaftSerf

Building a Distributed Log

Inspired by Kafka

Build a Kafka-style replicated log from scratch — append-only segments, a binary wire protocol, Serf membership, Raft metadata consensus, pull-based replication and producer/consumer clients.

13 parts · about 5 hours of reading · full source on GitHub

Start reading →

What you'll build

  • An append-only log with segments, sparse offset index, and memory-mapped I/O
  • A binary wire protocol and length-prefixed TCP transport layer
  • Cluster membership and failure detection with HashiCorp Serf
  • Distributed metadata consensus with HashiCorp Raft
  • Pull-based replication with ISR tracking and high-watermark advancement
  • Producer and consumer client libraries with leader discovery and ack modes