GoDistributed SystemsKafkaRaft

Building a Distributed Log in Go

A hands-on guide to building a Kafka-inspired, replicated distributed log from scratch — not just as an exercise, but to understand why Kafka is designed the way it is. Every design decision is motivated by a real production concern.

Start reading →

What you'll build

  • An append-only log with segments, sparse offset index, and memory-mapped I/O
  • A binary wire protocol and length-prefixed TCP transport layer
  • Cluster membership and failure detection with HashiCorp Serf
  • Distributed metadata consensus with HashiCorp Raft
  • Pull-based replication with ISR tracking and high-watermark advancement
  • Producer and consumer client libraries with leader discovery and ack modes

Table of contents

  1. 1Introduction — Building a Kafka-Style Distributed Log in GoStart here→
  2. 2Concepts and Terminology — Offsets, Segments, LEO, ISR, Raft, and Serf→
  3. 3Project Structure and Architecture — Go Package Layout for a Distributed Log→
  4. 4Building the Low-Level Log — Segments, Sparse Index, and Storage in Go→
  5. 5Wire Protocol and Codec — Binary Framing and Message Types in Go→
  6. 6TCP Transport and RPC Server — Network Layer for a Distributed Log→
  7. 7Cluster Discovery with Serf — Gossip-Based Membership and Failure Detection→
  8. 8Raft Consensus and the Coordinator — Distributed Metadata with HashiCorp Raft→
  9. 9Topic Management — Leaders, Replicas, and Metadata in a Distributed Log→
  10. 10Pull-Based Replication, ISR, and High Watermark in a Distributed Log→
  11. 11Producer and Consumer APIs — Leader Discovery, Ack Modes, and Offset Management→
  12. 12Running a Multi-Node Distributed Log Cluster End-to-End→