Introduction — Building a Kafka-Style Distributed Log in Go

Introduction — Building a Kafka-Style Distributed Log in Go

In this section you build a distributed log from scratch in Go — a Kafka-style replicated, ordered log that runs across multiple machines. By the end you will have a working system with Raft consensus, Serf-based cluster discovery, pull-based replication, and producer/consumer APIs.

Git repository

You can find the full source code for the project on GitHub:

https://github.com/Mohitkumar/distributed-log

Clone and build locally:

https://github.com/Mohitkumar/distributed-log.git
cd distributed-log
make build

This is not a toy. You will implement the same ideas that power Apache Kafka, Redpanda, and similar event-streaming platforms: append-only storage with segments and sparse indexes, a wire protocol for RPC, consensus for metadata, replication with in-sync replicas (ISR) and high watermarks, and topic management with leader election.

What is a log?

A log is an append-only, ordered sequence of records. You never update or delete in place — you only append at the end. Each record gets a monotonic offset (a position number) that defines its order. This simplicity gives you three powerful guarantees:

Property What it means
Durability Once a record is written and fsynced, it stays. No overwrites means no partial-write corruption.
Replay Any consumer can read from any offset and reproduce the exact sequence of events.
Ordering Physical order on disk matches logical order. Offset 42 always comes before offset 43.

A single-node log is a file on disk. A distributed log spreads that file across multiple nodes with replication so that no single machine failure loses data.

About Apache Kafka

Apache Kafka is the most widely deployed distributed log. Understanding Kafka's architecture helps frame what you are building:

  • Topics — named logs. A topic called orders is an independent ordered stream.
  • Partitions — each topic is split into partitions for parallelism. Order is guaranteed per partition, not across partitions.
  • Producers — clients that append records to a topic's partition. The partition leader assigns offsets.
  • Consumers — clients that read records by offset. Consumer groups coordinate so each partition is read by one consumer in the group.
  • Brokers — the servers that store partitions and serve produce/fetch requests.
  • Replication — each partition has one leader and N-1 followers. The leader serves writes; followers replicate to stay in sync. If the leader dies, a follower takes over.
  • ZooKeeper / KRaft — cluster metadata (which broker leads which partition, topic configs) is coordinated by a consensus system.

Our distributed log implements the same layered architecture: local storage (segments), a consensus layer (Raft for metadata), membership (Serf for discovery), and replication (pull-based with ISR tracking).

Use cases of a distributed log

A distributed log is the right abstraction whenever you need ordered, durable, replayable events across machines:

Use case Why a log helps
Message broker Producers append; consumers read by offset. Decouples senders from receivers while preserving order per partition. Unlike traditional message queues, messages are not deleted after consumption — multiple consumers can read the same data independently.
Event sourcing Application state is derived by replaying the log from the beginning. The log is the single source of truth. Need a new read model? Replay the log into a new projection.
Stream processing Process events in order — filter, aggregate, join, window — and write results to another topic or an external store. The log provides exactly-once ordering semantics per partition.
Change data capture (CDC) Databases write changes to a log; downstream systems (caches, search indexes, data warehouses) consume the log to stay in sync.
Database replication A primary's write-ahead log (WAL) is replicated to read replicas. The log ensures every replica applies the same writes in the same order.
Audit trail Append-only history of every action — who did what, when. Immutability makes it tamper-evident. Useful for compliance, debugging, and analytics.

In every case, the log provides order, durability, and replay.

What you will learn

This section walks through the complete implementation in a bottom-up order. Each section builds on the previous one:

1. Core concepts

Kafka-style terminology — offsets, segments, sparse indexes, LEO (Log End Offset), HW (High Watermark), ISR (In-Sync Replicas) — and the algorithms that power a distributed log: Raft consensus, Serf gossip, pull-based replication.

2. Project structure

The Go project layout, package responsibilities, and the three-layer architecture (storage → consensus → cluster) that organizes the codebase.

3. Low-level log (storage layer)

Build the on-disk storage from the bottom up:

  • Record encoding — a binary format: [Offset 8 bytes][Length 4 bytes][Value].
  • Segment — one append-only .log file paired with a sparse .idx index file.
  • Index — memory-mapped, fixed-size entries with binary search for fast offset lookups.
  • Log — manages multiple segments: append to the active segment, roll to a new one when full, find the right segment for reads.
  • LogManager — wraps the log with LEO and high watermark tracking.

4. Wire protocol

Design a length-prefixed frame format for TCP communication. Define message types (Produce, Fetch, CreateTopic, etc.), request/response structs, and a JSON-based codec. Build a replication batch format for efficient bulk transfer.

5. Transport and RPC

Build a TCP transport layer (server and client) and an RPC server that routes incoming frames to handler functions. Implement produce, fetch, topic management, and discovery handlers.

6. Discovery with Serf

Use HashiCorp Serf for decentralized cluster membership via gossip. Nodes join, discover peers, and learn RPC addresses without a static configuration file. Handle join, leave, and failure events to keep the cluster view current.

7. Raft consensus

Integrate HashiCorp Raft for metadata coordination. Build a Finite State Machine (FSM) that applies metadata events (topic creation, leader changes, ISR updates). Adapt the local log as a Raft log store. Wire Raft into the coordinator so the cluster agrees on who leads which topic.

8. Topic management

Implement the TopicManager that tracks all topics, their leaders, and replicas. Topic creation flows through Raft for cluster-wide consistency. Leader assignment picks the least-loaded node; replicas are distributed across available nodes.

9. Replication

Build a pull-based replication thread: follower replicas periodically fetch batches from the leader. The leader tracks each replica's LEO and computes ISR membership. The high watermark advances when all ISR replicas have caught up. Consumers only see data up to the high watermark.

10. Producer and consumer APIs

Implement producer and consumer client libraries with automatic leader discovery and reconnection. Support three ack modes (none, leader, all). Build consumer offset tracking with a persistent offset log.

11. Running the cluster

Package everything with Docker, start a 3-node cluster, create topics, produce and consume messages, and verify replication and fault tolerance end to end.

Prerequisites

  • Go 1.25+ installed.
  • Familiarity with Go basics (structs, interfaces, goroutines, channels).
  • Docker and Docker Compose (for the multi-node cluster).
  • A text editor and terminal.

No prior experience with distributed systems or Kafka is required — each concept is explained before it is implemented.

Next steps

Start with the Concepts page to understand the terminology and algorithms, then move to Project Structure to set up the codebase. After that, build the storage layer and work your way up through the network stack to a running cluster.