Raft and Serf — Consensus, Gossip, and Building a Cluster with HashiCorp's Libraries

Every distributed system has to answer three questions before it can do anything useful:

1. Membership — which machines are in the cluster right now, and how do I reach them?
2. Leadership — who is allowed to make decisions, so two machines never decide differently?
3. Consensus — once a decision is made, how does every machine end up with the same record of it, even after crashes?

Two families of protocols answer these questions, and they make opposite trade-offs. Raft gives you strong agreement among a small group of machines, and stops making progress rather than ever disagreeing. Gossip gives you cheap, eventually-consistent knowledge across thousands of machines, and keeps working through almost any failure, at the price of never being sure everyone agrees right now.

HashiCorp ships production-grade Go implementations of both: hashicorp/raft (the core of Consul, Nomad and Vault) and hashicorp/serf (built on hashicorp/memberlist). This post explains how each protocol works, where each one fits, and then walks through how a Kafka-style distributed log combines them.

---

#Part 1: Raft

#The idea: a replicated state machine

Raft solves one specific problem: keep an ordered log of commands identical on a group of servers, so that each server can feed the same commands into the same deterministic state machine and end up in the same state.

If the log is the same everywhere and the state machine is deterministic, the state is the same everywhere. All the hard work is in keeping the log the same while servers crash, restart and lose network connectivity.

Raft's design goal was understandability. Paxos solves the same problem but is famously hard to implement correctly; Raft splits the problem into three parts you can reason about separately: leader election, log replication, and safety.

#Roles and terms

At any moment, each server is in one of three roles:

Role	What it does
Follower	Passive. Responds to the leader and to candidates. Most servers are followers most of the time.
Candidate	A follower that stopped hearing from a leader and is asking for votes.
Leader	Handles every client write, replicates the log, and sends heartbeats so followers know it is alive.

Time is divided into terms, numbered 1, 2, 3, … Each term starts with an election, and at most one leader can win a term. Terms act as a logical clock: every message carries the sender's term, and any server that sees a higher term than its own immediately updates and becomes a follower. That single rule is how a stale leader that was partitioned away discovers it has been replaced.

#Leader election

A leader sends periodic heartbeats. If a follower hears nothing for its election timeout, it assumes the leader is gone:

1. It increments its term, becomes a candidate, and votes for itself.
2. It sends RequestVote to every other server.
3. Each server grants at most one vote per term, first come first served — and only to a candidate whose log is at least as up to date as its own.
4. A candidate that collects votes from a majority becomes leader and starts sending heartbeats.

Two details make this work in practice:

• Randomized timeouts. If every follower timed out at the same instant, they would all vote for themselves and nobody would win. Each server picks a random timeout from a range, so usually one server asks first and wins before anyone else wakes up.
• The up-to-date check. A server refuses to vote for a candidate whose log is behind its own (compared by last term, then by last index). Since a committed entry lives on a majority, and a winner needs votes from a majority, the two majorities overlap — so every new leader already has every committed entry. This is Raft's leader completeness property, and it is why Raft never needs to copy committed data to a new leader.

#Log replication

Once elected, the leader is the only server that accepts writes:

1. A client sends a command to the leader.
2. The leader appends it to its own log with the current term.
3. It sends AppendEntries to every follower, carrying the new entries plus the index and term of the entry just before them.
4. A follower accepts only if its log contains that previous entry. If not, it rejects, and the leader retries one entry further back until the logs match — then overwrites whatever the follower had after that point.
5. Once the entry is stored on a majority, it is committed. The leader applies it to its state machine, answers the client, and tells followers the new commit index in the next message so they apply it too.

The consistency check in step 4 gives the log matching property: if two logs have an entry with the same index and term, they are identical up to that point. Followers never need to be repaired entry by entry; the leader just finds the last point of agreement and replays from there.

#Majorities and fault tolerance

Everything in Raft needs a majority (a quorum): winning an election, committing an entry. That fixes how many failures a cluster survives:

Servers	Majority	Failures tolerated
1	1	0
3	2	1
4	3	1
5	3	2
7	4	3

Two consequences:

• Use odd sizes. Four servers tolerate the same single failure as three, but need one more acknowledgement per write.
• Keep the group small. Every write waits for a majority, so larger groups mean slower writes. Production Raft groups are almost always 3 or 5 servers. Systems that need more scale run many Raft groups (CockroachDB and TiKV run one per data range).

When a majority is unreachable, Raft stops accepting writes. It will never let two halves of a partitioned cluster commit different things. In CAP terms, Raft chooses consistency over availability.

#Membership changes and snapshots

Two practical features complete the picture:

• Changing the server set. Adding or removing a server is itself written to the log as a configuration entry, so every server agrees on who the voters are. hashicorp/raft changes one server at a time (AddVoter, RemoveServer), which keeps old and new majorities overlapping and avoids two leaders during the change. It also supports non-voters: servers that receive the log but do not count toward the majority.
• Snapshots. The log cannot grow forever. Periodically each server serializes its state machine into a snapshot and deletes the log entries the snapshot covers. A server that falls too far behind (or a brand-new one) receives the snapshot instead of the whole history.

#Where Raft is used

System	What Raft keeps consistent
etcd (and so every Kubernetes cluster)	the key-value store holding all cluster state
Consul, Nomad, Vault	service catalog, job state, secrets (all via hashicorp/raft)
CockroachDB, TiKV, YugabyteDB	each range of table data is its own Raft group
Kafka KRaft	cluster metadata: topics, partitions, leaders, ISR
RabbitMQ quorum queues	the contents of a replicated queue

The pattern is consistent: Raft guards small, critical, frequently-read state — metadata, configuration, locks, leases — where disagreement would be a disaster and a few milliseconds per write is fine.

---

#Part 2: Gossip

#The idea: spread information like a rumor

Raft gives strong guarantees, but every write goes through one leader and waits for a majority. That does not scale to "tell all 2,000 machines that machine 1,417 just died".

Gossip (also called an epidemic protocol) works the way rumors spread in a crowd. Periodically, each node picks a few random peers and tells them what it knows. Those peers tell a few random peers of their own, and so on.

With a fan-out of k peers per round, the number of informed nodes grows roughly k-fold each round, so information reaches all N nodes in about O(log N) rounds. No node talks to everyone, no node is special, and losing any few nodes or messages barely slows the spread. The cost is that knowledge is eventually consistent: for a short window, different nodes have different views.

#Failure detection: SWIM

Spreading facts is half the job; the other half is noticing failures. The naive approach — every node heartbeats every other node — costs O(N²) messages. memberlist implements SWIM (Scalable Weakly-consistent Infection-style Membership), which keeps the load per node constant:

1. Every protocol period, each node picks one random member and pings it.
2. If no ack arrives, it asks k other members to ping the target on its behalf (ping-req). This separates "B is dead" from "the link between A and B is flaky".
3. If nobody gets an ack, B is marked suspect, and the suspicion is gossiped.
4. If B is alive and hears it is suspected, it refutes by gossiping a higher incarnation number. Otherwise, after a timeout, it is declared dead.

memberlist adds Lifeguard refinements on top (a node that is itself slow, say from GC pauses, lowers its confidence in its own failure judgements), and a periodic TCP push-pull sync that exchanges full state with one random peer to repair anything gossip missed.

#What Serf adds

memberlist gives you a membership list and failure detection. Serf builds a small, useful layer on top:

Serf feature	What it gives you
Member events	EventMemberJoin, EventMemberLeave, EventMemberFailed, EventMemberUpdate, EventMemberReap on a Go channel
Tags	a small key/value map per member, gossiped with it — perfect for "my RPC address is X, my role is Y"
Graceful leave vs failure	Leave() broadcasts an intent to leave, so others see left instead of failed
User events and queries	broadcast custom messages, or ask a question and collect responses
Lamport clocks	order events and de-duplicate messages that arrive more than once

#Where gossip is used

System	What gossip spreads
Consul	LAN and WAN membership and health of every agent (Serf)
Nomad	discovery of server nodes across regions (Serf)
Cassandra, ScyllaDB	ring membership, token ownership, schema versions
Redis Cluster	node state and slot ownership over the cluster bus
Many service meshes and load balancers	which instances are alive, and their load

#Raft and gossip side by side

	Raft	Gossip (Serf)
Guarantee	strong: one agreed, ordered log	eventual: views converge over time
During a partition	minority side stops making progress	every side keeps working with its own view
Scale	3–7 voters per group	thousands of nodes
Cost per update	one round trip to a majority	O(log N) rounds, constant load per node
Good for	metadata, configuration, locks, leader decisions	membership, failure detection, addresses, health

They are not competitors. Gossip is a great way to find out what is happening; Raft is the way to decide what to do about it.

---

#Part 3: Building a cluster with both

The rest of this post uses the distributed log — a Kafka-style replicated log written in Go — as the worked example. Every broker runs both libraries:

• Serf tells every broker which other brokers exist, whether they are alive, and their Raft and RPC addresses.
• Raft replicates a small metadata log: which topics exist, which broker leads each topic, and which replicas are in sync.
• The Raft leader acts as the cluster's controller: it is the only broker that writes metadata, and the one that reacts to membership changes.

#Membership: setting up Serf

broker/cluster/discovery/discovery.go wraps Serf in a Membership type. The important part is the configuration:

func (m *Membership) setupSerf() (err error) {
    addr, err := net.ResolveTCPAddr("tcp", m.BindAddr)
    if err != nil {
        return err
    }
    config := serf.DefaultConfig()
    config.Init()
    config.MemberlistConfig.BindAddr = addr.IP.String()
    config.MemberlistConfig.BindPort = addr.Port
    config.MemberlistConfig.LogOutput = io.Discard
    if m.Network != nil {
        mt, err := newNetworkTransport(addr.IP.String(), addr.Port, m.Network)
        if err != nil {
            return err
        }
        config.MemberlistConfig.Transport = mt
    }
    m.events = make(chan serf.Event)
    config.EventCh = m.events
    rpcAddr, err := m.Config.RPCAddr()
    if err != nil {
        return err
    }
    config.Tags = map[string]string{
        "rpc_addr":  rpcAddr,
        "raft_addr": m.RaftConfig.Address,
    }
    config.NodeName = m.NodeConfig.ID
    m.serf, err = serf.Create(config)
    if err != nil {
        return err
    }
    go m.eventHandler()
    if m.StartJoinAddrs != nil {
        _, err = m.serf.Join(m.StartJoinAddrs, true)
        if err != nil {
            m.logger.Warn("serf join failed", zap.Error(err), zap.Strings("addrs", m.StartJoinAddrs))
        }
    }
    return nil
}

Four decisions in this function do most of the work:

• NodeName is the broker ID. The same ID is used as the Raft server ID, so a Serf member and a Raft voter are trivially matched.
• Tags carry addresses. Each broker advertises its raft_addr and rpc_addr as tags. Nobody needs a static list of addresses: joining the gossip pool is enough for every broker to learn how to reach every other.
• StartJoinAddrs only needs one live member. A new broker joins through any existing broker; push-pull sync then hands it the full member list.
• The transport is pluggable. Tests swap in a networkTransport that can drop packets between chosen brokers, so a test can partition gossip exactly like a real network would.

#Turning Serf events into decisions

Serf events arrive on a channel. The handler ignores events about itself and forwards the rest to a small interface:

type Handler interface {
    Join(id, raftAddr, rpcAddr string) error
    Leave(name string) error
}

func (m *Membership) eventHandler() {
    for e := range m.events {
        switch e.EventType() {
        case serf.EventMemberJoin:
            for _, member := range e.(serf.MemberEvent).Members {
                if m.isLocal(member) {
                    continue
                }
                m.handleJoin(member)
            }
        case serf.EventMemberLeave, serf.EventMemberFailed:
            for _, member := range e.(serf.MemberEvent).Members {
                if m.isLocal(member) {
                    continue
                }
                m.handleLeave(member)
            }
        }
    }
}

func (m *Membership) handleJoin(member serf.Member) {
    raftAddr := member.Tags["raft_addr"]
    rpcAddr := member.Tags["rpc_addr"]
    if raftAddr == "" {
        raftAddr = rpcAddr
    }
    if rpcAddr == "" {
        rpcAddr = raftAddr
    }
    err := m.handler.Join(member.Name, raftAddr, rpcAddr)
    if err != nil {
        m.logError(err, "failed to join", member)
    }
}

The Handler is the bridge between the two worlds. Serf says "broker-4 appeared at 10.0.0.4"; the handler decides what that means for Raft. Note that a graceful leave and a detected failure are treated the same way: either way, the broker should stop being a voter.

#Consensus: setting up Raft

broker/cluster/raft/node.go builds the Raft node. hashicorp/raft asks you to supply every moving part explicitly:

func setupRaft(fsm raft.FSM, cfg config.RaftConfig, network transport.Network) (*raft.Raft, *raft.Config, raft.ServerAddress, *raftboltdb.BoltStore, *logStore, error) {
    raftBindAddr := cfg.Address
    if cfg.BindAddress != "" {
        raftBindAddr = cfg.BindAddress
    }
    raftAdvertiseAddr := cfg.Address
    raftConfig := raft.DefaultConfig()
    raftConfig.SnapshotThreshold = uint64(SnapshotThreshold)
    raftConfig.SnapshotInterval = time.Duration(SnapshotInterval) * time.Second
    raftConfig.LocalID = raft.ServerID(cfg.ID)
    raftConfig.LogLevel = cfg.LogLevel

    advertiseAddr, err := net.ResolveTCPAddr("tcp", raftAdvertiseAddr)
    if err != nil {
        return nil, nil, "", nil, nil, fmt.Errorf("failed to resolve Raft advertise address %s: %w", raftAdvertiseAddr, err)
    }
    stream, err := newStreamLayer(raftBindAddr, advertiseAddr, network)
    if err != nil {
        return nil, nil, "", nil, nil, fmt.Errorf("failed to make TCP transport bind %s advertise %s: %w", raftBindAddr, raftAdvertiseAddr, err)
    }
    trans := raft.NewNetworkTransport(stream, 3, 10*time.Second, os.Stderr)
    snapshots, err := raft.NewFileSnapshotStore(cfg.Dir, RetainSnapshotCount, os.Stderr)
    if err != nil {
        return nil, nil, "", nil, nil, fmt.Errorf("failed to create snapshot store at %s: %w", cfg.Dir, err)
    }
    boltDB, err := raftboltdb.NewBoltStore(filepath.Join(cfg.Dir, "raft.db"))
    if err != nil {
        return nil, nil, "", nil, nil, fmt.Errorf("failed to create bolt store: %w", err)
    }
    logStore, err := NewLogStore(cfg.Dir)
    if err != nil {
        return nil, nil, "", nil, nil, fmt.Errorf("failed to create log store: %w", err)
    }
    ra, err := raft.NewRaft(raftConfig, fsm, logStore, boltDB, snapshots, trans)
    if err != nil {
        return nil, nil, "", nil, nil, ErrNewRaft(err)
    }
    return ra, raftConfig, trans.LocalAddr(), boltDB, logStore, nil
}

raft.NewRaft takes six things, and each maps to a concept from Part 1:

Argument	Raft concept	In distributed-log
raftConfig	timeouts, snapshot policy, server ID	LocalID is the broker ID
fsm	the state machine	FSM wrapping the cluster metadata store
logStore	the replicated log	the broker's own segment log, reused for __cluster_metadata__
boltDB	stable store: current term, who I voted for	BoltDB file raft.db
snapshots	snapshot storage	files in the Raft directory, keep 10
trans	RequestVote / AppendEntries transport	TCP through the same pluggable network as Serf

Reusing the broker's segment log as Raft's LogStore is a nice touch: the metadata log is stored with the same code as topic data. The only wrinkle is that Raft numbers entries from 1 while the segment log numbers offsets from 0:

// Raft uses 1-based log indices (first entry is index 1). Our log uses 0-based offsets.
func (l *logStore) FirstIndex() (uint64, error) {
    if l.IsEmpty() {
        return 0, nil
    }
    return l.LowestOffset() + 1, nil
}

#Bootstrapping exactly once

A brand-new cluster has no configuration, so nobody can win an election. Exactly one broker is started with Boostatrap set, and it writes an initial configuration containing only itself:

func (c *RaftNode) Start() error {
    cfg := c.cfg.RaftConfig
    raftConfig := c.raftConfig
    if cfg.Boostatrap {
        configuration := raft.Configuration{
            Servers: []raft.Server{
                {
                    ID:      raftConfig.LocalID,
                    Address: c.LocalAddr,
                },
            },
        }
        if err := c.raft.BootstrapCluster(configuration).Error(); err != nil {
            return ErrBootstrapCluster(err)
        }
    }
    return nil
}

A one-server cluster has a majority of one, so this broker elects itself leader immediately. Every other broker starts with no configuration and waits to be added. Bootstrapping more than one broker independently would create several separate clusters that each think they are the whole cluster — the classic Raft bootstrap mistake.

#Membership meets consensus: adding voters

Here is where the two libraries meet. When Serf reports a new member, Membership calls Join on the Raft node:

func (c *RaftNode) Join(id, raftAddr, rpcAddr string) error {
    if !c.IsLeader() {
        c.Logger.Debug("not leader, skipping join", zap.String("joining_node_id", id), zap.String("raft_addr", raftAddr), zap.String("rpc_addr", rpcAddr))
        return nil
    }
    c.Logger.Info("join requested", zap.String("joining_node_id", id), zap.String("raft_addr", raftAddr), zap.String("rpc_addr", rpcAddr))
    configFuture := c.raft.GetConfiguration()
    if err := configFuture.Error(); err != nil {
        return err
    }
    serverID := raft.ServerID(id)
    serverAddr := raft.ServerAddress(raftAddr)
    for _, srv := range configFuture.Configuration().Servers {
        if srv.ID == serverID || srv.Address == serverAddr {
            if srv.ID == serverID && srv.Address == serverAddr {
                return nil
            }
            removeFuture := c.raft.RemoveServer(serverID, 0, 5*time.Second)
            if err := removeFuture.Error(); err != nil {
                return err
            }
        }
    }
    addFuture := c.raft.AddVoter(serverID, serverAddr, 0, 5*time.Second)
    if err := addFuture.Error(); err != nil {
        c.Logger.Error("raft add voter failed", zap.Error(err), zap.String("node_id", id))
        return err
    }
    c.Logger.Info("node joined cluster", zap.String("joined_node_id", id), zap.String("raft_addr", raftAddr), zap.String("rpc_addr", rpcAddr))
    return nil
}

Every broker receives the same Serf event, but only the Raft leader acts on it — followers return immediately. That gives a single, ordered decision without any extra coordination. The method is also idempotent: a broker that is already a voter at the same address is a no-op, and a broker that restarted with a new address is removed and re-added.

Leave is the mirror image — the leader removes the voter:

func (c *RaftNode) Leave(id string) error {
    if !c.IsLeader() {
        c.Logger.Debug("not leader, skipping leave", zap.String("leaving_node_id", id))
        return nil
    }
    c.Logger.Info("leave requested", zap.String("leaving_node_id", id))
    removeFuture := c.raft.RemoveServer(raft.ServerID(id), 0, 5*time.Second)
    if err := removeFuture.Error(); err != nil {
        c.Logger.Error("raft remove server failed", zap.Error(err), zap.String("node_id", id))
        return err
    }
    c.Logger.Info("node left cluster", zap.String("left_node_id", id))
    return nil
}

The split of responsibilities is the key design idea: Serf decides that something happened; Raft records what the cluster does about it. Serf can be wrong for a moment (a slow broker is suspected), but every broker still applies the same sequence of voter changes, in the same order, because those changes go through the Raft log.

The cluster's own view of "who is a live member" combines both sources — Raft's agreed voter list, filtered by Serf's current view of who is alive:

func (c *Cluster) AliveNodeIDs() []string {
    raftIDs, err := c.node.RaftServerIDs()
    if err != nil {
        c.Logger.Warn("alive node ids: raft config unavailable", zap.Error(err))
        return nil
    }
    c.mu.RLock()
    ml := c.memberLister
    c.mu.RUnlock()
    if ml == nil {
        return raftIDs
    }
    alive := make(map[string]struct{})
    for _, name := range ml.AliveMembers() {
        alive[name] = struct{}{}
    }
    out := make([]string, 0, len(raftIDs))
    for _, id := range raftIDs {
        if _, ok := alive[id]; ok {
            out = append(out, id)
        }
    }
    return out
}

Addresses come only from Serf tags (NodeRPCAddr), so there is no separately replicated table of addresses to keep in sync.

#Consensus: the FSM and metadata events

With membership in place, Raft's job is to replicate metadata. Every change is a MetadataEvent: a type plus a protobuf payload.

const (
    MetadataEventTypeCreateTopic MetadataEventType = iota
    MetadataEventTypeDeleteTopic
    MetadataEventTypeLeaderChange
    MetadataEventTypeIsrUpdate
    MetadataEventTypeAddNode
    MetadataEventTypeRemoveNode
    MetadataEventTypeUpdateNode
)

type MetadataEvent struct {
    EventType MetadataEventType
    Data      []byte
}

The write side runs on the Raft leader: encode the event and hand it to raft.Apply, which appends it to the log, replicates it and waits for it to commit.

func (c *Cluster) ApplyCreateTopicEvent(topic string, replicaCount uint32, leaderNodeID string, replicaNodeIds []string) error {
    if !c.node.IsLeader() {
        c.Logger.Debug("not leader, skipping create topic event", zap.String("topic", topic))
        return nil
    }
    eventData, err := raft.EncodeCreateTopicEvent(raft.CreateTopicEvent{
        Topic:          topic,
        ReplicaCount:   replicaCount,
        LeaderNodeID:   leaderNodeID,
        LeaderEpoch:    1,
        ReplicaNodeIds: replicaNodeIds,
    })
    if err != nil {
        return err
    }
    data, err := raft.EncodeMetadataEvent(&raft.MetadataEvent{
        EventType: raft.MetadataEventTypeCreateTopic,
        Data:      eventData,
    })
    if err != nil {
        return err
    }
    c.Logger.Info("apply create topic event", zap.String("topic", topic), zap.String("leader_node_id", leaderNodeID))
    if err := c.node.ApplyEvent(data); err != nil {
        c.Logger.Error("raft apply create topic failed", zap.Error(err), zap.String("topic", topic))
        return err
    }
    return nil
}

func (c *RaftNode) ApplyEvent(data []byte) error {
    f := c.raft.Apply(data, 5*time.Second)
    if err := f.Error(); err != nil {
        return ErrRaftApply(err)
    }
    return nil
}

The read side runs on every broker. Once an entry commits, hashicorp/raft calls FSM.Apply with it — on the leader and on every follower, in the same order:

func (c *FSM) Apply(l *raft.Log) interface{} {
    metadataEvent, err := DecodeMetadataEvent(l.Data)
    if err != nil {
        return err
    }
    c.mu.Lock()
    defer c.mu.Unlock()
    return c.metadataStore.Apply(metadataEvent)
}

and the metadata store turns each event into a state change:

// Apply applies a single Raft-committed metadata event to the store.
func (s *ClusterMetadataStore) Apply(ev *raft.MetadataEvent) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    switch ev.EventType {
    case raft.MetadataEventTypeCreateTopic:
        e, err := raft.DecodeCreateTopicEvent(ev.Data)
        if err != nil {
            return err
        }
        s.createTopicLocked(e.Topic, e.LeaderNodeID, e.LeaderEpoch, e.ReplicaNodeIds)
    case raft.MetadataEventTypeDeleteTopic:
        e, err := raft.DecodeDeleteTopicEvent(ev.Data)
        if err != nil {
            return err
        }
        delete(s.Topics, e.Topic)
    case raft.MetadataEventTypeLeaderChange:
        e, err := raft.DecodeLeaderChangeEvent(ev.Data)
        if err != nil {
            return err
        }
        if t := s.Topics[e.Topic]; t != nil {
            oldLeaderID := t.setLeader(e.LeaderNodeID, e.LeaderEpoch)
            if oldLeaderID != e.LeaderNodeID && oldLeaderID != "" {
                t.addReplicaIfAbsent(oldLeaderID, false)
            }
        }
    case raft.MetadataEventTypeIsrUpdate:
        e, err := raft.DecodeIsrUpdateEvent(ev.Data)
        if err != nil {
            return err
        }
        if t := s.Topics[e.Topic]; t != nil {
            t.setReplicaISR(e.ReplicaNodeID, e.Isr)
        }
    default:
        return fmt.Errorf("unknown or unsupported event type for cluster metadata: %d", ev.EventType)
    }
    return nil
}

Two rules keep the FSM correct:

• Apply must be deterministic. It runs on every broker, possibly years apart when a log is replayed. It must not read the clock, generate random IDs or talk to the network — anything decided at write time (like which broker leads the topic) must be inside the event.
• Reads are local. Every broker's ClusterMetadataStore is a full copy, so questions like "who leads topic orders?" are answered from memory without touching Raft. The trade-off is that a follower can be a few milliseconds behind; distributed-log tolerates that because clients retry when they reach a broker that is no longer the topic's leader.

#Snapshots

The FSM also has to snapshot and restore itself, so the metadata log can be truncated:

// Snapshot serializes the metadata store under the lock so Persist() works on a
// frozen copy and is safe to call without holding the FSM lock.
func (c *FSM) Snapshot() (raft.FSMSnapshot, error) {
    c.mu.RLock()
    defer c.mu.RUnlock()
    data, err := c.metadataStore.Snapshot()
    if err != nil {
        return nil, err
    }
    return &metadataSnapshot{data: data}, nil
}

func (c *FSM) Restore(r io.ReadCloser) error {
    defer r.Close()
    data, err := io.ReadAll(r)
    if err != nil {
        return err
    }
    return c.metadataStore.Restore(data)
}

hashicorp/raft decides when to snapshot on its own: here it checks every 10 seconds and snapshots once 10,000 new entries have accumulated. It then writes the result to disk in the background via Persist. Serializing under a read lock and persisting a frozen byte slice means snapshotting never blocks Apply for longer than the copy itself.

#Leadership: two levels

The cluster has two kinds of leader, and keeping them apart is what makes it scale:

1. The Raft leader is the controller. It is the only broker that writes metadata, and every metadata write path starts with an IsLeader() check. Creating a topic must be sent to it:

func (tm *TopicManager) CreateTopic(ctx context.Context, req *protocol.CreateTopicRequest) (*protocol.CreateTopicResponse, error) {
    c := tm.coordinator
    if !c.IsLeader() {
        return nil, fmt.Errorf("create topic must be sent to Raft leader: %w", ErrCannotReachLeader)
    }
    if c.TopicExists(req.Topic) {
        return nil, ErrTopicExistsf(req.Topic)
    }
    candidates := c.AliveNodeIDs()
    leaderNodeID, err := c.NodeIDWithLeastTopics(candidates)
    if err != nil {
        return nil, err
    }
    replicaNodeIds, err := PickReplicaNodeIds(leaderNodeID, int(req.ReplicaCount), candidates)
    if err != nil {
        return nil, ErrCreateTopic(err)
    }
    tm.Logger.Info("create topic via Raft", zap.String("topic", req.Topic), zap.String("leader_node_id", leaderNodeID), zap.Strings("replica_node_ids", replicaNodeIds))
    if err := c.ApplyCreateTopicEvent(req.Topic, req.ReplicaCount, leaderNodeID, replicaNodeIds); err != nil {
        return nil, err
    }
    return &protocol.CreateTopicResponse{Topic: req.Topic, ReplicaNodeIds: replicaNodeIds}, nil
}

Notice that the controller chooses the topic's leader (the alive broker leading the fewest topics) and writes that choice into the event. The FSM just records it.

2. Each topic has its own data leader. Produce and fetch traffic for a topic goes to that topic's leader, not to the Raft leader, so data load spreads across all brokers. Each topic leader carries a leader epoch — the per-topic equivalent of a Raft term — which increases on every change so stale leaders can be fenced off.

Raft leadership itself needs no code at all: hashicorp/raft runs elections internally. The application only asks raft.State() == raft.Leader and raft.LeaderWithID().

#Putting it together: a broker fails

Here is the whole system handling a crash, end to end:

1. Detect (Serf). SWIM probes to broker 2 fail; it becomes suspect, then failed. Every broker receives EventMemberFailed.
2. Decide membership (Raft). On the Raft leader, Leave("broker-2") calls RemoveServer. The configuration change commits through the Raft log.
3. React to the committed change. The cluster registers a Raft observer for peer changes, so it reacts the moment the removal takes effect — whatever triggered it:

func (c *Cluster) watchPeerChanges() {
    events, stop := c.node.WatchPeerChanges()
    defer stop()
    for ev := range events {
        if ev.Removed {
            c.Logger.Info("raft peer removed", zap.String("node_id", ev.NodeID))
            c.mu.RLock()
            fn := c.onNodeRemoved
            c.mu.RUnlock()
            if fn != nil {
                fn(ev.NodeID)
            }
        } else {
            c.Logger.Info("raft peer added", zap.String("node_id", ev.NodeID))
        }
    }
}

4. Decide topic leadership (Raft). The callback finds every topic broker 2 was leading, picks a live in-sync replica as the new leader, and applies a LeaderChange event with the epoch incremented:

func (tm *TopicManager) ReassignLeadersForDeadNode(nodeID string) {
    if !tm.coordinator.IsLeader() {
        return
    }
    type leaderChange struct {
        topic     string
        newLeader string
        epoch     int64
    }
    var changes []leaderChange
    for _, topicName := range tm.coordinator.TopicNames() {
        info, ok := tm.coordinator.TopicInfo(topicName)
        if !ok || info.LeaderNodeID != nodeID {
            continue
        }
        var newLeader string
        for _, rs := range info.Replicas {
            if rs.NodeID == nodeID || !rs.IsISR {
                continue
            }
            if !tm.coordinator.IsNodeAlive(rs.NodeID) {
                continue
            }
            newLeader = rs.NodeID
            break
        }
        if newLeader == "" {
            tm.Logger.Warn("no ISR replica for leadership", zap.String("topic", topicName), zap.String("old_leader_node_id", nodeID))
            continue
        }
        changes = append(changes, leaderChange{topic: topicName, newLeader: newLeader, epoch: info.LeaderEpoch + 1})
    }
    if len(changes) == 0 {
        return
    }
    // Applied asynchronously so the caller (e.g. the reconciliation loop) isn't
    // blocked for the duration of the resulting Raft round-trips.
    go func() {
        for _, ch := range changes {
            if err := tm.coordinator.ApplyLeaderChangeEvent(ch.topic, ch.newLeader, ch.epoch); err != nil {
                tm.Logger.Warn("leader change apply failed", zap.String("topic", ch.topic), zap.Error(err))
            }
        }
    }()
}

5. Record (FSM). Every broker applies the LeaderChange, so every broker now routes the topic to its new leader — and the old leader, if it comes back, sees a higher epoch and knows it is no longer in charge.

If the Raft leader itself is the broker that died, one extra step happens first: the remaining brokers' election timeouts fire and they elect a new Raft leader. Only then can step 2 run — which is exactly right, because nobody else is allowed to change the metadata.

#Design lessons

A few principles fall out of this design and apply to any system built on these two libraries:

• Use gossip to observe, consensus to decide. Serf is fast and tolerant but can be briefly wrong; Raft is slower but never disagrees with itself. Feeding Serf events into Raft operations gets the speed of one and the safety of the other.
• Let only the leader act on observations. Every broker sees every Serf event; only the Raft leader turns it into a decision. No extra locking or election is needed.
• Put decisions inside the log entry. The controller picks leaders and replicas before calling Apply, so the FSM stays deterministic.
• Reconcile, don't only react. A Serf event that arrives while Raft leadership is changing is seen by no leader: the old one is gone and the new one was not yet elected. Comparing Serf's alive members with Raft's voters whenever a broker becomes leader (Consul runs exactly this kind of reconcile loop) closes that gap.
• Make handlers idempotent. Events can be duplicated or replayed (after a leader change, after a restart). Join checking the existing configuration first is what makes repeated events harmless.
• Keep the Raft group small and the data path off it. Raft carries only metadata — topics, leaders, ISR. Records themselves are replicated by topic leaders with their own protocol, so Raft is never the bottleneck for throughput.
• Think about automatic removal. Removing a failed voter keeps the majority reachable for the next failure, but a broker that is merely slow can be removed and re-added repeatedly. Tuning Serf's suspicion timeouts, or only removing voters after a grace period, trades failover speed against churn.

---

#Summary

Question	Answered by	How
Who is in the cluster?	Serf (gossip + SWIM)	member events, tags carrying Raft and RPC addresses
Who decides?	Raft leader election	the Raft leader acts as controller; topic leaders are chosen by it
What did we decide?	Raft log + FSM	every metadata change is a log entry applied in order on every broker

Raft and gossip solve different halves of the same problem. Gossip spreads facts cheaply to everyone; Raft turns a few important facts into decisions everyone agrees on. hashicorp/serf and hashicorp/raft make both available as libraries, and the glue between them — a Handler with Join and Leave, an FSM, and a leader check — is small enough to read in an afternoon.

If you want to build this yourself step by step, the Building a Distributed Log guide covers the Serf and Raft layers in detail, along with the storage, protocol and replication code around them.