Chaos Testing a Distributed Log — Fault Injection and Linearizability with Porcupine

Every page so far has made a promise: an AckAll write survives the loss of any single node, a consumer never reads past the high watermark, a dead follower drops out of the ISR, a restarted node catches back up on its own. In the previous section you checked a few of those promises by hand — docker stop, produce again, see that the data is still there.

In this section you'll turn those checks into an automated chaos suite: tests that start a real 3-node cluster, hard-kill nodes or partition the network in the middle of a workload, and then check the cluster's guarantees with precise assertions and with a linearizability checker. The tests live under tests/chaos/, and the fault-injecting network lives in tests/faultnet.go:

File	What it holds
tests/chaos/main_test.go	Port allocation for the package
tests/chaos/fault_test.go	Helpers for waiting on cluster state, plus single-fault tests (kill leader, kill/restart follower, kill loop)
tests/chaos/scenarios_test.go	Multi-invariant scenarios: multi-topic re-election, follower fault integrity (LEO/ISR/HW), and an end-to-end producer → consumer loop under leader kill
tests/chaos/porcupine_test.go	Concurrent producers under leader or follower kill, checked for linearizability with Porcupine
tests/chaos/partition_test.go	A restarted leader truncating its divergent tail, and a partitioned leader being fenced and then converging after the partition heals
tests/faultnet.go	FaultNet, an in-process network the tests can partition

Before building the suite, it helps to see the wider field: what chaos testing is, where it came from, and the different methods teams use to break systems on purpose. That background explains which methods this suite uses and why.

#What chaos testing is

#Breaking things on purpose

Chaos testing means deliberately injecting failures into a running system and watching whether it keeps its promises. Two traditions grew up around the same idea, and they ask different questions.

Chaos engineering began at Netflix around 2010 with Chaos Monkey, a tool that randomly terminated production servers during business hours. Netflix had just moved to AWS, where instances could disappear at any time. Instead of hoping services could tolerate that, they made it happen constantly, so every team had to build for it. Chaos Monkey grew into the Simian Army: Latency Monkey added network delay, and Chaos Kong took out an entire AWS region. The question chaos engineering asks is about resilience: does the service stay up, and do users notice?

Correctness testing under faults comes from the databases world. Its best-known example is Kyle Kingsbury's Jepsen project, which since 2013 has found data-loss and consistency bugs in Kafka, MongoDB, Cassandra, etcd, Redis, CockroachDB, and many others. A Jepsen test runs a concurrent workload, injects faults through a component called the nemesis, records every operation's call and result, and then checks the full history against a consistency model. The question is about safety: did the system ever return a wrong answer, lose an acknowledged write, or break the consistency it advertises?

A distributed log needs both. Resilience matters because producers and consumers should ride through a node failure. Safety matters more, because a log that silently drops acknowledged records is worse than one that is briefly unavailable. This chapter leans toward the Jepsen style.

#The principles of chaos engineering

The Principles of Chaos Engineering (principlesofchaos.org) describe the method as an experiment with five parts:

1. Define steady state. Choose a measurable output that shows the system is healthy: request success rate, p99 latency, or, for a log, "every acked write is readable in order."
2. Form a hypothesis. Predict that steady state will hold in both a control group and an experimental group.
3. Vary real-world events. Inject failures that actually happen: crashes, network partitions, latency, full disks, clock jumps, traffic spikes.
4. Try to disprove the hypothesis. Compare the experimental group with the control group. Any difference in steady state is a weakness you just found.
5. Minimize the blast radius. Start small, on one instance or a slice of traffic, and have a way to stop the experiment immediately.

The original principles also recommend running experiments in production and automating them to run continuously, since staging environments never quite match production traffic, configuration, or scale. For a system you're building from scratch, the test suite is the right place to start. The same experiment structure applies there: steady state becomes your invariants, and the hypothesis becomes your assertions.

#Methods of chaos testing

The methods below differ in what they break and how much control the test has over timing. Most mature systems combine several of them.

#Process faults

Process faults are the simplest and most common. The usual forms:

• Crash: kill -9, docker kill, or deleting a Kubernetes pod. The process gets no chance to clean up.
• Graceful stop: SIGTERM or a planned shutdown. This checks the shutdown and handoff path rather than crash recovery.
• Pause: SIGSTOP followed later by SIGCONT, or docker pause. The process is frozen but its sockets stay open, the way a long garbage-collection pause or an overloaded VM behaves. Paused processes are dangerous because they wake up still believing what was true before the pause, for example that they're the leader.
• Restart loops: repeated kill and restart cycles on the same node. These catch resource leaks and state that doesn't survive a restart.

#Network faults

Networks fail in more ways than "up" or "down":

• Partitions: split the cluster into groups that can't reach each other. Asymmetric partitions, where A reaches B but B can't reach A, are especially good at exposing bugs, because different subsystems end up disagreeing about who is alive.
• Latency and jitter: delay packets, possibly by different amounts each time.
• Packet loss, duplication, and reordering.
• Bandwidth limits: throttle a link until it saturates.

Common tools: Linux tc with netem for delay, loss, and reordering; iptables or nftables rules for partitions; Toxiproxy, a programmable TCP proxy built for tests; Pumba for Docker containers; and Chaos Mesh NetworkChaos resources on Kubernetes.

#Clock faults

Leases, timeouts, and time-based retention all depend on clocks. Clock faults include jumping a node's clock forward or backward, making it drift slowly, and skewing different nodes by different amounts. Tools include libfaketime, Jepsen's clock nemesis, and Chaos Mesh TimeChaos. A system that relies only on monotonic clocks and timeouts, rather than comparing wall-clock times across nodes, should be immune. Clock faults are how you check that it really is.

#Disk and filesystem faults

Storage fails in subtle ways:

• Torn writes: only part of a write reaches disk before a crash.
• Lost unsynced data: writes that were never fsynced vanish on power loss.
• I/O errors: fsync or write returns EIO.
• Full disk: writes fail with ENOSPC.
• Slow disk: high latency on every I/O call.
• Silent corruption: bit flips that only a checksum would catch.

Tools include LazyFS, a FUSE filesystem that drops unsynced writes on simulated power loss; CharybdeFS; the Linux device-mapper targets dm-flakey and dm-delay; Chaos Mesh IOChaos; and, at the crude end, truncating or corrupting files between a stop and a restart. Research tools such as ALICE systematically explore which crash states a storage format can end up in.

#Resource exhaustion

Starve a node of CPU, memory, file descriptors, or disk space with tools like stress-ng or cgroup limits. Exhaustion turns into slowness rather than a clean failure, so it checks that timeouts, backpressure, and ISR lag detection behave sensibly when a node is alive but struggling.

#Application-level fault injection (failpoints)

Instead of breaking the environment, put named fault hooks directly in the code: "fail here," "sleep here," "panic between appending to the log and updating the index." Tests switch them on at runtime. FoundationDB's BUGGIFY, TiKV's fail-rs, and etcd's gofail (for Go) all work this way. Failpoints reach windows that external faults can only hit by luck, such as a crash between two specific lines, at the cost of changing production code to add the hooks.

#Deterministic simulation testing

The most powerful method, and the most work to set up. The whole cluster runs in one thread on a simulated network, simulated disk, and simulated clock, all driven by a single random seed. The simulator injects faults and chooses message orderings. Because everything is deterministic, any failure can be replayed exactly from its seed, and thousands of simulated hours run in minutes of real time. FoundationDB was built this way from day one, and TigerBeetle (its VOPR simulator) and the Antithesis platform apply the same idea. The cost is that the system has to route all I/O and time through interfaces the simulator controls.

#History checkers

Fault injection only finds bugs if something notices the wrong answer. Simple assertions ("offset 5 holds message 5") work when the expected result is known in advance. Under concurrency it isn't, so you record a history and check it against a model:

• Linearizability checkers: Jepsen's Knossos and the Go library Porcupine search for a single total order of operations consistent with real time and with each result.
• Transactional checkers: Jepsen's Elle finds anomalies such as lost updates and write skew in transactional histories.

#Formal methods

Before writing code, you can model the protocol itself in TLA+ or P and let a model checker explore every interleaving of a small configuration. Amazon has used TLA+ to find design bugs in S3 and DynamoDB, and Kafka's replication protocol has public TLA+ specifications. Model checking verifies the design. Chaos testing verifies that the implementation matches it.

#Platforms and game days

For running experiments against deployed systems there are dedicated platforms: Chaos Monkey, Gremlin, LitmusChaos, Chaos Mesh, AWS Fault Injection Service, and Azure Chaos Studio. These schedule faults, limit the blast radius, and stop the experiment automatically if health checks fail. Many teams also run game days: planned sessions where engineers inject a fault into production or a realistic staging environment and practice detecting, diagnosing, and recovering from it. Game days test people and runbooks, not just code.

#Choosing methods

Method	Finds	Cost to adopt	Reproducible?
Process kill/restart	Failover, recovery, leaked state	Low	Partly; timing varies
Network faults	Split brain, stale leaders, timeout bugs	Medium; needs a proxy or network control	Partly
Clock faults	Lease and timeout bugs	Medium	Partly
Disk faults	Torn writes, recovery and checksum bugs	Medium	Partly
Failpoints	Crashes in exact code windows	Medium; changes the code	Yes, per failpoint
Deterministic simulation	Rare interleavings of many faults	High; all I/O must be simulated	Fully, by seed
History checkers	Wrong answers under concurrency	Low to medium	Same as the faults behind them
Formal methods	Protocol design bugs	Medium to high	Fully
Production platforms and game days	Operational and human gaps	Medium	No

A practical path for a system like this log: start with process faults plus assertions and a linearizability checker, which this chapter builds. Then add network partitions and disk faults, then a randomized nemesis loop. Consider failpoints or deterministic simulation once the cheaper methods stop finding bugs.

#Step 1: Decide what chaos testing has to prove

#Faults are normal operating conditions

Unit tests check that a function returns the right value. Integration tests check that components talk to each other correctly. Neither tells you what happens when a node dies between two steps of a protocol — after the leader appended a record but before the follower fetched it, or after Serf noticed a failure but before Raft committed the new leader.

Those windows are where distributed systems lose data, and they're exactly what chaos testing targets. The approach is simple to describe:

1. Start a real cluster — real Raft, real Serf, real TCP, real on-disk segments.
2. Run a workload against it.
3. Inject a fault while the workload is running.
4. Let the cluster recover on its own, with no help from the test.
5. Check the guarantees the system claims to provide.

#The invariants

Before writing a single test, write down what "correct" means. For this log, these are the properties the earlier pages built:

Invariant	Where it comes from
No acked write is lost. Every AckAll produce that returned success is present, at its assigned offset, on whichever node leads next.	waitForAllFollowersToCatchUp in the producer path, ISR-based leader selection
Order is preserved. Offsets on every replica hold the same values in the same order.	Pull-based replication from the leader's LEO
A new leader is elected for every topic the dead node led, and topics it didn't lead keep their leader.	ReassignLeadersForDeadNode driven by Serf + Raft
HW tracks the live ISR. A dead follower is expired from the ISR, and the HW advances to what the leader and surviving followers actually have.	ExpireStaleISR with ISRLagTime
Restarted nodes catch up from their own on-disk state without manual intervention.	Replication thread + reconciliation loop
Consumers see a gap-free, duplicate-free sequence, bounded by the HW, across a failover.	HW check in LogManager.Read, consumer client reconnection
A deposed leader converges. A leader that comes back holding records nobody else has discards them and ends with exactly the new leader's log.	Leader epoch cache and the divergence check in replication
An isolated leader is fenced. A leader cut off from its followers never acknowledges an AckAll write, never exposes records past what the followers have, and stops directing clients to itself.	ISR acknowledgment, the quorum check on ISR expiry, the quorum check on leader lookup
The log is linearizable under concurrent appends: there is one total order of writes consistent with what every client observed.	All of the above, combined

Each test in this section checks one or more rows of this table.

#Step 2: Gate the suite behind a build tag

Chaos tests are slow — Serf's failure detector needs real wall-clock time to declare a node dead, and Raft needs time to commit a new leader. They're also noisier than unit tests. Keep them out of the default go test ./... run with a build tag at the top of every file in the package:

//go:build chaos

package chaos

and give them their own Makefile targets:

# Fault-injection and Porcupine linearizability tests: real cluster with nodes hard-killed
# mid-workload (tests/chaos). Slower and noisier (real Serf failure detection), gated
# behind the "chaos" build tag so it's excluded from `make test` / `go test ./...`.
test-chaos:
    $(GO_CMD) test -tags chaos -v ./tests/chaos/...

# All suites, including chaos.
test-all: test test-chaos

Every test spins up its own cluster, and Go runs each test package as a separate process, so packages can run concurrently. To keep them from fighting over ports, each package claims its own port range in TestMain:

// tests/chaos/main_test.go
func TestMain(m *testing.M) {
    tests.SetPortBase(35000)
    os.Exit(m.Run())
}

allocPorts in tests/testutil.go then hands each cluster nine consecutive ports from that base — a Serf bind port, a Raft port, and an RPC port for each of the three nodes.

#Step 3: Build the fault injector — Kill and Restart

#Kill: stop a node without telling anyone

The test harness's tests.StartRealThreeNodeCluster returns three RealTestServer values, each running the full broker stack in-process. The fault you inject is Kill:

// tests/testutil.go
func (rts *RealTestServer) Kill() error {
    var errs []error

    if rts.TopicManager != nil {
        rts.TopicManager.StopReplicationThread()
    }
    if rts.RpcServer != nil {
        if err := rts.RpcServer.Stop(); err != nil {
            errs = append(errs, err)
        }
    }
    if rts.Membership != nil {
        if err := rts.Membership.Shutdown(); err != nil {
            errs = append(errs, err)
        }
    }
    if rts.Coordinator != nil {
        if err := rts.Coordinator.Shutdown(); err != nil {
            errs = append(errs, err)
        }
    }
    rts.RpcServer = nil
    rts.Membership = nil
    rts.Coordinator = nil
    rts.TopicManager = nil
    rts.ConsumerMgr = nil
    if len(errs) > 0 {
        return fmt.Errorf("kill errors: %v", errs)
    }
    return nil
}

The important detail is Membership.Shutdown() rather than Membership.Leave(). Leave broadcasts a graceful departure through Serf, so the other nodes would know immediately — that's a planned decommission, not a crash. Shutdown just stops responding. The survivors have to discover the failure the same way they would in production: Serf's failure detector stops getting probe acks, marks the node suspect, then failed, and the cluster reacts with leader re-election and ISR changes. The test exercises the real detection path rather than a shortcut.

#Restart: come back from disk

// tests/testutil.go
func (rts *RealTestServer) Restart(t testing.TB) {
    t.Helper()
    fresh := startRealNode(t, rts.startCfg)
    rts.Coordinator = fresh.Coordinator
    rts.TopicManager = fresh.TopicManager
    rts.ConsumerMgr = fresh.ConsumerMgr
    rts.RpcServer = fresh.RpcServer
    rts.Membership = fresh.Membership
    rts.Addr = fresh.Addr
    rts.raftAddr = fresh.raftAddr
}

Restart builds a brand-new node from the same start configuration — same node ID, same data directory — so it comes back up from whatever Raft log, snapshots, and segments it left on disk, and rejoins as a returning member.

#Step 4: Wait on observable state, never on sleeps

The single most common bug in chaos tests is time.Sleep(5 * time.Second) followed by an assertion. It passes on your laptop and fails on a loaded CI runner. Instead, poll for the state you need, with a deadline, and fail with a clear message if it never arrives.

#Waiting for a new leader

// tests/chaos/fault_test.go
func waitForTopicLeaderID(t testing.TB, nodes []*tests.RealTestServer, topicName, excludeNodeID string, timeout time.Duration) string {
    t.Helper()
    deadline := time.Now().Add(timeout)
    for time.Now().Before(deadline) {
        for _, n := range nodes {
            if n == nil || n.Coordinator == nil {
                continue
            }
            leaderID, ok := n.Coordinator.TopicLeaderNodeID(topicName)
            if ok && leaderID != "" && leaderID != excludeNodeID {
                return leaderID
            }
        }
        time.Sleep(200 * time.Millisecond)
    }
    t.Fatalf("no leader for topic %q (other than %q) within %s", topicName, excludeNodeID, timeout)
    return ""
}

Pass the dead node's ID as excludeNodeID after a kill, and the helper only returns once some live node's own Raft-replicated view shows a different leader. Pass "" to wait for any leader, for example right after CreateTopic. Killed nodes have a nil Coordinator, so they're skipped automatically.

#Waiting for the cluster to notice a death

// tests/chaos/fault_test.go
func waitForNodeRemoved(t testing.TB, nodes []*tests.RealTestServer, deadNodeID string, timeout time.Duration) {
    t.Helper()
    deadline := time.Now().Add(timeout)
    for time.Now().Before(deadline) {
        removed, checked := true, false
        for _, n := range nodes {
            if n == nil || n.Coordinator == nil || n.NodeID == deadNodeID {
                continue
            }
            checked = true
            if n.Coordinator.IsNodeAlive(deadNodeID) {
                removed = false
            }
        }
        if checked && removed {
            return
        }
        time.Sleep(200 * time.Millisecond)
    }
    t.Fatalf("node %q was not removed as a raft voter within %s", deadNodeID, timeout)
}

Call this before every Restart. If a node restarts before the survivors have finished reaping its old membership entry, the rejoining Serf instance can collide with the not-yet-removed entry for the same name and address. A real process supervisor gets this ordering for free, since it has to notice the crash before restarting. The test reproduces that ordering explicitly.

#Waiting for ISR recovery and follower catch-up

scenarios_test.go adds two more helpers in the same style. waitForISR polls a node's TopicInfo until every expected replica reports IsISR. ISR membership only recovers once a replica resumes fetching and catches up through RecordReplicaLEOFromFetch, not the moment it restarts. waitForFollowerLEO polls a follower's local log until it reaches a target LEO. That one exists because AckAll returns as soon as the leader has served the follower's fetch for the target offset, which can be a moment before the follower finishes writing those records to disk.

#Step 5: Single-fault tests

With the injector and the wait helpers in place, each test reads as a short script: set up, inject, recover, assert.

#Kill the leader during produce — no acked write is lost

TestFault_KillLeaderDuringProduce creates a topic with ReplicaCount: 2 (a leader plus two replicas, so it survives any single node loss), sends ten AckAll messages, kills the node the producer is talking to, and keeps sending:

// tests/chaos/fault_test.go
t.Logf("killing leader node %s (%s)", deadNode.NodeID, deadNode.Addr)
if err := deadNode.Kill(); err != nil {
    t.Fatalf("Kill %s: %v", deadNode.NodeID, err)
}

sendCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
for i := 0; i < 10; i++ {
    val := "post-kill-" + strconv.Itoa(i)
    if _, err := pc.Send(sendCtx, []byte(val), protocol.AckAll); err != nil {
        t.Fatalf("Send post-kill %d: %v", i, err)
    }
    acked = append(acked, val)
}

The topic-aware producer client from the producer/consumer page reconnects to the new leader on its own, so Send simply blocks until failover completes. The test then reads every offset straight off the new leader's log, using ReadUncommitted so nothing is hidden behind the HW, and checks each one against the list of acked values:

for i, want := range acked {
    entry, err := l.ReadUncommitted(uint64(i))
    if err != nil {
        t.Fatalf("ReadUncommitted offset %d: %v", i, err)
    }
    const offWidth = 8
    got := string(entry[offWidth:])
    if got != want {
        t.Fatalf("offset %d: expected %q, got %q — acked message lost or reordered after leader kill", i, want, got)
    }
}

#Kill and restart a follower — it catches up from disk

TestFault_KillAndRestartFollower produces five messages with AckLeader, kills a follower, produces five more (the leader doesn't need the dead follower for AckLeader), records the leader's LEO, waits for the death to be observed, restarts the follower, and polls until its local LEO reaches the target. No test code re-adds the node to the replica set. Reconciliation and replication have to do it on their own.

#Kill the same follower in a loop

TestFault_KillFollowerLoop repeats the kill → produce → restart → catch-up cycle three times on the same node. It asserts nothing beyond the single restart test, but lifecycle bugs show up on the second or third cycle: goroutines that never exit, connections cached to a dead address, stale registrations that a fresh start doesn't clean up.

#Step 6: Multi-invariant scenarios

Single-fault tests check one property each. The scenarios in scenarios_test.go check several invariants together, before and after the fault, so a fix for one property can't quietly break another.

#Many topics, one dead node

TestChaos_MultiTopicLeaderElection creates three topics, records which node leads each, and kills whichever node leads the most:

// tests/chaos/scenarios_test.go
for _, topicName := range topics {
    if ledSet[topicName] {
        newID := waitForTopicLeaderID(t, nodes, topicName, deadNodeID, 30*time.Second)
        if newID == deadNodeID {
            t.Fatalf("topic %q: still shows dead node %q as leader", topicName, deadNodeID)
        }
    } else {
        // Topic wasn't led by the dead node — its leader must be untouched.
        survivor := nodeByID(nodes, leaderOf[topicName])
        id, ok := survivor.Coordinator.TopicLeaderNodeID(topicName)
        if !ok || id != leaderOf[topicName] {
            t.Fatalf("topic %q: leader changed from %q to %q without its leader dying", topicName, leaderOf[topicName], id)
        }
    }
}

The second branch matters as much as the first. Failover should be surgical: moving leadership for topics whose leader is still healthy causes needless client reconnects and an availability blip. After the dead node restarts, the test confirms every topic still resolves to a live leader.

#Follower fault integrity — LEO, ISR, and HW together

TestChaos_FollowerFaultIntegrity is the most thorough single scenario. It runs in three phases:

1. Baseline. Produce 30 AckAll messages. Check that every follower's log holds exactly those values in order, that the leader and follower LEOs are 30, that both followers are in the ISR, and that the leader's stored HW equals TopicMinISRLeo.
2. Fault. Kill one follower and produce 20 more with AckLeader. The surviving follower must catch up to all 50 in order. The HW must then self-heal to 50: the leader's periodic ExpireStaleISR demotes the dead follower once it has gone ISRLagTime without fetching, and the HW stops waiting on it. The test shortens ISRLagTime to 2 seconds so this happens quickly. It's the same mechanism as the production default of 30 seconds (matching Kafka's replica.lag.time.max.ms), only tuned for a fast test.
3. Recovery. Restart the dead follower. Its log must catch up to exactly the same 50 values, both followers must return to the ISR, and the HW must again equal the full LEO.

// tests/chaos/scenarios_test.go — phase 2
leader.TopicManager.ISRLagTime = 2 * time.Second
...
for {
    midHW = leader.Coordinator.TopicMinISRLeo(topicName, leaderLog.LEO())
    if midHW >= total {
        break
    }
    if !time.Now().Before(deadline) {
        t.Fatalf("HW stuck at %d (want %d): still pinned to dead follower %s's stale ISR entry ...", ...)
    }
    time.Sleep(200 * time.Millisecond)
}

Without ISR expiry, one crashed follower would pin the HW for good, and the topic would go read-only for every consumer. This assertion keeps that from happening.

#The full loop — producer to consumer across a failover

The tests above read logs directly. TestChaos_ProducerConsumerFullLoopUnderLeaderKill goes through the real wire protocol from end to end. It sends 40 AckAll messages through the producer client and kills the leader at message 15. Then it opens a consumer client and polls offsets 0 to 39:

// tests/chaos/scenarios_test.go
for offset := uint64(0); offset < total; offset++ {
    pollCtx, cancel := context.WithTimeout(ctx, 20*time.Second)
    entry, err := cc.Poll(pollCtx, offset, 50*time.Millisecond)
    cancel()
    if err != nil {
        t.Fatalf("Poll offset %d: %v", offset, err)
    }
    if entry.Offset != offset {
        t.Fatalf("Poll offset %d: got entry for offset %d instead", offset, entry.Offset)
    }
    consumed = append(consumed, string(entry.Value))
}
assertEqualValues(t, "consumer-observed order vs producer-observed order", produced, consumed)

The server rejects any read past the HW, so if a consumer reaches offset i, then i was committed. Matching the producer's sequence exactly means no gaps, no duplicates, and no reordering across the leader change.

#Step 7: Check linearizability with Porcupine

#Why fixed-order assertions aren't enough

Every test so far uses a single producer, so the expected sequence is known in advance. Real clusters have many concurrent producers, and the correct final order isn't predetermined. Any interleaving is fine as long as it is consistent with what each client saw. A broken failover could let two concurrent appends both get offset 41, or let a client be told "offset 12" for a value that actually landed at 13. A single-producer test can't catch either bug.

The property you want is linearizability: there's some single total order of operations in which each operation takes effect at an instant between its call and its return, and every result matches that order. Porcupine is a Go checker that takes a recorded history of operations (input, output, call time, return time) and a sequential model of the system, and searches for such an order.

#The model: a single FIFO log

// tests/chaos/porcupine_test.go
type logOp struct {
    isAppend bool
    value    string
    offset   int
}

type logOutput struct {
    offset int
    value  string
    found  bool
}

var logModel = porcupine.Model{
    Init: func() interface{} { return []string{} },
    Step: func(state, input, output interface{}) (bool, interface{}) {
        st := state.([]string)
        in := input.(logOp)
        out := output.(logOutput)
        if in.isAppend {
            if out.offset != len(st) {
                return false, st
            }
            next := make([]string, len(st)+1)
            copy(next, st)
            next[len(st)] = in.value
            return true, next
        }
        if in.offset < 0 || in.offset >= len(st) {
            return false, st
        }
        return out.found && out.value == st[in.offset], st
    },
    Equal: func(a, b interface{}) bool {
        return slices.Equal(a.([]string), b.([]string))
    },
}

The model's state is just the slice of values appended so far. An append is legal only if the server reported the offset equal to the current length, which means it landed at the next slot. A read at offset i is legal only if at least i+1 values exist and the value returned is exactly what's at index i. Step returns the new state alongside the verdict, which is why the append path copies the slice instead of mutating it: Porcupine explores many candidate orderings from the same state.

#Recording a concurrent history

Producers run in parallel goroutines, so the recorder needs a mutex:

type opRecorder struct {
    mu      sync.Mutex
    history []porcupine.Operation
}

func (r *opRecorder) record(clientID int, in logOp, call time.Time, out logOutput, ret time.Time) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.history = append(r.history, porcupine.Operation{
        ClientId: clientID,
        Input:    in,
        Call:     call.UnixNano(),
        Output:   out,
        Return:   ret.UnixNano(),
    })
}

TestChaos_LinearizableProduceUnderLeaderKill starts four producers, each sending 25 AckAll messages and recording the call time, return time, and assigned offset of every one. A separate goroutine waits 300 ms, finds the current leader, and kills it while all four producers are mid-stream:

go func() {
    time.Sleep(300 * time.Millisecond)
    leaderID := waitForTopicLeaderID(t, nodes, topicName, "", 15*time.Second)
    leader := nodeByID(nodes, leaderID)
    if leader == nil {
        return
    }
    t.Logf("chaos: killing leader %s mid-produce", leader.NodeID)
    _ = leader.Kill()
}()

Once all producers finish, the test reads every offset back from the new leader and adds those reads to the same history, so appends and reads are checked against one model. Then it runs the checker:

result, info := porcupine.CheckOperationsVerbose(logModel, rec.history, 0)
if result != porcupine.Ok {
    const visPath = "/tmp/mlog-chaos-porcupine.html"
    if f, ferr := os.Create(visPath); ferr == nil {
        defer f.Close()
        if verr := porcupine.Visualize(logModel, info, f); verr == nil {
            t.Logf("visualization written to %s", visPath)
        }
    }
    t.Fatalf("produce/read history under leader kill is not linearizable (result=%v)", result)
}

On failure, porcupine.Visualize writes an interactive HTML timeline of every client's operations and the longest linearizable prefix Porcupine found. That turns "the test failed" into "these two appends overlapped and both claimed offset 41", which is where debugging starts.

TestChaos_LinearizableProduceUnderFollowerKill runs the same workload but kills a follower. With a follower down, every AckAll send has to wait until ExpireStaleISR removes the dead follower from the ISR, so the history only checks out if ISR expiry finishes inside the producers' timeouts. The test shortens ISRLagTime to 2 seconds so there's comfortable margin under the 5-second AckAll wait.

#Step 8: Partition the network with FaultNet

#Why a crash isn't enough

Kill covers the most common failure, but a crashed node is the easy case: it stops doing anything. A partitioned node keeps running. It still believes whatever it believed when the network broke, including that it leads its topics. It keeps accepting connections from any client that can still reach it. Most of the hard bugs in replicated systems live in that gap between "I think I'm the leader" and "the cluster has moved on".

#The seam: transport.Network

To partition nodes inside one test process, every node-to-node connection has to pass through something the test controls. The earlier pages built that seam: transport.Network, a two-method interface (DialTimeout for TCP, AllowPacket for inbound UDP). config.Config.Network picks the implementation. Raft dials peers through it (its stream layer), Serf gossips through it (a memberlist transport wrapper), and replication fetches dial through it (TopicManager.SetNetwork). In production it's transport.DefaultNetwork. A test gives each node its own view of a network it can cut.

A TCP proxy per link, in the style of Toxiproxy, looks like the obvious alternative, but it can't work here. Nodes learn each other's addresses through gossip: Serf tags carry every node's Raft and RPC address, and memberlist advertises each node's gossip address. Everyone dials the same advertised address, so there's no per-link address where a proxy could sit. Injecting at the dial site sidesteps that, because the dialing node always knows who it is.

#FaultNet

tests.FaultNet is that network. The cluster harness registers each node's three addresses (gossip, Raft, RPC) so the network can map any address back to a node:

// tests/faultnet.go
type FaultNet struct {
    mu      sync.Mutex
    owner   map[string]string        // canonical "ip:port" -> node ID
    cut     map[[2]string]bool       // directed (from node, to node) links that are cut
    blocked map[[2]string]bool       // (from node, canonical addr) single-address blocks
    conns   map[*faultConn]struct{} // live connections, so a new cut can sever them
}

func (f *FaultNet) Node(nodeID string) transport.Network // one node's view
func (f *FaultNet) Partition(groups ...[]string)         // groups can't reach each other
func (f *FaultNet) CutOneWay(from, to string)            // asymmetric cut
func (f *FaultNet) BlockAddr(fromNode, addr string)      // block one destination address only
func (f *FaultNet) Heal()

The rules follow how the real protocols behave:

• TCP needs both directions. A dial from node A to an address owned by node B fails if either A→B or B→A is cut. The dialing side enforces it, since that's the side that knows both ends. An inbound connection's source port is ephemeral and identifies nobody.
• Existing connections die at the cut. Every connection dialed through the network is tracked. When a cut is made, the connections it forbids are closed immediately, so a pooled Raft connection or a cached replication connection can't keep a partitioned link alive.
• UDP is one-way. Memberlist sends UDP from its bound gossip socket, so an inbound packet's source address is the sender's gossip address. A packet is dropped if the link from its sender to the receiver is cut.
• Unregistered addresses are never blocked. The test's own producer and consumer clients don't go through FaultNet at all, so a test can still reach both sides of a partition.

func (f *FaultNet) tcpAllowedLocked(from, addr string) bool {
    if f.blocked[[2]string{from, addr}] {
        return false
    }
    to, ok := f.owner[addr]
    if !ok {
        return true
    }
    return !f.cut[[2]string{from, to}] && !f.cut[[2]string{to, from}]
}

A blocked dial fails immediately with a *net.OpError, so callers classify it as an ordinary network failure. A real partition usually shows up as timeouts rather than instant failures. Failing fast keeps the tests quick and doesn't change any decision the cluster makes: every caller treats both as "unreachable".

tests.StartRealThreeNodeClusterOnNetwork(t, prefix, fn) builds the usual 3-node cluster with each node's config.Config.Network set to fn.Node(nodeID), and Restart keeps a node on the same network.

#Step 9: Test partitions and divergent logs

#A restarted leader with a divergent tail

TestChaos_RestartedLeaderTruncatesDivergentTail creates the divergent-log scenario from the replication page on purpose:

1. Produce 10 AckAll records, so every replica has offsets 0–9.
2. Block only the followers' connections to the leader's RPC port with BlockAddr. Replication stops, but the leader keeps its Raft and Serf links, so it stays leader.
3. Produce 5 AckLeader records. The leader acknowledges them, and only the leader has them. After two replication ticks, the test checks that the followers still hold exactly 10 records.
4. Kill the leader and heal the network.
5. A new leader is elected. Produce 7 AckAll records through it. Offsets 10–16 now mean something different on the new leader than on the old leader's disk.
6. Restart the old leader.

Then the test waits for every node, the old leader included, to hold exactly the new leader's 17 records:

// tests/chaos/partition_test.go
want := append(append([]string{}, committed...), afterFailover...)
for _, n := range nodes {
    waitForLogValues(t, n, topicName, want, 45*time.Second)
}

waitForLogValues waits for an exact match, not just a LEO of 17. A node that kept its divergent tail reaches LEO 17 too, with the wrong records at offsets 10–14, so checking only LEO would miss the bug entirely. The test also checks that the restarted node's epoch history equals the new leader's. Step 3 makes the scenario deterministic: without BlockAddr, whether any records stay unreplicated would depend on when the leader is killed relative to the 1-second replication tick.

#A partitioned leader is fenced, then converges

TestChaos_PartitionedLeaderIsFencedAndConverges cuts the topic leader off from both followers while the test can still reach every node:

fn.Partition([]string{oldLeaderID}, nodeIDs(majority))

While the partition lasts, the test checks what the isolated leader can and can't do:

• It still accepts AckLeader writes. Those are acknowledged by the leader alone, so losing them is within that ack mode's contract. They also give the old leader a divergent tail to clean up later.
• It never acknowledges an AckAll write. It can't reach the ISR, so the write times out with an error.
• Its high watermark doesn't move past the 10 records the followers have, and a consumer Fetch at offset 10 is refused. The test shortens the old leader's ISRLagTime to 5 seconds, so the partition outlasts it. That means ISR expiry has had every chance to fire. Only the quorum check in expireStaleISR keeps the old leader from demoting its unreachable followers and exposing its own writes.
• It stops directing clients to itself. Once it reports no Raft leader, FindTopicLeader on it returns an error instead of its own address.

Meanwhile, the majority side elects a new leader through the normal path (Serf marks the old leader failed, Raft removes it, ReassignLeadersForDeadNode picks an in-sync replica) and accepts 10 more AckAll writes through a producer bootstrapped with only the majority's addresses.

Then the test heals the partition. Serf's reconnect loop finds the old leader again, Raft adds it back as a voter, its metadata catches up to the new leader and epoch, and replication takes over. Its first fetch carries epoch 1 and an offset past the majority's epoch-1 records, so the new leader answers Diverging. The old leader truncates its zombie writes and fetches forward. The test waits for every node to hold exactly the 20 committed records, writes 5 more, and checks that all three nodes get them and agree on epoch history.

#Checking that the tests catch the bugs

A fault-injection test only earns trust once you've seen it fail. For each guarantee, disable the code that provides it, rebuild, and run the test:

Remove	Test that fails, and where
The divergence check (always serve records)	Both tests: the old leader ends with the right LEO and wrong records at offsets 10–14
The quorum check in expireStaleISR	Partition test: the isolated leader's HW jumps to its own LEO
The strict offset >= hw bound in LogManager.Read	Partition test: the isolated leader serves offset 10, a write that's about to be discarded
The quorum check in GetTopicLeaderRPCAddr	Partition test: the isolated node still names itself as leader

If a test passes with its guarantee removed, it isn't testing that guarantee.

#Step 10: Run the suite

make test-chaos

or, to run a single test:

go test -tags chaos -v -run TestChaos_LinearizableProduceUnderLeaderKill ./tests/chaos/...

Expect the suite to take minutes, not seconds — most of that is Serf's failure detection and Raft elections running in real time. Each passing test logs a final ✓ line summarizing what it proved, for example:

✓ all 20 AckAll messages survived leader kill
✓ HW self-healed to 50 after dead follower node2 expired from ISR
✓ 200 operations (100 appends, 100 reads) linearizable under leader kill
✓ isolated leader node-1 was fenced during the partition and converged on 25 records after healing

Because fault timing varies from run to run, a chaos suite finds bugs statistically. Run it repeatedly, especially after touching replication, ISR, or leader election code:

go test -tags chaos -count=20 -run TestChaos_ ./tests/chaos/...

A test that fails once in twenty runs is reporting a real race, not flakiness. Read the logs and the Porcupine visualization before you consider retrying it.

#Step 11: Extend the fault model

The suite injects two kinds of fault: a clean process stop and a symmetric network partition, one at a time. Together they cover leader failover, ISR expiry, catch-up, fencing an isolated leader, and divergent-log truncation. Production clusters fail in more ways than that, though. Each fault below is worth a test of its own, and each one targets a specific invariant from Step 1.

#Asymmetric partitions

FaultNet.CutOneWay already models a link that works in only one direction. Gossip, Raft heartbeats, and replication fetches each have their own idea of who's alive, and one-way failures are where those views disagree. For example, cut leader→follower UDP only: the follower's probes of the leader fail, while the leader still hears the follower. Then check that the cluster settles on a single leader rather than flapping.

#Partitions on a real deployment

FaultNet partitions nodes inside one process. To run the same scenarios against the Docker Compose cluster, use iptables rules inside the containers, tc netem for latency and loss, or Chaos Mesh's NetworkChaos on Kubernetes. The assertions carry over unchanged. Only the fault injector differs.

#Losing quorum

Kill two of three nodes. Raft has no majority, so no metadata changes can commit and no AckAll write can be acknowledged. Check that producers get errors or timeouts rather than false successes. Then restart one node and check that the cluster resumes on its own, with every write acknowledged before the outage still present.

#Killing the Raft leader rather than the topic leader

The Raft metadata leader and a topic's data leader can be different nodes. Target the Raft leader specifically, during a CreateTopic or right after another node died, while the cluster is still mid-reassignment. The invariant: every topic ends up with exactly one live leader, and no CreateTopic that returned success disappears.

#Slow nodes, not dead nodes

Add latency and packet loss to one follower's link instead of killing it. The follower stays alive in Serf, but its fetches fall behind. It should leave the ISR through lag rather than death, and AckAll latency should recover once it does. Pausing a process (SIGSTOP, then SIGCONT a few seconds later, or docker pause) is a sharper version of the same fault. It mimics a long GC pause, and a paused leader that wakes up must not keep acting on leadership it has already lost.

#Crash consistency on disk

Kill stops a node cleanly inside a live process, so the operating system's page cache survives and every write the broker made is still on disk. A real power loss can leave a torn write at the end of a segment, or an index entry pointing past the end of the log file. Test recovery by running the broker as a real child process and sending SIGKILL mid-write, or more deterministically, by truncating a few bytes off the active segment and index between Kill and Restart. On startup the segment must detect the partial record, truncate back to the last complete one, and rebuild its index. Disk-full and fsync-error injection belong in the same category.

#Ambiguous operations in the linearizability check

The Porcupine tests currently fail the test if any Send returns an error. In a real failover, a send can time out after the server appended the record. The client doesn't know whether it took effect. Record those operations as indeterminate instead of failing: give them an unknown output and a return time of "infinitely late". Porcupine then accepts histories where the write either did or didn't happen. This lets you shorten producer timeouts and inject harsher faults without false failures.

Also run reads concurrently with appends, through the consumer client rather than ReadUncommitted. That way the checker also verifies that consumers never see a value that later disappears, which is the HW guarantee from the consumer's point of view.

#Randomized, long-running schedules

Every test here follows a script: one fault, at one point in time. Once you've built several injectors, combine them into a nemesis loop that picks a random fault (kill, restart, partition, heal, pause), applies it at a random moment, repeats for several minutes under concurrent load, heals everything, and then runs the full invariant check and the Porcupine check. Seed the random generator, log the seed, and make the seed a flag so any failure can be replayed exactly. This is how Jepsen finds the bugs that scripted tests miss: the ones that need two faults in a particular order.

#Consumer offsets under failover

Committed consumer offsets are state too. Commit offsets from a consumer group, kill the leader, and check that a reconnecting consumer resumes from the committed offset. It shouldn't restart from zero or skip ahead.

#Summary

Piece	Purpose
chaos build tag	Keeps slow, real-cluster tests out of go test ./...; run with make test-chaos
Kill / Restart	Hard-stops a node without a graceful Serf leave, then brings it back from its own data directory
Wait helpers	Poll observable cluster state (leader, voter removal, ISR, LEO) with deadlines instead of sleeping
Single-fault tests	Leader kill loses no acked write; restarted followers catch up; repeated restarts don't leak state
Scenarios	Surgical multi-topic re-election; LEO, ISR, and HW invariants before, during, and after a follower fault; gap-free consumption across a failover
Porcupine	Concurrent produce histories under leader or follower kill are linearizable against a single-FIFO-log model, with an HTML visualization on failure
FaultNet	An in-process network behind transport.Network that Raft, Serf, and replication all dial through; partitions, one-way cuts, and single-address blocks
Partition tests	A restarted leader truncates its divergent tail; a partitioned leader is fenced (no AckAll acks, no HW movement, no leader advertisements) and converges after healing
Next faults	Asymmetric partitions, quorum loss, Raft-leader kills, slow and paused nodes, torn writes, ambiguous operations, and seeded nemesis loops

A distributed log is only as trustworthy as the faults it has survived under test. With this suite in place, every guarantee from the earlier pages has a test that checks it while the cluster is failing.