Distributed Systems Challenges in Practice — Idempotency, Retries, Fencing and Scaling in a Workflow Engine

On one machine, a function call either happens or it doesn't. Across machines, there is a third outcome: you don't know. The request might have arrived and been processed, arrived and been lost, or never arrived at all — and the caller sees the same thing in all three cases: a timeout.

Almost every hard problem in distributed systems comes from that third outcome. This post walks through the problems that show up in practice, and how Composer — a distributed workflow engine written in Go, on PostgreSQL and Redis — solves each one.

A quick map of the system, so the examples make sense:

• A coordinator holds the workflow rules. Every input — an API call, a worker's result, a timer — becomes an event that is applied to one workflow run.
• PostgreSQL is the source of truth: runs, tasks, timers, and an outbox of messages to send.
• Redis Streams carry tasks to workers, which execute HTTP calls or user code and report results.

Challenge	Composer's answer
Partial failure inside one decision	one database transaction per event
Dual writes (database + message queue)	transactional outbox
Concurrent updates to the same run	a row lock per run
Duplicate delivery and retries	at-least-once delivery, leases, idempotent handlers, idempotency keys
Zombie workers and late results	fencing with an attempt number
Detecting dead workers	lease renewal as a heartbeat, plus recovery sweeps
Lost messages	re-dispatching tasks that have no pending message
Retry storms and poison messages	exponential backoff with jitter, delivery limits
Scaling	stateless coordinators, sharded queues, workers scaled per queue

---

#1. Partial failure: one decision, many writes

The problem. When a task finishes, the engine has to do several things: mark the task completed, create the next task, maybe finish the run. If the process crashes halfway, you can end up with a completed task and no next task — a workflow that silently stops forever.

The solution: one transaction per event. Composer splits the decision from the writes. A pure state machine decides what should change and returns a list of actions; the coordinator then writes all of those actions inside one PostgreSQL transaction:

// handleWith locks and loads the run, lets prepare pick the definition and
// event, then decides and persists — all in one transaction.
func (c *Coordinator) handleWith(ctx context.Context, runID string, logger zerolog.Logger,
    prepare func(*statemachine.Run) (*model.WorkflowDAG, statemachine.Event, error)) (*statemachine.Transition, error) {

    var tr *statemachine.Transition
    err := c.inTx(ctx, func(t *txn) error {
        run, err := c.repo.LoadRun(ctx, t, runID)
        if err != nil {
            return err
        }
        dag, ev, err := prepare(run)
        if err != nil {
            return err
        }
        tr, err = c.decide(ctx, t, logger, dag, run, ev, time.Now())
        return err
    })
    if err != nil {
        logger.Error().Err(err).Msg("event not applied")
    }
    return tr, err
}

If anything fails — a crash, a constraint violation, a lost database connection — the transaction rolls back and the run is exactly as it was. The caller gets an error and retries the whole event.

One subtle detail: side effects that are not in the database, like metrics and log lines, must only happen if the transaction commits. Otherwise dashboards would count events that never happened. Composer attaches them to the transaction as after-commit hooks:

// txn is a transaction plus side effects (metrics, logs) that must only
// happen if it commits.
type txn struct {
    pgx.Tx
    afterCommit []func()
}

func (t *txn) onCommit(fn func()) {
    t.afterCommit = append(t.afterCommit, fn)
}

// inTx runs fn in a transaction and commits it if fn succeeds.
func (c *Coordinator) inTx(ctx context.Context, fn func(t *txn) error) error {
    tx, err := c.DB.Begin(ctx)
    if err != nil {
        return fmt.Errorf("begin transaction: %w", err)
    }
    defer tx.Rollback(ctx) //nolint:errcheck // no-op after commit

    t := &txn{Tx: tx}
    if err := fn(t); err != nil {
        return err
    }
    if err := tx.Commit(ctx); err != nil {
        return fmt.Errorf("commit transaction: %w", err)
    }
    for _, f := range t.afterCommit {
        f()
    }
    return nil
}

---

#2. The dual-write problem

The problem. A transaction protects writes to one database. But dispatching a task means writing to two systems: the task row in PostgreSQL and a message in Redis. There is no transaction that spans both, so whichever order you choose, a crash in between breaks something:

The solution: a transactional outbox. Instead of publishing to Redis directly, the coordinator inserts the message into an outbox table in the same transaction as the task row. Now both writes commit or neither does:

// EnqueueDispatch writes the task's dispatch message to the outbox, addressed
// to one shard of the task's queue. The outbox enqueuer publishes it no
// earlier than notBefore.
func (r repository) EnqueueDispatch(ctx context.Context, q db.Querier, d *taskDispatch, notBefore time.Time) error {
    payload, err := json.Marshal(map[string]any{
        "task_id":          d.TaskID,
        "run_id":           d.RunID,
        "workflow_id":      d.WorkflowID,
        "workflow_version": d.WorkflowVersion,
        "node_id":          d.NodeID,
        "node_type":        d.NodeType,
        "attempt":          d.Attempt,
        "input":            string(d.Input),
    })
    if err != nil {
        return fmt.Errorf("marshal dispatch payload: %w", err)
    }
    streamName := stream.TaskStream(d.Queue, stream.ShardFor(d.TaskID, r.streamShards))
    if _, err := q.Exec(ctx, `
        INSERT INTO outbox (id, task_id, stream, payload, type, status, next_attempt_at)
        VALUES ($1, $2, $3, $4::jsonb, 'TASK_EXECUTION', 'PENDING', $5)`,
        uuid.NewString(), d.TaskID, streamName, payload, notBefore); err != nil {
        return fmt.Errorf("insert outbox entry for task %s: %w", d.TaskID, err)
    }
    return nil
}

A separate enqueuer reads committed outbox rows, publishes them to Redis, and deletes each row after a successful publish:

The failure modes change from "lost forever" or "phantom task" to just one: the enqueuer publishes a row and crashes before deleting it, so the row is published again. A duplicate is much easier to handle than a loss — section 4 shows how.

#Making the outbox fast

An outbox that is polled every second adds up to a second of latency to every step. Composer adds a trigger that fires NOTIFY when outbox rows are inserted. PostgreSQL delivers notifications only when the transaction commits, which is exactly the moment the row becomes visible:

CREATE OR REPLACE FUNCTION composer_notify_outbox() RETURNS TRIGGER
LANGUAGE plpgsql AS $$
BEGIN
    PERFORM pg_notify('outbox_new', '');
    RETURN NULL;
END;
$$;
DROP TRIGGER IF EXISTS outbox_notify ON outbox;
CREATE TRIGGER outbox_notify AFTER INSERT ON outbox
    FOR EACH STATEMENT EXECUTE FUNCTION composer_notify_outbox();

The enqueuer LISTENs on a dedicated connection and wakes immediately. Polling stays as a fallback for lost notifications and for retries whose delay has come due.

#Many enqueuers, no double work

Every coordinator replica runs an enqueuer. To stop two of them from publishing the same row, each one claims rows with FOR UPDATE SKIP LOCKED — rows locked by another enqueuer are simply skipped instead of waited on:

UPDATE outbox SET status = 'IN_FLIGHT', updated_at = NOW()
 WHERE id IN (
       SELECT id FROM outbox
        WHERE status IN ('PENDING', 'RETRY')
          AND (next_attempt_at IS NULL OR next_attempt_at <= NOW())
        ORDER BY created_at
        LIMIT $1
          FOR UPDATE SKIP LOCKED)
RETURNING id, stream, payload, attempt_count, max_retries, next_attempt_at, backoff_seconds

This turns a plain table into a work queue that any number of consumers can drain in parallel.

---

#3. Concurrent updates: two events for the same run

The problem. A workflow forks into two branches, and both finish at the same instant on two different workers. Both reports arrive at the coordinator in parallel. If each transaction reads the run without seeing the other's write, each concludes the other branch is still running — and neither starts the join node. The run hangs forever.

This is write skew: each transaction writes only its own task row, and each decides based on a read the other one changes. Transactions alone don't prevent it at PostgreSQL's default isolation level.

The solution: a row lock per run. The first statement of every event's transaction locks the run's row:

// LockRun takes the run's row lock for the rest of the transaction. Every
// state transition of a run happens under this lock, which serializes
// concurrent completions so join decisions always see each other's writes.
func (repository) LockRun(ctx context.Context, q db.Querier, runID string) (*runRow, error) {
    run := &runRow{ID: runID}
    err := q.QueryRow(ctx, `
        SELECT workflow_definition_id, workflow_version, status
          FROM workflow_runs
         WHERE id = $1 AND deleted_at IS NULL
           FOR UPDATE`,
        runID).Scan(&run.WorkflowID, &run.WorkflowVersion, &run.Status)
    if errors.Is(err, pgx.ErrNoRows) {
        return nil, fmt.Errorf("%w: workflow run %s", ErrNotFound, runID)
    }
    if err != nil {
        return nil, fmt.Errorf("lock workflow run: %w", err)
    }
    return run, nil
}

The lock is fine-grained: it serializes events within one run, while thousands of other runs proceed in parallel. And because the lock lives in PostgreSQL, not in the coordinator's memory, any coordinator replica can handle any run — there is no need to route a run to a particular machine.

The database also backs this up with a constraint. A unique index on (run_id, node_id) means that even if two decisions somehow raced, the second attempt to create a task for the same node would fail:

CREATE UNIQUE INDEX IF NOT EXISTS idx_task_instances_run_node ON task_instances(run_id, node_id);

---

#4. Duplicate delivery, retries and idempotency

The problem. Retries are the only way to survive failures, but a retry can't tell a lost request from a lost response:

Every messaging system therefore has to pick one:

Guarantee	What it means	Cost
At-most-once	never retry	work is silently lost
At-least-once	retry until acknowledged	work can run more than once
Exactly-once	each effect happens exactly once	only possible when every participant cooperates (deduplication, transactions)

Composer chooses at-least-once and makes duplicates harmless at every layer.

#Layer 1: duplicate messages are dropped by a lease

A task message can be delivered twice: the outbox republished it, or a crashed worker's message was reclaimed. Before executing, a worker must acquire the task's lease. It's one conditional UPDATE, so the database decides who wins:

func (m *Manager) Acquire(ctx context.Context, runID, taskID, owner string, ttl time.Duration) (bool, error) {
    return m.exec(ctx, `
        UPDATE task_instances
           SET state = 'RUNNING',
               lease_owner = $1,
               lease_expires_at = NOW() + make_interval(secs => $2),
               started_at = NOW(),
               updated_at = NOW()
         WHERE run_id = $3 AND id = $4
           AND (state IN ('PENDING', 'QUEUED')
                OR (state = 'RUNNING' AND lease_expires_at < NOW()))
        RETURNING id`,
        owner, ttl.Seconds(), runID, taskID)
}

A second copy of the message finds the task already RUNNING under a live lease (or already COMPLETED), the UPDATE matches no row, and the worker drops the message. Two workers can never execute the same task at the same time.

#Layer 2: duplicate results are ignored by the state machine

A worker may report the same result twice (its first report succeeded but the acknowledgement was lost). The state machine ignores any result for a task that is no longer active:

func (m *Machine) onNodeCompleted(ev NodeCompleted) error {
    n := m.run.Nodes[ev.NodeID]
    if !n.matches(ev.TaskID, ev.Attempt) || !n.isTaskActive() {
        return m.ignore("stale or duplicate completion of %s attempt %d", ev.NodeID, ev.Attempt)
    }
    m.emit(CompleteTask{TaskID: ev.TaskID, NodeID: ev.NodeID, Attempt: ev.Attempt, Output: ev.Output})
    if m.terminal() {
        return nil // e.g. cancelled while running: keep the output, schedule nothing
    }
    return m.advanceFrom(ev.NodeID, ev.Output)
}

An ignored report is still answered with "acknowledged", so the worker stops retrying it. Making the handler idempotent — applying it twice has the same effect as applying it once — is what makes at-least-once delivery safe.

#Layer 3: external side effects carry an idempotency key

The engine can deduplicate everything inside its own database. It cannot reach into a payment provider. For that, the HTTP executor sends the task ID as an Idempotency-Key header:

    method := params.Method
    if method == "POST" || method == "PUT" || method == "PATCH" {
        if req.Header.Get("Content-Type") == "" && len(params.Body) > 0 {
            req.Header.Set("Content-Type", "application/json")
        }
        // Use task_instances.id as the idempotency key so services that honour
        // the Idempotency-Key header (Stripe, many REST APIs) deduplicate
        // retried requests automatically.  The header is only set when the task
        // ID is present in the context and the caller has not already set one.
        if taskID := TaskIDFromContext(ctx); taskID != "" && req.Header.Get("Idempotency-Key") == "" {
            req.Header.Set("Idempotency-Key", taskID)
        }
    }

The task ID stays the same across every redelivery and every retry of a step, so an API that honors the header performs the charge once, however many times the request arrives. This is how "exactly-once" is actually achieved in practice: at-least-once delivery plus deduplication at the receiver.

---

#5. Zombie workers and fencing

The problem. Leases have a nasty edge case. A worker pauses — a long garbage-collection pause, a VM migration, a network stall — for longer than its lease. The lease expires, another worker takes over — and the step may well fail there and be retried as a new attempt. Then the first worker wakes up, unaware that any time has passed, and reports its old result.

Without protection, that stale report would complete a task that is now being run again as a different attempt, with a result from the superseded one. A lease alone can't stop this, because the paused worker doesn't know its lease expired.

The solution: a fencing token. Every task carries an attempt number. It travels with the task through the whole loop — dispatch message, lease, execution, report — and every write that applies a result requires it to match:

The check appears twice. In the state machine (n.matches(ev.TaskID, ev.Attempt) above), and again in SQL, where the WHERE clause only matches the current attempt of a still-active task:

// CompleteTask marks the referenced attempt COMPLETED and releases its lease.
// It reports false when the attempt is stale or the task already finished.
func (repository) CompleteTask(ctx context.Context, q db.Querier, ref taskRef, output json.RawMessage) (*finishedTask, bool, error) {
    return scanFinishedTask(q.QueryRow(ctx, `
        UPDATE task_instances
           SET state = 'COMPLETED', output_json = $1::jsonb,
               completed_at = NOW(), finished_at = NOW(), updated_at = NOW(),
               lease_owner = NULL, lease_expires_at = NULL
         WHERE run_id = $2 AND id = $3 AND node_id = $4 AND attempt = $5
           AND state IN ('PENDING', 'QUEUED', 'RUNNING')
        RETURNING node_type, started_at`,
        nullJSON(output), ref.RunID, ref.TaskID, ref.NodeID, ref.Attempt))
}

A retry bumps the attempt in the same statement that resets the task, so any report for the old attempt can never match again:

UPDATE task_instances
   SET state = 'PENDING', attempt = attempt + 1, last_error = $1::jsonb,
       started_at = NULL, updated_at = NOW(),
       lease_owner = NULL, lease_expires_at = NULL
 WHERE run_id = $2 AND id = $3 AND node_id = $4 AND attempt = $5
   AND state IN ('PENDING', 'QUEUED', 'RUNNING')

Two cases are covered between them. A report for a superseded attempt fails the attempt = $5 check. A report for the current attempt of a task that already finished — for example after an expired lease is re-dispatched with the same attempt and both workers report — fails the state IN (…) check, so exactly one of the two results is applied: whichever arrives first.

The general rule, from Martin Kleppmann's well-known analysis of distributed locks: a lock or lease is not enough on its own; the resource being protected must check a monotonically increasing token. Here the resource is the task row, and the token is the attempt number.

---

#6. Detecting failure: heartbeats and leases

The problem. You can't tell a dead worker from a slow one. You can only notice that it stopped saying anything.

The solution: the lease renewal is the heartbeat. A worker renews its lease every third of the lease TTL while it works. Renewal is per task, so liveness is tracked exactly where it matters — there is no separate "worker registry" that could disagree with the tasks:

// renewLeaseLoop renews the task lease every TTL/3 until ctx is canceled. If
// the lease is lost it cancels the execution with errLeaseLost.
func (w *Worker) renewLeaseLoop(ctx context.Context, msg *TaskMessage, cancel context.CancelCauseFunc, logger zerolog.Logger) {
    renewInterval := max(time.Duration(w.cfg.LeaseTTLSeconds/3)*time.Second, 5*time.Second)
    ticker := time.NewTicker(renewInterval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            renewed, err := w.coordClient.RenewTaskLease(ctx, msg, w.cfg.ConsumerName, w.cfg.LeaseTTLSeconds)
            switch {
            case err != nil:
                // Transient; the lease survives until its TTL, so keep trying.
                logger.Warn().Err(err).Msg("failed to renew task lease")
            case !renewed:
                cancel(errLeaseLost)
                return
            }
        }
    }
}

Three details make this robust:

• Renewing at TTL/3 means one slow or failed renewal never costs the lease — two more tries fit before it expires.
• Lease times come from the database clock (NOW()), never the worker's, so clock skew between machines can't break it.
• A worker that loses its lease stops. If renewal reports the lease gone, the execution is cancelled and its result discarded. Another worker owns the task now.

When a worker dies, two loops recover its work. Other workers reclaim its unacknowledged stream messages (XAUTOCLAIM), and every coordinator runs a sweep that resets expired leases and re-dispatches them — in one statement, safe for many coordinators to run at once:

func (r repository) RequeueExpiredLeases(ctx context.Context, q db.Querier) (int64, error) {
    tag, err := q.Exec(ctx, `
        WITH expired AS (
            UPDATE task_instances
               SET state = 'PENDING', lease_owner = NULL, lease_expires_at = NULL,
                   started_at = NULL, updated_at = NOW()
             WHERE state = 'RUNNING' AND (lease_expires_at IS NULL OR lease_expires_at < NOW())
            RETURNING id, run_id, node_id, node_type, queue, attempt, input_json
        )`+dispatchInsert+`
          FROM expired t JOIN workflow_runs r ON r.id = t.run_id`,
        r.shards())
    if err != nil {
        return 0, fmt.Errorf("requeue expired leases: %w", err)
    }
    return tag.RowsAffected(), nil
}

The UPDATE … RETURNING feeds straight into the outbox INSERT, so resetting the task and re-dispatching it are atomic, and each expired task is claimed by exactly one coordinator.

---

#7. Lost messages

The problem. Even with an outbox, a message can vanish after it's published — Redis restarts without persistence, say. Then a task sits PENDING in the database with nothing in any queue. Nobody will ever run it.

The solution: treat the database as the truth and the queue as a hint. A recovery sweep looks for exactly that shape — a pending task of a running workflow, untouched for ten minutes, with no outbox row — and dispatches it again:

func (r repository) RedispatchPendingTasks(ctx context.Context, q db.Querier, olderThan time.Duration) (int64, error) {
    tag, err := q.Exec(ctx, `
        WITH lost AS (
            UPDATE task_instances t
               SET updated_at = NOW()
              FROM workflow_runs r
             WHERE r.id = t.run_id
               AND r.status = 'RUNNING'
               AND t.state = 'PENDING'
               AND t.updated_at < NOW() - make_interval(secs => $2)
               AND NOT EXISTS (SELECT 1 FROM outbox o WHERE o.task_id = t.id)
            RETURNING t.id, t.run_id, t.node_id, t.node_type, t.queue, t.attempt, t.input_json
        )`+dispatchInsert+`
          FROM lost t JOIN workflow_runs r ON r.id = t.run_id`,
        r.shards(), olderThan.Seconds())
    if err != nil {
        return 0, fmt.Errorf("redispatch pending tasks: %w", err)
    }
    return tag.RowsAffected(), nil
}

If the task was merely waiting in a busy queue, the extra message is a duplicate — and duplicates are already harmless thanks to the lease. That is the payoff of solving duplication properly: recovery can be aggressive and simple, because sending something twice costs nothing.

---

#8. Retry storms and poison messages

The problem. Retrying immediately and forever causes two kinds of trouble. When a dependency goes down, every client retrying at once keeps it down (a retry storm). And a message that crashes every worker that touches it (a poison message) will take the fleet down one worker at a time.

The solution: back off, add jitter, and give up visibly.

Step retries use exponential backoff, capped:

// RetryDelay returns the backoff before the given retry attempt (1-based).
func (n *Node) RetryDelay(attempt int) time.Duration {
    if n.Retry == nil || n.Retry.BackoffSeconds <= 0 || attempt < 1 {
        return 0
    }
    delay := time.Duration(n.Retry.BackoffSeconds) * time.Second << (attempt - 1)
    if delay <= 0 || delay > maxRetryBackoff {
        return maxRetryBackoff
    }
    return delay
}

The delay costs no worker and no timer goroutine: the retry's outbox row is simply written with next_attempt_at in the future, and the enqueuer won't claim it until then.

Failed publishes to Redis back off with jitter, so that when Redis recovers, every enqueuer doesn't retry at the same instant:

func computeBackoffDuration(baseSeconds, attempt int) time.Duration {
    if baseSeconds <= 0 {
        baseSeconds = 5
    }
    // exponential factor
    exp := float64(baseSeconds) * math.Pow(2, float64(attempt-1))
    // full jitter with guaranteed minimum of baseSeconds
    jitter := float64(baseSeconds) + rand.Float64()*exp
    return time.Duration(int64(jitter)) * time.Second
}

And both paths have a limit. An outbox row that fails too often moves to a dead-letter table. A stream message delivered more than MaxDeliveries times is reported to the engine as a failed step, so the node's retry policy and the run's failure handling apply like any other error — instead of the message circulating forever.

---

#9. Scaling: coordinators and workers, independently

The problem. Load grows in different places. Some workflows are orchestration-heavy (many short steps, lots of decisions); others are execution-heavy (slow HTTP calls, big Python jobs). A design that scales everything together wastes money; a design with one special machine has a ceiling.

The solution: keep every process stateless, and scale each tier on its own.

Coordinators are stateless. All state is in PostgreSQL and per-run ordering comes from the row lock, so any replica can handle any request. There's no leader election and no sticky routing — add replicas behind a load balancer. Background jobs are written to be safe on every replica at once (SKIP LOCKED, single-statement sweeps, timer events the state machine ignores if they're no longer relevant), so they need no coordination either.

Queues are sharded. Each task queue is split into several Redis streams, and a task's shard comes from a hash of its ID:

// ShardFor picks the shard for a task. Any stable spread works: shards only
// distribute load, they carry no ordering guarantee.
func ShardFor(taskID string, shards int) int {
    if shards <= 1 {
        return 0
    }
    h := fnv.New32a()
    _, _ = h.Write([]byte(taskID))
    return int(h.Sum32() % uint32(shards))
}

Each shard is its own key, so in Redis Cluster one busy queue spreads across nodes.

Workers scale per queue. A node can name a dedicated queue (queue: partner-api), and a worker pool serves only the queues it's started with. A slow partner API gets its own pool and can't starve everything else; Python steps scale independently of HTTP steps. Inside a worker, a bounded hand-off channel means a worker fetches only as many messages as it has free execution slots — so work stays spread across the fleet instead of piling up in one process:

// readLoop fetches new messages from one stream while the hand-off channel has room.
func (w *Worker) readLoop(ctx context.Context, s stream.Consumer) {
    for ctx.Err() == nil {
        free := cap(w.deliveries) - len(w.deliveries)
        if free <= 0 {
            w.waitForSpace(ctx)
            continue
        }
        msgs, err := s.ReadNew(ctx, int64(free))
        if err != nil {
            if ctx.Err() == nil {
                w.cfg.Logger.Error().Err(err).Msg("read new messages failed")
                time.Sleep(time.Second)
            }
            continue
        }
        for _, msg := range msgs {
            if !w.hand(ctx, delivery{msg: msg, from: s}) {
                return
            }
        }
    }
}

The database stays fast as it grows. Run data is range-partitioned by month on the run ID (a time-ordered UUIDv7), every hot query filters on the run ID so it touches one partition, indexes are kept few and partial, and old months are dropped whole instead of deleted row by row.

Long waits cost nothing. A workflow that waits two days for an approval is a row with a wake_at time, not a blocked thread. A timer loop on every coordinator fires due waits; if two coordinators fire the same one, the second finds the task no longer waiting and ignores it.

---

#The patterns, in one place

Pattern	The problem it solves	Where in Composer
One transaction per decision	partial writes leaving inconsistent state	inTx around lock → decide → persist
Transactional outbox	writing to a database and a queue atomically	outbox row in the same transaction; enqueuer publishes after commit
LISTEN/NOTIFY	outbox latency	trigger wakes the enqueuer at commit
FOR UPDATE SKIP LOCKED	many consumers of one table	outbox claiming
Row lock per entity	lost updates under concurrency	SELECT … FOR UPDATE on the run
Unique constraints	duplicate creation	one task per (run_id, node_id)
Leases	two workers running one task	conditional UPDATE on the task row
Fencing token	zombie workers, stale results	attempt number checked in the state machine and in SQL
Heartbeat = lease renewal	detecting dead workers	renew every TTL/3; database clock only
Idempotent handlers	duplicate delivery	stale/duplicate reports ignored, still acknowledged
Idempotency keys	duplicate external side effects	task ID as Idempotency-Key
Reconciliation sweeps	lost messages, crashed workers	expired-lease and lost-dispatch recovery
Backoff + jitter + DLQ	retry storms, poison messages	step retries, outbox retries, delivery limits
Stateless services, sharded queues	scaling	coordinators, stream shards, per-queue worker pools

The deepest lesson is how these patterns lean on each other. The outbox is safe because duplicates are harmless. Duplicates are harmless because of leases, idempotent handlers and fencing. Recovery can be blunt because sending something twice costs nothing. Pick at-least-once delivery, make every step idempotent, keep the truth in one transactional store — and most of the remaining distributed-systems problems become ordinary engineering.

If you want to build it yourself, the Building a Distributed Workflow Engine guide walks through every one of these pieces from an empty folder.