Introduction — Building a Distributed Workflow Engine
Building a Distributed Workflow Engine·Part 1 of 11·6 min read

Introduction — Building a Distributed Workflow Engine

On this page · 5 sections
  1. What is a workflow engine?
  2. Well-known workflow engines
  3. What you will build
  4. Guarantees you are aiming for
  5. How this chapter is organized

In this chapter you build Composer, a distributed workflow engine. You describe a process as a graph of steps — call an API, run some Python, branch on the result, wait a day for an approval — and the engine runs that graph for you across many machines, without losing a step and without running one step twice at the same time.

Git repository

The full source code is on GitHub:

https://github.com/Mohitkumar/composer

Clone and build it locally:

git clone https://github.com/Mohitkumar/composer.git
cd composer
make install-deps
make proto
make build

Every code listing in this chapter comes from that repository. You can read along, or type it in yourself package by package.

What is a workflow engine?

Most backend work is not one function call. It is a sequence of steps, and each step can fail:

An order process as four steps: charge card, reserve stock, wait for approval, ship order; each step raises a failure question order placedCharge cardHTTP callcrash after this?who continues?Reserve stockHTTP calltimeout: retry?how long to wait?Wait approvalhours / daysblock a threadfor two days?Ship orderHTTP call10,000 ordersat once?
Every step of a real process can fail in its own way

You could write this as one big function. But then you have to answer hard questions yourself:

  • The process crashes after "Charge card". Who remembers to continue with "Reserve inventory"?
  • "Reserve inventory" times out. How many times do you retry, and how long do you wait between tries?
  • "Wait for approval" takes two days. Do you keep a thread blocked for two days?
  • Ten thousand orders arrive at once. How do you spread the work across machines?

A workflow engine answers these questions once, for every process. You give it a definition (the graph), and for each incoming request it starts a run. The engine remembers where every run is, decides what happens next, hands work to workers, retries failures, and parks long waits without holding any resources.

A workflow engine takes definitions and start requests, remembers every run, and dispatches tasks to many workers definitionYAMLstart runinput JSONworkflow engineremembersevery rundispatchestasksdecidesresultsretries · timers · waits · recoveryworkerworkerworkerexecute ⇄ report · many machines
The engine remembers, decides and dispatches; workers execute

Well-known workflow engines

You are building in a well-explored space. Knowing the big names helps you place Composer:

Engine Model
Netflix Conductor Workflows are JSON graphs of tasks. A central server keeps state; workers poll for tasks.
Temporal / Cadence Workflows are code. The engine records an event history and replays your code to rebuild state.
AWS Step Functions Workflows are JSON state machines (Amazon States Language), run as a managed service.
Apache Airflow Scheduled batch DAGs for data pipelines.

Composer sits closest to Conductor and Step Functions: a workflow is a declarative graph (YAML), and the engine is a state machine over that graph. There is no code replay. This keeps the engine small and easy to reason about.

What you will build

Composer architecture: clients call the coordinator, which runs the state machine against PostgreSQL in one transaction per event; the outbox enqueuer publishes tasks to Redis Streams; workers lease, execute and report over gRPC Clientsweb UI · APIcoordinator × N (stateless)REST :8080gRPC :50051workflow state machineevents → actionspersister: actions → SQL + outboxtimerslease recoveryoutboxenqueuerPostgreSQLruns · tasks · events · outboxworkers × MHTTP · CODERedis Streamstasks.<queue>.<shard>language serversPython · JSREST + JWTtransitionone tx per event(run row lock)due timers,expired leasesNOTIFYXADDXREADGROUPlease ·reportExecute
Composer at a glance: a stateless coordinator, PostgreSQL as the source of truth, Redis Streams for dispatch

By the end of this chapter you will have:

  • A workflow model: nodes (HTTP, CODE, SWITCH, FORK, JOIN, WAIT, WAIT_FOR_EVENT), edges with CEL conditions, retries and queues.
  • A pure state machine that turns events (NodeCompleted, TimerFired, …) into actions (CreateTask, DispatchTask, …). It touches no database and no clock, so you can unit-test every rule.
  • A coordinator that wraps the machine: lock the run, decide, persist, commit — one transaction per event.
  • A transactional outbox that turns "dispatch this task" into a row in the same transaction, then publishes it to Redis Streams within milliseconds.
  • Workers that read from sharded streams, claim each task with a lease, execute it, and report back with a fencing token.
  • Recovery loops that find tasks whose worker died or whose message was lost, and dispatch them again.

Guarantees you are aiming for

Write these down before you write any code. Every design decision in this chapter serves one of them.

Guarantee How you will get it
A run never loses a step Every decision and its effects commit in one PostgreSQL transaction.
A step never runs twice at the same time A worker must hold the task's lease before it executes.
A late or duplicate result changes nothing Results carry (task_id, attempt); anything that does not match is ignored.
A crashed worker does not stall a run Leases expire; recovery re-dispatches the task.
Long waits cost nothing WAIT and WAIT_FOR_EVENT are rows in the database, not blocked threads.

Note what is not on the list: exactly-once side effects. If a worker calls a payment API and then crashes before reporting, the step runs again. Composer gives you at-least-once execution, and helps you make steps idempotent (HTTP steps send the task ID as an Idempotency-Key header).

How this chapter is organized

You build the engine from the inside out. First the pieces with no dependencies, then the pieces that wrap them:

# Page What you build Package
1 Concepts vocabulary: run, node, task, event, action, lease, outbox —
2 Project structure Go module, packages, dependencies —
3 Workflow model nodes, edges, the indexed DAG pkg/model
4 State machine events → actions, routing, joins, waits pkg/statemachine
5 Storage schema, partitions, the repository migrations, pkg/coord
6 Coordinator lock → decide → persist → commit pkg/coord
7 Outbox & streams reliable dispatch to Redis Streams pkg/outbox, pkg/stream
8 Workers & leases read, lease, execute, report pkg/lease, worker/
9 Failure recovery expired leases, lost messages, timers pkg/coord
10 Running Composer docker compose, first workflow, tests —

Each page ends with something you can run or test. Let's start with the vocabulary.