Core Concepts — Runs, Tasks, Events, Leases and the Outbox

Before you write code, learn the handful of ideas the whole engine is built on. Each one solves one specific problem. Once you see the problem, the idea feels obvious.

#Definition, run, node, task

Term	Meaning
Definition	A versioned graph of nodes and edges, written in YAML. One version per workflow is deployed.
Run	One execution of a definition version with its own input. A run is pinned to the version it started on, so deploying v4 never changes a run that started on v3.
Node	A step in the graph: "call this URL", "run this Python", "branch here".
Task	What a node becomes inside a run. When a run reaches a node, it creates at most one task for that node. The task has a status, an attempt number, an input and an output.

Rule to remember: nodes are static, tasks are per run. The database enforces "one task per node per run" with a unique index on (run_id, node_id).

#Three kinds of nodes

Not every node needs a worker:

Keeping control and wait nodes inside the coordinator means a SWITCH costs no network hop, and a two-day WAIT costs only a database row.

#Events and actions

Everything that happens to a run is an event:

Event	What happened
RunStarted	a client started the run
NodeCompleted	a worker finished a task attempt
NodeFailed	a worker's task attempt failed
TimerFired	a WAIT's timer (or a WAIT_FOR_EVENT's timeout) came due
SignalReceived	an external event arrived for the run
RunRetried	a user restarts a finished run from some node
RunCancelled	a user cancels the run

The engine reacts to one event at a time and produces a list of actions — the changes that must be saved:

This split — what happened in, what to change out — is the heart of the design.

#Functional core, imperative shell

The component that turns events into actions is a pure function in spirit. It does not open database connections, read the clock, or send messages. It gets everything it needs passed in.

Why bother?

• Testability. You can test "a join waits for both branches" with a few lines and no database.
• Atomicity. The shell writes all actions in one transaction. Either the whole decision is saved, or none of it is.
• Clarity. All workflow rules live in one small package. All SQL lives in another.

#One lock per run

Two workers can finish two branches of the same run at the same instant. If both events are processed in parallel, each might see the other branch as still running, and neither would start the join node. The run would hang.

The fix is simple: process events for one run one at a time. The shell takes a row lock on the run (SELECT … FOR UPDATE) at the start of every transaction. Different runs proceed in parallel; events within a run are serialized.

Because the lock lives in PostgreSQL, the coordinator itself holds no state. You can run as many coordinator replicas as you like.

#Leases: one executor at a time

A task message can be delivered to a worker more than once (a worker crashed, a message was reclaimed). You must not let two workers execute the same task at the same time.

Before executing, a worker asks the coordinator for a lease: "I, worker W, own task T for the next 90 seconds." The lease is a pair of columns on the task row (lease_owner, lease_expires_at). Only one worker can hold it. While working, the worker renews it every 30 seconds. If the worker dies, the lease expires and another worker may take over.

#Fencing with the attempt number

Leases stop concurrent execution. They do not stop a late report. Picture this: worker A is slow, its lease expires, worker B re-runs the task and reports success, the task retries for some other reason, and then A's ancient report arrives.

Each task carries an attempt number. Every dispatch, lease and report includes it. The engine applies a result only if (task_id, attempt) matches the task's current state. Anything else is ignored. The attempt number acts as a fencing token.

#The transactional outbox

When the state machine decides "dispatch task T", two things must happen:

1. Save the task row in PostgreSQL.
2. Publish a message to Redis so a worker picks it up.

If you do them separately, a crash in between leaves you with a task nobody will run (saved, not published) or a message for a task that does not exist (published, rolled back). There is no transaction that spans PostgreSQL and Redis.

The outbox pattern fixes this. Instead of publishing directly, you insert an outbox row in the same transaction as the task row. A separate loop reads committed outbox rows and publishes them to Redis, deleting each row after a successful publish.

The worst case is now a duplicate publish (the enqueuer crashes after publishing but before deleting the row). Duplicates are harmless because of the lease.

#At-least-once execution

Put the pieces together and you get the engine's execution guarantee:

• A step always eventually runs (outbox + recovery).
• A step never runs twice concurrently (lease).
• A step may run again after its previous attempt's lease was lost (crash).

That is at-least-once. Side-effecting steps should be idempotent. With these ideas in hand, set up the project.