Project Structure — Go Package Layout for a Workflow Engine

Now set up the project. This page creates the Go module, lays out the packages, and explains which package owns which job. Keep this page open as a map while you build the rest of the chapter.

Step 1: Initialize the Go module

mkdir composer && cd composer
go mod init github.com/mohitkumar/composer

Step 2: Create the directory layout

mkdir -p cmd/{coordinator,worker,language-server-py,language-server-js}
mkdir -p pkg/{model,statemachine,coord,template,lease,outbox,redisq,db,auth,util}
mkdir -p pkg/stream/redis pkg/api/{http,grpc,proto}
mkdir -p worker/executors migrations infra tests docs/diagrams benchmark web

The finished tree looks like this:

composer/
├── cmd/
│   ├── coordinator/          coordinator binary (flags, bootstrap)
│   ├── worker/               worker binary (queues, streams, concurrency)
│   ├── language-server-py/   Python sandbox server (RestrictedPython + audit hooks)
│   └── language-server-js/   JavaScript sandbox server (isolated-vm)
├── pkg/
│   ├── model/                definitions, DAG index, run context, statuses
│   ├── statemachine/         the workflow state machine: events, state, actions (pure)
│   ├── coord/                coordinator shell: engine, persister, repository, definitions, runs
│   ├── template/             CEL template resolver
│   ├── lease/                task leases
│   ├── outbox/               outbox enqueuer (LISTEN/NOTIFY + polling)
│   ├── stream/               stream abstraction, queue/shard naming
│   │   └── redis/            Redis Streams implementation
│   ├── redisq/               thin Redis client wrapper
│   ├── db/                   pgx pool + DB / Querier interfaces
│   ├── api/
│   │   ├── http/             REST API (chi)
│   │   ├── grpc/             worker-facing gRPC service
│   │   └── proto/            .proto files + generated code
│   ├── auth/                 JWT login
│   └── util/                 OpenTelemetry metrics and traces
├── worker/                   worker runtime: readers, executor pool, executors, coordinator client
│   └── executors/            HTTP executor, Executor interface
├── migrations/               database schema (partitions, triggers, seed data)
├── tests/                    end-to-end tests against real PostgreSQL and Redis
├── infra/                    Docker Compose, Helm chart, bootstrap scripts
├── web/                      Next.js web UI
└── benchmark/                k6 load tests

Step 3: Understand the layers

The packages form layers. Each layer only imports layers below it:

Each layer imports only the layers below it

Two rules keep this clean:

pkg/statemachine never imports pkg/coord, pgx or Redis. It only knows about pkg/model and two tiny interfaces (Store, TemplateResolver) that the shell implements.
The worker never touches PostgreSQL. It talks to the coordinator over gRPC and to Redis for messages. Only the coordinator owns the database.

Step 4: Add the dependencies

go get github.com/jackc/pgx/v5            # PostgreSQL driver and pool
go get github.com/redis/go-redis/v9       # Redis Streams
go get github.com/google/cel-go           # CEL expressions in templates and conditions
go get github.com/google/uuid             # UUIDv7 run and task IDs
go get google.golang.org/grpc google.golang.org/protobuf
go get github.com/go-chi/chi/v5           # HTTP router
go get github.com/rs/zerolog              # structured logging
go get github.com/spf13/cobra github.com/spf13/viper   # flags + env config
go get github.com/Yiling-J/theine-go      # definition cache
go get go.opentelemetry.io/otel           # metrics and traces
go get github.com/testcontainers/testcontainers-go     # real Postgres/Redis in tests

Dependency	Why this one
pgx/v5	Fast native PostgreSQL driver. You need `LISTEN/NOTIFY`, which pgx exposes directly through `WaitForNotification`.
go-redis/v9	Full support for Streams: `XADD`, `XREADGROUP`, `XAUTOCLAIM`, `XPENDING`.
cel-go	Google's Common Expression Language. Safe (no loops, no I/O), fast, and familiar to anyone who wrote Kubernetes policies.
uuid	`uuid.NewV7()` gives IDs that sort by creation time — you will use that to partition tables by run ID.
theine-go	A W-TinyLFU cache. A few hot workflows get most of the traffic; this policy keeps them cached.
testcontainers-go	End-to-end tests start real PostgreSQL and Redis containers.

Step 5: The processes you will run

Composer is not one binary. It is a small set of processes, each scaled independently:

Five kinds of process, each scaled on its own

Process	Scales by
Coordinator	Adding replicas. It is stateless; the run row lock serializes work per run.
Worker	Adding replicas or raising `WORKER_CONCURRENCY`.
Language servers	Independently of workers. User code runs here, in sandboxes.
PostgreSQL	Partitioning keeps hot indexes small; retention drops whole months.
Redis	Shards per queue (`--stream_shards`), so one queue spreads across Redis Cluster nodes.

Step 6: The build commands

The Makefile wraps everything you need:

make install-deps   # Go modules, npm and uv dependencies
make proto          # generate gRPC code (Go, Python)
make build          # bin/coordinator, bin/worker, language servers, web UI
make test           # unit + container-backed tests (needs Docker)
make docker-up      # the whole stack in Docker Compose

You now have a skeleton. The next page fills in the bottom layer: the workflow model.

Project Structure — Go Package Layout for a Workflow Engine

#Step 1: Initialize the Go module

#Step 2: Create the directory layout

#Step 3: Understand the layers

#Step 4: Add the dependencies

#Step 5: The processes you will run