crabster

Documentation / The project

Roadmap

{% raw %} This document breaks the construction of Crabster into sequential phases (with a few possible parallelizations noted). Each phase specifies: objective, deliverables, tasks, exit criteria (Definition of Done), and dependencies.

Convention: phases 0 through 10 make up the V1 (API-only) scope, detailed here. Phase 11 onward belongs to V2 and is detailed in the V2 scope. Phase C (documentation) is ongoing and spans both versions.

Overview🔗

PhaseTitleScopeDepends on
0Project foundationsV1—
1CLI skeleton + minimal projectV10
2Template engine & orchestrationV11
3CDL language + CRUD generationV12
4Authentication & securityV13
5Multi-database supportV13
6API documentation, validation, errorsV13
7Generated tests & qualityV13, 4
8Docker, deployment, CI/CDV11, 3
9ObservabilityV11
10Blueprints & extensibilityV1 (mechanism)2
CUser documentation & communityOngoing1 (starts as soon as there's something to document)
11-18(see V2 scope)V2Full V1

Phases 4, 5, and 6 can be run in parallel by different contributors once Phase 3 stabilizes (they all depend on the IR and generation engine, but not on each other).

Phase 0 — Project foundations🔗

Objective: lay the organizational and technical groundwork before writing any generation code.

Tasks

Deliverables

Definition of Done

Phase 1 — CLI skeleton + generated minimal project🔗

Objective: crabster new produces a minimal Axum project that compiles, responds on a health check, and runs.

Tasks

Deliverables

Definition of Done

Baseline measurement (2026-09-04) — macOS, Apple Silicon, Cargo registry already populated:

StepTime
Generating the 9 files~1 ms
Initial cargo build of the generated project (cold dependencies)~11 s
Total, generation → server answering~12 s

Well under the 5-minute goal, but this figure only covers the core template: it will grow as SeaORM (Phase 3) and test dependencies (Phase 7) are added, and a machine with no Cargo cache also pays for the registry download. Re-measure at every phase that adds dependencies to the generated project.

Phase 2 — Template engine & orchestration🔗

Objective: replace Phase 1's hardcoded generation with the modular Tera engine described in the architecture doc — the foundation for everything that follows.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-05) — DoD verified. An external blueprint (one directory, one module.toml, zero lines of Rust) replaces the core module and generates a project that compiles, through crabster new --blueprint.

Two deliberate departures from the original wording:

Phase 3 — CDL language + CRUD generation🔗

Objective: the most critical phase — domain modeling → fully generated CRUD API. This is the core of Crabster's value.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete. crabster import-cdl examples/shop.cdl produces 37 files for 4 records; the project compiles without a warning, passes its own tests, and its API answers. An end-to-end test in CI generates it, builds it, runs its tests and then exercises it: creation, filtering, ordering, a pattern rejecting a value, a duplicate on a unique column answering 409, a missing reference answering 409, and deleting a record something still points at answering 409.

References are generated too: the column, the constraint, the SeaORM relations on both sides, and migrations ordered so the referenced table is created first.

Ordering is generated for every list, with no attribute to write: ?sort=<field>&order=asc|desc, over a per-record whitelist, with a stable secondary sort on id and a 400 naming the accepted columns when the request names none of them.

crabster record <Name> --field "..." adds a record to an existing project. The project keeps the model it came from in .crabster/, which lets the old model be rendered again and compared file by file: identical means the file is generated and may move; different means it was edited by hand, and the command stops and names it, having written nothing, unless --force says otherwise. Migrations already applied keep their numbers, and a recorded model that no longer describes the project is refused rather than followed. A project from crabster new, with no model at all, gains its first record just as well.

Three defects that contradicted the phase's own invariant — "a model that parses is a model that can be generated" — were closed along the way: five record names the migration module already occupied, a field named active that shadowed the model in the update handler, and an attribute written on a line of its own, which silently attached to the field above it.

Phase 4 — Authentication & security🔗

Objective: stateless JWT, password hashing, role-based authorization.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete. service { auth jwt } generates src/auth/: the accounts table and its migration, POST /auth/register, /auth/login, /auth/refresh, GET /auth/me and GET /auth/users. Twenty-seven tests in the generated project, among them the ones that hold the DoD — registering, logging in, reading a protected route, a wrong password, a refresh token spent as an access token, a missing role — and a heavy generator test that compiles a project with authentication, runs its tests, and checks that those cases actually ran, because a suite that stopped running them would still pass.

Three details that separate example code from code anyone can ship. A login that finds no account hashes against a dummy anyway: otherwise an unknown address answers measurably faster than a wrong password, which is how a list of real addresses is harvested from a service that never returns one. The token's kind is checked on the way in: a refresh token lasts a month and is signed with the same key, so a route that accepted either would be handing out month-long access. And the key is refused twice — the placeholder from .env.example outside the dev profile, and anything shorter than 32 bytes everywhere — both at startup, so a service that cannot sign anything never gets as far as being called healthy.

Decision — an extractor, not a macro. The roadmap promised a "macro/attribute to restrict a route". It is Authenticated in a handler's arguments and claims.require_role("ADMIN"): a method reads the same as the code around it, appears in a stack trace, and obliges nobody to know what it expands into. Naming the extractor is what protects the route, so no route can be listed as protected and not be. The generated CRUD endpoints are not protected by default: protecting them all would change the meaning of every endpoint a model declares, and the generated README shows the line to write to protect one.

Decision — no rate limiting is generated. The HTTP hardening that is delivered lives in core: a request timeout answered as 408, a concurrency limit, a body limit, nosniff, and CORS closed by default with no wildcard available. The rate limiter is not: inside the process it counts one replica's traffic, so the configured number means something different every time the deployment is scaled, and what is worth limiting is usually per caller — an identity that layer does not have. It belongs at whatever terminates TLS in front of the service, and that is written in src/http.rs as much as here.

The two items deferred from step 1 are delivered here, against the module that justified them. Activation: [activation] in the manifest, Service.settings carried by the parser with its position, refused by the engine — the only crate that knows which modules are installed, blueprints included — and --with/--without to override. The resolved list is written into .crabster/project.toml and read back, without which crabster record on an auth-jwt project would report every one of its files as an orphan. And reservations: [[reserves]] in the manifest, checked only while the module is generated, because AuthUser is a name a project without accounts is free to take.

What did not move, and why. The names core and record occupy stay in crabster-cdl rather than rising into the manifests. What holds them honest is not a list but a test that renders a probe model, reads the result with syn, and requires every imported name to be a reserved one — a stronger tie than a manifest, which would only restate what the templates contain. What [[reserves]] adds is the part that was missing: a module the parser has never seen can declare its own.

A limit taken on knowingly. Sessions are stateless, as the objective asks, so a token cannot be revoked: it stays valid until it expires whatever happens to the account. Hence fifteen minutes for access and thirty days for refresh, with roles read from the account on every refresh. A revocation list is state, and adding one would undo the decision this phase states.

Phase 5 — Multi-database support🔗

Objective: PostgreSQL (reference), MySQL (written mysql; MariaDB is wire-compatible), SQLite, selectable at generation time.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete, with one reservation named below. All three databases are selectable and the generated code is the same for all three: SeaORM carries the dialect, sea-query renders the types, and no template branches on {{ database }} to produce SQL. The three places where it is not transparent are in view.rs and nowhere else — Timestamp becomes a DATETIME on MySQL, which has no timestamptz; Decimal is pinned to DECIMAL(19,4) everywhere; Bytes is a BLOB. The limit that remains is documented rather than fixed: on SQLite a Decimal travels through an f64, because sqlx-sqlite does not carry rust_decimal. It is written in §4 of the language and in the README of any SQLite project holding a Decimal.

docker-compose.yml is generated for PostgreSQL and MySQL, on exactly the host, port, user, password and database name .env.example points at — both files are written from one value, and a test fails if they drift apart. Nothing for SQLite: there is no server to start, and an empty Compose file would be a file to explain.

The two CI jobs that run a generated project against a real database now start that file, rather than a service defined in the workflow. What CI runs is therefore what the generated README asks a reader to run — cp .env.example .env, docker compose up -d --wait, cargo run — and the deliverable is verified on every run instead of being a file nobody executes.

Decision — the db-postgres, db-mysql and db-sqlite modules will not exist. There would be nothing in them: the connection string is one line of .env.example, the pool is SeaORM's, the driver is a Cargo.toml feature, and the column types come from the view rather than from a template. Three modules whose entire content is a variable substitution would be three places to forget a database instead of one. The neutrality comes from SeaORM; making it explicit would make it false.

Decision — no prodDatabaseType/devDatabaseType: one database per project. With two, the schema is not the same on both sides — Timestamp is a DATETIME on MySQL and a timestamptz elsewhere, and a Decimal does not round-trip on SQLite. A project developed on SQLite and deployed on PostgreSQL would pass its tests against a schema its production never has: exactly the class of fault the option claims to avoid. The need behind the option is real — testing without a server to start — and Phase 7 is what answers it, with testcontainers against the database actually targeted.

The reservation, since lifted (Phase 7). The Definition of Done asks that shop.cdl pass its integration tests on all three databases. That was at first only half true: all three were exercised in CI, but the generated project's own tests ran on SQLite whatever it targeted, because its dev-dependencies pinned sqlx-sqlite. Phase 7 removed that pin: a PostgreSQL project runs its tests against PostgreSQL, and the two CI jobs that start a real database now run cargo test inside it. The DoD is met with nothing held back.

Phase 6 — API documentation, validation, error handling🔗

Objective: an externally consumable API contract (essential since V1 is API-only and there's no frontend to "mask" a poorly documented API).

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete. The error contract is RFC 9457 — the revision of the RFC 7807 this roadmap aimed at — served as application/problem+json, for every failure and from every layer, including the ones axum would otherwise reject itself in plain text.

type is the field the status cannot replace. A 409 for a taken unique value and a 409 for a reference that does not hold are two different problems under one code, and the prose in detail is not something a client branches on. The type is a relative URI the project serves: GET /problems/unique-conflict explains that kind, GET /problems lists them all. This is where about:blank would have been simpler and wrong — it means "nothing to say beyond the status code", which is precisely the opposite. The RFC asks that a type URI which is a locator have documentation behind it; the text served is the same constants the error code carries, so the two cannot drift, and a generated test walks the list to check that every type leads somewhere.

Decision — the Definition of Done was corrected, not the code: a refused body answers 422, not 400. RFC 9110 defines 422 as a request that is well formed and semantically wrong, which is exactly the case; axum already answers 422 one layer out for a body that does not match the shape; and collapsing both into 400 would make "I could not read this" indistinguishable from "I read it and will not have it". The 400 in the DoD was Spring inheritance. The rest of it holds word for word: an invalid request returns a structured error listing the offending fields, in errors.

The OpenAPI 3.1 document is generated from the handlers themselves, served at /api-docs/openapi.json, with a console at /swagger-ui whose assets are compiled into the binary — the project builds with no network and serves with none. Every operation carries an explicit operation_id, because a generator produces a list in every record's module and a specification may not name two operations the same; a test checks it on a two-record model. Another requires every handler to carry its attribute, verified in both directions.

The linter, run. The generated project ships its own .spectral.yaml, and the CI job the-api-contract starts the project, asks it for its document, and lints it with the project's own ruleset — checking against stricter rules than the ones a reader is handed would be checking something nobody else can reproduce. At --fail-severity warn, above the "no errors" that was asked for: the document comes out with nothing at either level, and a bar set where the work already stands is a bar that never moves. The 25 warnings it started with were fixed, not switched off: every operation gained a summary and a description, and the document declares a relative server. One rule is off, info-contact, with its reason written in the file: who answers for an API is decided where it runs.

Phase 7 — Generated tests & quality🔗

Objective: every generated project ships with a runnable test suite that provides confidence without manual work.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete. Two things gave way.

The tests run against the database the project targets. That was Phase 5's reservation, and it came down to one line: the dev-dependencies pinned sqlx-sqlite, so a PostgreSQL project tested on SQLite — which is to say, tested a different project. The line is gone; the tests use the project's own driver. For PostgreSQL and MySQL, src/testing.rs starts one container for the whole test binary and creates a database per test, because these tests count rows and a shared database would make them depend on the order they happened to run in. For SQLite, an in-memory database per test: no server, no container, no Docker. And TEST_DATABASE_URL bypasses the container, which is what the two CI jobs that already start a real database use — Docker in Docker would have bought nothing.

Every record has tests now. Three of shop.cdl's four had none: Customer was ruled out by a @matches, Address and Order by a mandatory reference. The most constrained records were the ones with nothing checking them.

A mandatory reference rules out nothing any more: each record's module exposes a way to create one, and a child calls its parent's — through the parent's own endpoint, as a caller would have to. Cycles being refused by the language, the recursion cannot run away.

Nor does a @matches, in most cases: the sample value is generated from the regular expression itself and then checked against it. Three things make the result usable as a fixture — anchors are stripped before generating because the generator refuses them, sampling is done in bytes under unicode(false) so a value stays something a reader recognises rather than drifting into the astral planes, and the seed is fixed so the same model produces the same test twice. A table of well-known patterns would have covered the three everybody writes and left the rest of the language untested. What remains ruled out is named: a pattern no value was found for — a @matches("^[A-Z]{40}$") under a @length(..10) — and the records that point at it, since they cannot create the parent they need.

What each record gets. Three tests: a full create / read / update / delete lifecycle; a list that is paged and ordered, past-the-end page included, and a ?sort= the record does not accept; and what the endpoint refuses — 404 on a missing identifier for read, update and delete, and 422 on an empty body where the model requires something. They go through the whole router rather than calling handlers, which is the only way to exercise routing, extractors, layers and the error shape.

Decision — coverage is documented, not configured. cargo llvm-cov --html works on a generated project with nothing installed but the tool, and the generated README says so. A configuration file would be one more file to maintain in place of two words on a command line.

Phase 8 — Docker, deployment, CI/CD🔗

Objective: a generated project is deployable "day one".

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete, with the limit of its DoD stated below. A generated project carries three more files: a Dockerfile, a .dockerignore and its own .github/workflows/ci.yml.

The image. Four stages, two of which exist only for dependency caching: cargo rebuilds everything when any file changes, so a one-line fix would recompile the whole tree; cargo-chef reduces the manifest to a recipe that moves only when the dependencies do. The final stage is distroless/cc: the binary, its config/, and nothing else — no shell, no package manager, running as nonroot. Verified by building the image on the example and running it: 61.7 MB, /health/ready reaches the database, and the process runs as nonroot — which has to be checked with docker inspect, there being no id to run inside.

The Compose file gains the application, behind a profile. docker compose up -d --wait still starts the database alone, because while you are working on the project you want the server from cargo run and not from an image rebuilt after every edit. docker compose --profile app up --build brings up both. The application service sets APP_PROFILE=dev and APP__SERVER__HOST=0.0.0.0: the dev profile binds the loopback address, which inside a container means nothing outside it can reach the server.

The generated workflow runs the formatter, Clippy at -D warnings, the tests against the database the project actually targets, a security audit of the dependency tree, and a build of the image. No secret and no repository setting, so it passes on a repository that has just been created. The image is built and not pushed — where an image belongs is a decision about infrastructure, and pushing needs a registry and a credential the file cannot invent; the workflow says in a comment what to add.

A debt closed along the way, and it bit exactly here. Generated code was not clippy-clean, and Crabster's CI checked rustfmt and never Clippy. With the generated workflow running -D warnings, every project would have watched its own CI fail on the day it was created. Three lints existed — items_after_test_module in main.rs, because a module's contributed block ended with its tests and other items followed, and two .err().expect() in the authentication tests. Fixed, and Crabster's CI now runs Clippy on both shapes of generated project.

Decision — GitLab is documented, not generated. The roadmap said "document without necessarily generating". The generated README carries the equivalent .gitlab-ci.yml in full: you use one CI or the other, and generating both would leave a dead file in every project.

The limit of the DoD. "Pushed to a new GitHub repo, has its CI pass" is the one part nothing in this repository can execute. What is checked instead: every command the workflow runs passes on a generated project, Clippy at -D warnings included; the image builds and serves; and the file carries no unrendered {{ }} — Actions' ${{ }} and Tera's being the same two braces, a test holds that. What remains unverified is that GitHub accepts the file, and that is said rather than assumed.

Phase 9 — Observability🔗

Objective: functional parity with Spring Actuator for baseline monitoring.

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete. Every generated project answers on four paths, with nothing to configure. /health/live reaches nothing: a restart policy reads it, and restarting a healthy process because a database went down turns one outage into two. /health/ready reaches everything the service depends on, answers 503 as soon as one is missing, and names which — a readiness endpoint that only returns a status code moves the question rather than answering it. /health remains, answering liveness, under the name it had before. /metrics renders counts, latencies and in-flight requests in the format Prometheus reads.

Readiness is composed, and it is the first real use of the slots from step 1. src/health.rs belongs to core, which knows nothing about a database; the probe that pings one is contributed by record through the slots in that file — an import, a field on Dependencies, the probe itself and its tests. A project with no model keeps the endpoint and has nothing to list, and the day a module brings a dependency of its own it adds its probe without core moving. The probe is a round trip, not a look at the pool: a pool hands out a connection whether or not the server at the other end is still there, so a check that only inspects the pool reports UP right through an outage. One generated test closes the pool and requires the 503; another requires that liveness does not follow it down.

Metrics are labelled by the matched route — /api/products/{id}, never /api/products/1 — so the number of series is bounded by the size of the project rather than by its traffic. The health probes and /metrics itself are left out of the counting: a liveness check every second would otherwise be the busiest endpoint the service has.

Correlation identifier. Every request carries an x-request-id, minted if it arrived without one, written into every log line of that request, and returned on the response. Layer order is the delicate part — set the identifier before anything logs, propagate it after — so it is written with a ServiceBuilder, which reads top to bottom in the order a request goes through them, and a test fails if the three ever end up out of order.

The DoD, run rather than asserted. examples/observability/ holds a docker-compose.yml and a prometheus.yml, and the CI job scraped-by-prometheus runs them: it generates shop.cdl, serves it, brings Prometheus up, and requires the target to be up and axum_http_requests_total{endpoint="/api/products"} to come back from the query API. A generated project is configured for none of it.

Decision — OpenTelemetry export is not generated. The roadmap wanted it "documented as an option"; it is, in the generated README and in Architecture §3.10. What a service ships its traces to is decided where it is deployed, not in its code, and an OTLP exporter wired in by default would be a dependency and an address nobody asked for. The hook exists: tracing is already the subscriber everything goes through, and request_span already names each request.

Phase 10 — Blueprints & extensibility🔗

Objective: open Crabster to community customization/extension, a condition for its long-term value (see Vision, principle 7).

Tasks

Deliverables

Definition of Done

Outcome (2026-09-07) — phase complete, with one task cut down to its useful shape. The layered mechanism has worked since Phase 2; what was missing was what makes it usable by somebody else.

The page. Writing a blueprint — the four things a blueprint can do, the manifest field by field, the variables it gets, slots, protected regions, what a blueprint cannot do, and how to distribute one. The DoD says "following only the documentation", so that page is the deliverable; everything else supports it.

The example blueprint: examples/blueprints/gitlab. It does all four things in four small files — it adds a module Crabster has never seen, turned on by a ci gitlab setting its own manifest claims; it adds .gitlab-ci.yml; it writes into the README through a slot in a file it does not own; and it replaces the GitHub workflow with an empty template, which removes it. The project it produces compiles, is rustfmt-clean and passes Clippy at -D warnings, because a blueprint that works and produces a project that does not build is a blueprint of no use. A CI job generates with it and checks all four on every run.

It is not Diesel, deliberately. The roadmap offered Diesel "as a candidate". Rewriting persistence, migrations, DTOs, handlers and error mapping against another ORM, tested on three databases, is not a step but a project — and the example that validates a mechanism has to be readable in one sitting. GitLab has a property Diesel does not: it fills a hole step 7 left on purpose ("you use one CI or the other, and generating both leaves a dead file in every project"), which is exactly the shape of decision a blueprint exists to reverse for the people it is wrong for.

Decision — the manifest declares its templates version, and it is required. A blueprint writes against slots, variables and file names the built-in templates define, and those move: before 1.0 a minor version may change any of them. Without the field, a blueprint from an earlier version fails deep inside a template on a slot that is gone, with a message that says nothing about the cause. Compared on major and minor — the patch level never changes a template. Added now, while no third-party blueprint exists: making it required later would break every one written in between. The built-in modules do not declare it, shipping in the same binary as the number they would be compared against.

Decision — --blueprint takes a path, not a crate name. The roadmap wrote <crate-or-path>. Resolving a name would mean downloading and running third-party templates from crates.io at generation time: a supply-chain question deserving better than a side effect of this phase. Cloning, or cargo add and then pointing at the path, puts the same decision in the user's hands while making it visible. Written into the page rather than left implicit.

The discovery convention is the crabster-blueprint keyword on crates.io, and the page carries the table of known blueprints, with one row in it. Publishing to crates.io is not something this repository can do; the convention and the table are.

Phases 11 through 18 — V2 scope🔗

Post-V1 work is detailed in the V2 scope: generated code lifecycle (upgrade and incremental merging), OAuth2/OIDC authentication, microservices, extended persistence, reverse engineering, frontend, Kubernetes, and blueprint ecosystem maturity.

Two things to keep in mind during V1, because they determine whether V2 is feasible:

Phase C — User documentation & community (ongoing, starts at Phase 1)🔗

Objective: Crabster only has value if people use it; user documentation is not an end-of-project artifact.

Tasks

Definition of Done

Cross-cutting risks to watch🔗