From a73362367d5c4a5c24001ae7a2f3873eb57a1a13 Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 10:28:28 +0300 Subject: [PATCH] docs: add repo layout, schema, implementation plan and verification Extends the design with the sections the sibling services carry: package tree with a strict downward dependency rule, the Postgres schema, a wave-based implementation plan, an end-to-end verification checklist and the open risks. Work is dispatched per the parallel-implementer convention: foundation commit carries every external dependency and the core package, then siblings importing it write disjoint directories in parallel. Phase 0 runs the prose-diff spike first because it is the assumption with no fallback. Projects and comments are deliberately absent from the schema: a project is a saved filter that is speculative before there are spaces to filter, and comment anchoring should be settled against a built review UI rather than committed to a schema first. --- docs/DESIGN.md | 291 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 242 insertions(+), 49 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 74727facaed74ce65fd343130e8561a06e0a3fe8..f55da31f0b31ec02c1fb75aabd6ae7d159e0129d 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -724,58 +724,251 @@ barely matters: Phase 2 serves `/query` at our own origin either way, and `hut` targets that origin directly. Federation is a config line to set if the gateway happens to exist. -## Phases +## Repo layout -**Phase 0 — de-risk gate.** Two spikes, both must pass before anything else is -built: +Mirrors the siblings: a pure `core/` with no external dependencies, one package +per subsystem, one `service/` layer that everything user-facing calls. -1. go-git write path in-process: create branch → commit blob → build merged tree - → two-parent commit, on a real bare repo, no shell-out. -2. Prose word-diff rendered against two real revisions of an actual existing spec. - **If the diff does not read well, the design needs rethinking before any - further code.** This is the compare.sr.ht/dolt.sr.ht spike discipline applied - to the riskiest assumption here. +``` +core/ pure domain. Space/doc/rev/ID validation, .spec.yml policy, + proposal state machine, sentinel errors. No external deps. +gitx/ bare-repo lifecycle, tree walk, blob read, proposal branches, + the tree-splice merge, refs-rule enforcement. go-git only. +db/ Postgres: proposals, ID registry, agent token, index stamps. +doc/ absorbed warren vault/ + render/: frontmatter, Archive, + goldmark rendering, wikilink resolution. Fed by gitx trees. +search/ absorbed warren index/ + search/: one global bleve index, + query-time project filtering. +prosediff/ word-level prose diff over rendered block structure. Net-new, + riskiest, and the Phase 0 gate. +authn/ unified-login cookie -> identity, agent token validation, + provenance trailer construction. +service/ orchestration: read, propose, merge, reconcile, digest. The + single layer REST, MCP and GraphQL all call. +api/ REST handlers (write plane + JSON reads). +mcpsrv/ absorbed warren mcpsrv/, wired to service/. +graph/ gqlgen read schema + resolvers (Phase 2). +web/ chi router, SourceHut chrome, templates, review UI, static. +hooks/ update + post-receive shims and the daemon-side RPC endpoints. +cmd/specsrht/ daemon entry point, startup validation. +cmd/specsrht-migrate/ brant wrapper, copied from doltsrht-migrate. +contrib/ nginx block, systemd unit, forced-command wrapper. +migrations/ brant .sql migrations. scss/ stylesheet entry. +``` + +Dependency direction is strictly downward: `core/` depends on nothing, +`service/` depends on everything below it, and `api`/`mcpsrv`/`graph`/`web` +depend only on `service/`. Nothing above `service/` may touch `gitx/` or `db/` +directly — that is what keeps the three agent-facing surfaces behaviourally +identical. + +## Postgres schema + +Git holds the documents; Postgres holds only what git cannot answer cheaply. +Every table here is reconstructable from refs by the reconciler, which is what +makes the non-transactional writes tolerable. + +```sql +-- Spaces exist as repos; this table is for listing and index bookkeeping. +CREATE TABLE space ( + id SERIAL PRIMARY KEY, + owner TEXT NOT NULL, -- "bigbes", no ~ prefix + name TEXT NOT NULL, + created TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (owner, name) +); + +-- Global, not per-project: a later import cannot collide. +CREATE TABLE document_id ( + doc_id TEXT PRIMARY KEY, -- "SPEC-0007" + space_id INTEGER NOT NULL REFERENCES space(id) ON DELETE CASCADE, + path TEXT NOT NULL, -- current path on the approved branch + updated_rev TEXT NOT NULL +); + +CREATE TABLE proposal ( + id SERIAL PRIMARY KEY, + space_id INTEGER NOT NULL REFERENCES space(id) ON DELETE CASCADE, + title TEXT NOT NULL, + rationale TEXT, + base_rev TEXT NOT NULL, -- the If-Match value; does not move + branch TEXT NOT NULL, -- "proposals/42" + state TEXT NOT NULL, -- open | merged | rejected + approval TEXT, -- human | policy, set on merge + merged_rev TEXT, + agent TEXT NOT NULL, -- "claude-code/spec-writer" + agent_session TEXT NOT NULL, + created TIMESTAMPTZ NOT NULL DEFAULT now(), + resolved TIMESTAMPTZ +); +CREATE INDEX ON proposal (state, created DESC); + +CREATE TABLE agent_token ( + id SERIAL PRIMARY KEY, + name TEXT NOT NULL, + token_hash BYTEA NOT NULL UNIQUE, + created TIMESTAMPTZ NOT NULL DEFAULT now(), + revoked TIMESTAMPTZ +); + +-- Index staleness: compared against the space's approved head. +CREATE TABLE index_stamp ( + space_id INTEGER PRIMARY KEY REFERENCES space(id) ON DELETE CASCADE, + rev TEXT NOT NULL, + indexed_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- "What landed since you last looked", for the policy-merged digest. +CREATE TABLE digest_mark ( + owner TEXT PRIMARY KEY, + seen_at TIMESTAMPTZ NOT NULL +); +``` + +Projects are deliberately absent: a project is a saved filter, so it is a name +plus a space-id list, and adding it before there are several spaces to filter +would be speculative. It arrives with the meta-project in Phase 2. + +`comment` is likewise absent — inline comments are post-v1, and the anchoring +model (`doc_id`, heading path, block index, block hash) should be settled by +building the review UI before it is committed to a schema. + +## Implementation plan + +Milestones are feature-shaped; within each, work is dispatched in **waves** +following the parallel-implementer convention: the root package is implemented +and committed first, then siblings that import it and write disjoint directories +go out in parallel. Every external dependency is added in the foundation commit +(with a throwaway smoke-import build to populate `go.sum`) so `go.mod` stays out +of every parallel wave's file set; `go mod tidy` runs once, at the very end. Each +parallel implementer builds and tests **only its own package** — never `./...`, +since siblings may not compile yet. **Ordering constraint.** Because the confirmed scope is a *fresh silo* with no read-only mounts, the store is **empty until somebody fills it** — a read plane -shipped first would have nothing to render. The human push path therefore moves -up into Phase 1: it is how content first exists, and it needs almost no service -code. - -**Phase 1 — core + storage + your push path.** Pure domain (space/doc/rev/ID -validation, frontmatter parse + schema validation), bare-repo lifecycle, -the git-object read layer (tree walk → `vault.FromPages`), Postgres schema + -migrations, the reconciler, and the **SSH push path**: an `update` hook that -validates and enforces the refs rule, plus `post-receive` that notifies. Both -hooks RPC into the daemon. No indexing yet — that arrives with the read plane in -Phase 2. End state: you can `git push` a space, bad pushes are rejected, and the -service can read what landed. - -**Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified -login, `?rev=` pinning, one global bleve index with query-time project filtering, -render cache, search, and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and -`api-origin` in config, federating into `api.sr.ht`. End state: agents can read -and search everything you have pushed — useful on its own, before any review -machinery exists. - -**Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model, -the agent token + provenance trailers, REST + MCP write tools (every write -response carrying the proposal URL). MCP tools -and GraphQL resolvers share one service layer — no parallel implementations. - -**Phase 4 — review plane.** Proposal pages at stable URLs (returned by every -write, so agents can hand you a link), inbox, prose diff, approve / reject, status -lifecycle, and the **digest of policy-merged content** (the firehose half is -unreviewed by design, so it must at least be *visible* or it rots silently). - -**Phase 5 — later.** Inline comments (anchoring above), webhooks and -notifications (`core-go/webhooks`, GraphQL-native — the Phase 2 schema is the -foundation), GraphQL **mutations** once the proposal state machine has stopped -moving, vector search, per-space token scoping and read-only mounts if the -boundary ever moves. - -**Dropped outright:** the web editor (you edit via clone and push), and with it -the concurrent-web-edit-vs-push conflict problem it would have created. +shipped first would have nothing to render. The human push path therefore lands +in Phase 1: it is how content first exists, and it needs little service code. + +### Phase 0 — de-risk gate (throwaway code, no packages) + +Two spikes. Both must pass before Phase 1 starts. + +1. **go-git write path.** On a real bare repo: create a branch, commit a blob, + splice a tree, write a two-parent commit, and read it back — all in-process. + Confirms the merge model is expressible with plumbing alone. +2. **Prose word-diff.** Render a word-level diff between two real revisions of an + existing spec of yours. **If it does not read well, stop and rethink** — the + browser-review decision and most of Phase 4 rest on it. This is the single + riskiest assumption in the document. + +Spike 2 is deliberately the one with no fallback plan, so it runs first. + +### Phase 1 — foundation, storage, and your push path + +**Foundation commit (serial).** `go.mod` with every external dependency +(go-git, bleve, goldmark, chi, lib/pq, yaml.v3, the MCP SDK, brant, plus the +`core-go` fork `replace`), then `core/` complete with table-driven tests: name +and path validation, ID grammar, `.spec.yml` parse and policy evaluation, +frontmatter schema validation, the proposal state machine. + +**Wave A (parallel — `gitx/`, `db/`, `authn/`).** Disjoint directories, all +importing only the committed `core/`. +- `gitx/` — bare-repo lifecycle, tree walk, blob read, branch create, tree-splice + merge, refs-rule predicate. Tests build fixtures programmatically. +- `db/` — the schema above plus queries; `schema.sql` and the first brant + migration. +- `authn/` — cookie decrypt to identity, agent-token validation, provenance + trailer construction. Tests forge cookies with a synthesized ini (random fernet + key + ed25519 seed), as dolt.sr.ht's `authn/` does. + +**Wave B (serial).** `service/` read + reconcile paths, `hooks/` (the `update` +and `post-receive` shims and their daemon RPC endpoints), `cmd/specsrht` skeleton, +`cmd/specsrht-migrate`. + +*End state:* `git push` a space; malformed pushes are rejected with a useful +message; the daemon can read what landed; the reconciler repairs a killed daemon. + +### Phase 2 — read plane + +**Wave A (parallel — `doc/`, `search/`).** `doc/` is warren's `vault` + `render` +with `Scan` replaced by a git-tree walk feeding the existing `FromPages` seam. +`search/` is warren's `index` + `search` against one global bleve index, with +query-time project filtering. Both are ports with focused tests, not new design. + +**Wave B (parallel — `web/`, `graph/`, `mcpsrv/` read tools).** All depend only +on `service/`. `web/` brings the chrome, cookie login and SCSS pipeline copied +from compare.sr.ht. + +*End state:* agents read and search everything you have pushed, over MCP, REST +and `/query`. **Useful on its own, before any review machinery exists** — which +is the point of this ordering. + +### Phase 3 — write plane + +`service/` write paths (propose, `If-Match` resolution, merge, auto-merge policy, +digest bookkeeping), then `api/` and `mcpsrv/` write tools in parallel. Every +write response carries the proposal URL. + +*End state:* an agent proposes and hands you a link; approving from an API call +merges; a stale base returns 409. + +### Phase 4 — review plane + +`prosediff/` productionised from the Phase 0 spike, then the review UI: proposal +pages at stable URLs, approve/reject, the inbox, and the policy-merged digest. + +*End state:* the full loop — bot produces, you curate in a browser, bots consume. + +### Phase 5 — later + +Inline comments (settle anchoring against the built UI first), webhooks and +notifications, GraphQL mutations once the proposal types stop moving, vector +search, per-space token scoping, read-only mounts. + +**Dropped outright:** the web editor, and with it the concurrent-web-edit-versus- +push conflict it would have created. + +## Verification (post-deploy, end-to-end) + +Run against the real instance, in order. Each step fails loudly rather than +degrading, which is the point of listing them. + +1. `GET https://spec.srht.bigb.es/healthz` → `ok`; the service appears in the nav + of git.sr.ht and meta.sr.ht (after restarting them). +2. `git push` a space with a valid document → accepted; it renders in the browser + and appears in search within one merge cycle. +3. `git push` a document with a duplicate `id:` → **rejected** by the `update` + hook with a message naming the collision; retry with + `--push-option=skip-validation` → accepted. +4. `git push --force` to the approved branch → rejected. +5. An agent token proposing to `proposals/*` → accepted; the same token + attempting the approved branch → rejected. +6. Propose via MCP → response carries a URL; opening it shows the prose diff; + approve → merges, and the document's approved text changes. +7. Propose against a stale base → 409 with the current head. +8. Propose into an `auto_merge` path → lands immediately, and shows in the digest + marked `approval: policy`, not `human`. +9. `?rev=` of a superseded revision → renders that revision, not the head. +10. `kill -9` the daemon mid-merge, restart → the reconciler repairs the proposal + row and the index stamp; no manual intervention. +11. Anonymous request → no content leaks; logged-in as bigbes → full access. + +## Open risks + +- **Prose diff quality (highest).** Everything downstream of the browser-review + decision assumes it reads well. No fallback is designed; Phase 0 exists to find + out early rather than late. +- **go-git and native `receive-pack` on the same refs.** Mitigated by a per-space + mutex and CAS retry, but the interoperation of the two locking implementations + is assumed rather than proven. Worth a deliberate concurrent-push test. +- **Empty-store bootstrapping.** With no mounts, the service is worthless until + content exists. Phase 1's push path is the mitigation; if it slips, the whole + thing feels stillborn. +- **warren absorption drift.** Once `vault`/`render`/`search` are copied in, they + fork from upstream warren. Accepted deliberately — but it means bug fixes do + not flow back, and that should be a conscious choice each time. +- **Comment anchoring, deferred not solved.** The tuple is specified; whether it + survives real reflowed prose is untested, and Phase 5 will discover it. ## Open items @@ -828,8 +1021,8 @@ into the sections above: ### Still open -- **Prose diff quality** — not a question to answer in prose but the Phase 0 gate - to run. Everything else assumes it comes out well. +- **Prose diff quality** — tracked under "Open risks" above; not a question to + answer in prose but the Phase 0 gate to run. - **Mixed ru/en per-document language routing** (above): still needs a decision once there is a real corpus to test against. - **Attachments and binaries** (above): unresolved, and cheap to defer until a