From 12e043e26812d9e5bd37d4b93684830f714ca71c Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 10:19:53 +0300 Subject: [PATCH] docs: settle the seven questions raised by review - Read contract keeps pinned ?rev= and the approved/draft split. The review UI needs blob->render at arbitrary revs regardless, so pinning is nearly free and makes X-Agent-Base auditable. - Materialized checkout dropped. One read path over git objects for approved head, pinned revs and proposal branches alike, which removes atomic swap, rev stamps, a cache directory and two crash-repair rows. Cost is bounded: vault.Scan touches the filesystem twice and vault.FromPages already accepts pre-loaded pages. - Volume is tens of documents a day, so warren's batch index rebuild is absorbed unchanged and incremental indexing is explicitly not built. - One agent token plus mandatory provenance. The refs rule is the boundary that bounds damage; per-space scoping defers to a column and a filter. - Human pushes are validated too, with a skip-validation push option. The risk is a typo corrupting the global ID registry, not malice. - External corpora left unspecified; global IDs are the only forward compatibility needed. - Name confirmed as spec.sr.ht. --- docs/DESIGN.md | 226 +++++++++++++++++++++++++++---------------------- 1 file changed, 125 insertions(+), 101 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 1d653a1ff2312495f33761f061679e8e4c825f62..74727facaed74ce65fd343130e8561a06e0a3fe8 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -48,6 +48,12 @@ modification**. Runs at `https://spec.srht.bigb.es`. | Cadence | **Bimodal.** Specs/RFCs get careful review; research notes and reports flow through with light or automatic approval. `.spec.yml` policy carries the split. | | Human edit path | **`git clone`, edit locally, push.** The web editor is not v1 and may never be. This makes a real git remote a v1 requirement — see "Two write paths" below. | | Review surface | **Browser, reached by a link the agent hands you.** Confirms the prose differ as a v1 build and keeps it as the Phase 0 gate. The inbox is the backstop for work no link reached, not the primary entry point. | +| Read contract | **Pinned `?rev=` plus an approved/draft split.** Nearly free, since the review UI needs blob→render at arbitrary revs anyway, and it makes `X-Agent-Base` provenance auditable rather than decorative. | +| Storage tiers | **One.** Git objects are read directly; the materialized checkout is dropped, leaving a bleve index and a blob-sha-keyed render cache as the only caches. | +| Volume | **Tens of documents a day or fewer.** warren's batch index rebuild is absorbed unchanged; incremental indexing is explicitly not built. | +| Agent tokens | **One token plus mandatory provenance.** The refs rule (agents write only `proposals/*`) is the boundary that matters; per-space scoping deferred. | +| Push validation | **Your pushes are validated too**, with a `--push-option=skip-validation` escape hatch. | +| Name | **`spec.sr.ht`** at `spec.srht.bigb.es`; module `sourcecraft.dev/bigbes/sr-ht-spec`, binary `specsrht`. | ## Architecture summary @@ -59,11 +65,10 @@ One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht` │ /~user/ you ──git push (ssh)───────────────────────┘ ▲ │ │ update + post-receive hooks - ├──► materialized checkouts (read/index cache) - │ /~user// - ├──► one global bleve index + ├──► one global bleve index (cache) + ├──► render cache, keyed by blob sha (cache) └──► Postgres (proposals, comments, agent - tokens + scopes, ID registry) + tokens, ID registry) you ──browser──────────► read + review UI (no editing) ``` @@ -79,25 +84,28 @@ small. It is not a complete export of *everything*: review decisions, proposal rationale and comments live in Postgres, so a clone gives you the documents and who wrote them, not the review record. -### Two-tier storage, and why - -- **Bare repo** — the push target and the source of truth. One repo per *space*. -- **Materialized checkout** of the approved branch — a plain directory of - markdown, rebuilt on every merge. This exists so warren's indexer and renderer - can work on files (which is what they already do), and so reads never pay - object-database costs. It is a **cache**: deletable, rebuildable from the bare - repo at any time. Rebuild **atomically** — build into a temp dir and rename - into place — or readers observe a torn tree mid-rebuild. Stamp each checkout - with the rev it was built from, so staleness is detectable rather than assumed. - -**Caveat this tier does not cover:** `?rev=` pinned reads have no checkout -to read from, since only the approved head is materialized. They need a second -read path — blob → renderer, straight from the object database, with no vault -scan. warren's `vault`/`render` split is file-oriented, so this is a real if -modest rework rather than pure absorption. It is also an argument for questioning -the checkout tier altogether: reading blobs at the approved ref plus a render -cache would make pinned and default reads *the same code path* and delete this -cache subsystem entirely. Open — see "Open items". +### One storage tier: git objects, no checkout + +**Every read resolves a git tree and reads blobs.** The approved head, a pinned +`?rev=`, and a proposal branch are all the same code path with a different +ref. Nothing is materialized to disk. + +Earlier drafts kept a second tier — a checkout of the approved branch, rebuilt on +each merge — so warren's file-oriented indexer and renderer could work unchanged. +It is dropped. Pinned reads are in scope (see the read contract), and the review +UI must render and diff proposal content regardless, so **the git-object read +path has to exist either way**. Keeping a checkout as well would mean two paths +that must agree, plus atomic rebuilds, rev stamps, staleness coupling, and a +cache directory to operate. + +**The absorption cost is small and bounded.** `vault.Scan(root string)` touches +the filesystem in exactly two places (`filepath.WalkDir`, `os.ReadFile`), and the +seam that replaces it already exists: `vault.FromPages(pages, aliases, assets)` +builds an `Archive` with no filesystem at all. A git-tree walk feeding +`FromPages` replaces the scan, and nothing downstream of `Archive` changes. + +What remains is a **render cache keyed by blob sha** — content-addressed, so it +can never go stale, and droppable at any moment. ## Domain model @@ -224,9 +232,16 @@ ignored. Rejection must therefore happen earlier: - **`update`** (per-ref, runs before the ref moves, can reject) — enforces the refs rule: an agent token may only touch `proposals/*`; the approved branch accepts fast-forwards from you only, never a force-update. Also where - frontmatter and ID-collision validation rejects a bad push. -- **`post-receive`** (after the fact, cannot reject) — materialize the checkout, - notify, reindex. + frontmatter and global-ID-collision validation rejects a bad push. + + **Your own pushes are validated too**, with a `--push-option=skip-validation` + escape hatch. The failure mode being guarded is not malice, it is a typo: a + duplicated or malformed `id:` corrupts the global registry and silently breaks + link resolution and search, and is far cheaper to reject at push time than to + find weeks later. The escape hatch exists so that a hook bug or a bad schema + can never lock you out of your own repository. +- **`post-receive`** (after the fact, cannot reject) — notify the daemon so it + reindexes the changed documents and updates the space's index rev stamp. #### The hook is service code, not a shell script @@ -243,10 +258,9 @@ unindexed one is a corruption you discover much later. (The reconciler in "Consistency and recovery" is the backstop that repairs the other direction.) **Cost note:** warren's `index.Build` is a **batch full rebuild** (`bleve.New` -plus a full vault rescan). Per-document incremental upsert/delete is therefore -**net-new work, not absorption** — see the reuse inventory. Whether it is -actually needed in v1 depends on write volume; a full rebuild is fine at tens of -documents a day. +plus a full vault rescan), with no incremental path. At the confirmed volume — +tens of documents a day or fewer — a full rebuild per merge is acceptable, so +this is absorbed as-is rather than rewritten. See the reuse inventory. Agents keep using the REST/MCP write plane; they never speak git at all. That asymmetry is deliberate — it is what makes `If-Match` and provenance trailers @@ -265,21 +279,23 @@ project: ~bigbes/+everything spaces: - ~bigbes/tarantool-rfcs - ~bigbes/home-ops -#mounts: # read-only external corpora — NOT in v1, see below -# - {type: dir, path: /Users/blikh/data/home/second-brain} ``` The requested **meta-project — "merge all my doc work into one searchable thing" — is just a project whose membership is everything.** There is no separate aggregate entity, no copying, and no sync job. -**Scope note, and a live tension.** "All my doc work" now means *all spaces owned -by this service*, not everything on disk: the confirmed boundary is -agent-authored specs only, with `second-brain` and the Confluence-synced RFCs -staying independent. So the meta-project unifies what spec.sr.ht owns. The mount -mechanism stays specified because it is the escape hatch if "searchable in one -place" later turns out to have meant *literally* everything — but it is not -built in v1, and the service is deliberately empty until agents fill it. +**Scope note.** "All my doc work" means *all spaces owned by this service*, not +everything on disk: the confirmed boundary is agent-authored specs only, with +`second-brain` and the Confluence-synced RFCs staying independent. The +meta-project therefore unifies what spec.sr.ht owns, and the service is +deliberately empty until you and your agents fill it. + +**Mounting external corpora read-only is a plausible future, deliberately left +unspecified.** No syntax, no mount types, no external-ID mapping — designing it +now would be guessing. The one thing keeping the door open is already decided: +**document IDs are globally unique**, so a later import cannot collide with what +is already here. That is the whole of the forward compatibility this needs. Two consequences that must be designed in from the start rather than retrofitted: @@ -428,14 +444,12 @@ whole-document `PUT`, which gives an agent no way to express "delete this" or ## Consistency and recovery -Four systems are touched by a merge — git refs, the materialized checkout, the -bleve index, and Postgres — and **none of it is transactional**. Earlier drafts -only said "the checkout is a rebuildable cache", which covers one of the four. -The rule that makes the rest tractable: +Three systems are touched by a merge — git refs, the bleve index, and Postgres — +and **none of it is transactional**. The rule that makes this tractable: > **Git refs are the source of truth for whether a proposal exists and whether it -> merged. Postgres holds metadata that is reconstructable from git. The checkout -> and the index are pure caches.** +> merged. Postgres holds metadata that is reconstructable from git. The index +> and the render cache are pure caches.** That gives every crash a defined repair, rather than a bespoke recovery per failure point: @@ -444,8 +458,7 @@ failure point: |---|---|---| | branch write and row insert | orphan `proposals/*` ref | reconciler recreates the row from the ref | | merge commit and row update | merged ref, row still `open` | reconciler marks merged (ref is truth) | -| merge and checkout rebuild | stale checkout | rev stamp mismatch → rebuild | -| checkout and reindex | stale index | index rev stamp mismatch → reindex | +| merge and reindex | stale index | per-space index rev stamp ≠ approved head → reindex | A **reconciler runs at startup and periodically**: scan `proposals/*` refs and each space's approved head, compare against rows and the index rev stamps, repair @@ -507,14 +520,26 @@ point of the permission model. Concretely, this deletes from v1: visibility levels (public/unlisted/private), approver lists, approval counts, request-changes round-trips, and per-human ACL -rows. It keeps, and arguably sharpens: **per-agent tokens scoped to a space set -and a role**, query-time index filtering by that scope, and the refs rule from -"Two write paths". The unified-login cookie is still needed — not to tell users -apart, but to tell *you* from an unauthenticated request. - -The practical value is blast-radius control: a research agent looping over a -notes space cannot touch `specs/`, and a compromised or confused token cannot -reach the approved branch of anything. +rows. The unified-login cookie is still needed — not to tell users apart, but to +tell *you* from an unauthenticated request. + +**v1 ships one agent token plus mandatory provenance, not per-agent scopes.** +The distinction worth drawing is between the two boundaries this design has: + +- **The refs rule is the boundary that matters, and it is free.** No agent + credential can move the approved branch — only `proposals/*`. That is what + actually bounds the damage a confused or runaway agent can do, and it holds + with a single shared token. +- **Per-space scoping is the boundary that can wait.** Preventing a + notes-writing agent from proposing into `specs/` is real defence in depth, but + every agent here is a session you launched yourself, and a bad proposal is + visible and rejectable rather than destructive. Adding scopes later is a column + on the token row plus a filter clause — not an architectural change. + +What is **not** optional is provenance: agent identity and session ID are +required on every write and recorded in git trailers. One token still yields a +full audit trail, because the trailers, not the credential, are what identify +who did what. ### Provenance @@ -551,16 +576,17 @@ consume this: `spec_search`, `spec_read`, `spec_propose`, `spec_comment`, **Two absorption gaps, both understated in earlier drafts:** - **`index.Build` is a batch full rebuild** (`bleve.New` + full vault rescan + - chunking). There is no incremental path, so "reindex only the changed - documents" is net-new work. Whether v1 needs it is a volume question — a full - rebuild is fine at tens of documents a day, and the honest answer is to measure - before building incremental upsert/delete. -- **`vault`/`render` are file-oriented**, so `?rev=` pinned reads need a - blob→renderer path that bypasses the vault scan (see "Two-tier storage"). - -Genuinely net-new: proposals, prose diff, review plane, agent tokens and scoping, -frontmatter lifecycle, the reconciler, and incremental indexing if volume demands -it. + chunking), with no incremental path. **At the confirmed volume — tens of + documents a day or fewer — this is fine and is absorbed as-is.** Incremental + per-document upsert/delete is explicitly *not* built. If merges ever get slow + enough to notice, that is the signal to revisit; worth instrumenting rebuild + duration so the trigger is measured rather than guessed. +- **`vault.Scan` is file-oriented**, so it is replaced by a git-tree walk feeding + the existing `vault.FromPages` seam (see "One storage tier"). Bounded and + small; everything downstream of `Archive` is untouched. + +Genuinely net-new: proposals, prose diff, review plane, provenance trailers, +frontmatter lifecycle, and the reconciler. ## SourceHut integration @@ -587,7 +613,7 @@ api-origin=https://spec.srht.bigb.es connection-string=postgresql://specsrht@localhost/spec.sr.ht?sslmode=disable ; Bare repos, service-owned: /~/ repos=/var/lib/spec -; Materialized checkouts + bleve project indexes. Pure cache; safe to delete. +; Bleve index + render cache. Pure cache; safe to delete at any time. cache=/var/cache/spec static-dir=/usr/share/sourcehut/spec.sr.ht/static ;bind-address=127.0.0.1:5091 @@ -718,22 +744,23 @@ code. **Phase 1 — core + storage + your push path.** Pure domain (space/doc/rev/ID validation, frontmatter parse + schema validation), bare-repo lifecycle, -materialized checkouts with atomic swap and rev stamps, Postgres schema + +the git-object read layer (tree walk → `vault.FromPages`), Postgres schema + migrations, the reconciler, and the **SSH push path**: an `update` hook that -validates and enforces the refs rule, plus `post-receive` that materializes. Both +validates and enforces the refs rule, plus `post-receive` that notifies. Both hooks RPC into the daemon. No indexing yet — that arrives with the read plane in Phase 2. End state: you can `git push` a space, bad pushes are rejected, and the -service knows about it. +service can read what landed. **Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified -login, `?rev=` pinning, project index with query-time scope filtering, search, -and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and +login, `?rev=` pinning, one global bleve index with query-time project filtering, +render cache, search, and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and `api-origin` in config, federating into `api.sr.ht`. End state: agents can read and search everything you have pushed — useful on its own, before any review machinery exists. **Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model, -agent tokens + scoping + provenance trailers, REST + MCP write tools. MCP tools +the agent token + provenance trailers, REST + MCP write tools (every write +response carrying the proposal URL). MCP tools and GraphQL resolvers share one service layer — no parallel implementations. **Phase 4 — review plane.** Proposal pages at stable URLs (returned by every @@ -744,7 +771,8 @@ unreviewed by design, so it must at least be *visible* or it rots silently). **Phase 5 — later.** Inline comments (anchoring above), webhooks and notifications (`core-go/webhooks`, GraphQL-native — the Phase 2 schema is the foundation), GraphQL **mutations** once the proposal state machine has stopped -moving, vector search, read-only mounts if the boundary ever moves. +moving, vector search, per-space token scoping and read-only mounts if the +boundary ever moves. **Dropped outright:** the web editor (you edit via clone and push), and with it the concurrent-web-edit-vs-push conflict problem it would have created. @@ -783,31 +811,27 @@ the concurrent-web-edit-vs-push conflict problem it would have created. - **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL. -### Raised by review, awaiting a decision - -Ordered by how much the answer would change the build: - -1. **Do agents genuinely need `?rev=` pinning and the approved/draft split?** It - is a founding premise of this document, but if every real consumer just wants - "current approved text", the pinning machinery, the dual read path, and part - of the token-scope model all simplify. -2. **Is the materialized checkout a requirement or an implementation detail?** - Dropping it for blob reads at the approved ref plus a render cache removes the - atomic-swap problem, the staleness coupling, and a whole cache tier — at the - cost of a deeper rework of warren's file-oriented scan. -3. **What is the real firehose volume?** Decides whether full-index-rebuild-on- - merge is fine (deleting the incremental-indexing work), whether retention - matters, and whether merge contention is real or theoretical. -4. **Scoped per-agent tokens in v1, or one token plus mandatory provenance?** If - every agent is a Claude Code session you launched, one token and trailers give - full provenance for a fraction of the machinery; scoping can arrive when a - genuinely autonomous agent does. -5. **Should your own pushes be validated at all, or are you trusted absolutely?** - If you can push whatever you like, the `update` hook shrinks to "no force-push - on approved" and the entire validation stack lives only in the API path, where - it is in-process and easy. -6. **Will existing corpora ever be imported or mounted?** If the honest answer is - never, delete mounts from this document entirely. -7. **Is `spec` the final name?** Low architectural leverage, but it locks the - config section, DNS, module path and nav entry on day one, and every example - here already hardcodes it. +### Answered by review round (2026-07-22) + +All seven questions raised by the independent review have been decided and folded +into the sections above: + +| Question | Answer | +|---|---| +| `?rev=` pinning and approved/draft split? | **Both.** Nearly free — the review UI needs blob→render at arbitrary revs regardless — and it makes `X-Agent-Base` auditable. | +| Materialized checkout? | **Dropped.** One read path over git objects; render cache keyed by blob sha. | +| Firehose volume? | **Tens of documents a day or fewer.** Batch index rebuild absorbed as-is; incremental indexing not built; retention a non-issue. | +| Agent token machinery? | **One token + mandatory provenance.** The refs rule is the boundary that matters; per-space scoping deferred to a column and a filter clause. | +| Validate your own pushes? | **Yes, with `--push-option=skip-validation`.** Guards typos that corrupt the global ID registry, without a lockout risk. | +| Import external corpora? | **Probably eventually; left unspecified.** Global IDs are the only forward compatibility required. | +| Name? | **`spec.sr.ht` at `spec.srht.bigb.es`**, module `sourcecraft.dev/bigbes/sr-ht-spec`, binary `specsrht`. | + +### Still open + +- **Prose diff quality** — not a question to answer in prose but the Phase 0 gate + to run. Everything else assumes it comes out well. +- **Mixed ru/en per-document language routing** (above): still needs a decision + once there is a real corpus to test against. +- **Attachments and binaries** (above): unresolved, and cheap to defer until a + spec actually needs an image. +- **LICENSE** (above).