1 files changed, 125 insertions(+), 101 deletions(-)

M docs/DESIGN.md

@@ 48,6 48,12 @@ modification**. Runs at `https://spec.srht.bigb.es`.
 | Cadence | **Bimodal.** Specs/RFCs get careful review; research notes and reports flow through with light or automatic approval. `.spec.yml` policy carries the split. |
 | Human edit path | **`git clone`, edit locally, push.** The web editor is not v1 and may never be. This makes a real git remote a v1 requirement — see "Two write paths" below. |
 | Review surface | **Browser, reached by a link the agent hands you.** Confirms the prose differ as a v1 build and keeps it as the Phase 0 gate. The inbox is the backstop for work no link reached, not the primary entry point. |
+| Read contract | **Pinned `?rev=` plus an approved/draft split.** Nearly free, since the review UI needs blob→render at arbitrary revs anyway, and it makes `X-Agent-Base` provenance auditable rather than decorative. |
+| Storage tiers | **One.** Git objects are read directly; the materialized checkout is dropped, leaving a bleve index and a blob-sha-keyed render cache as the only caches. |
+| Volume | **Tens of documents a day or fewer.** warren's batch index rebuild is absorbed unchanged; incremental indexing is explicitly not built. |
+| Agent tokens | **One token plus mandatory provenance.** The refs rule (agents write only `proposals/*`) is the boundary that matters; per-space scoping deferred. |
+| Push validation | **Your pushes are validated too**, with a `--push-option=skip-validation` escape hatch. |
+| Name | **`spec.sr.ht`** at `spec.srht.bigb.es`; module `sourcecraft.dev/bigbes/sr-ht-spec`, binary `specsrht`. |
 
 ## Architecture summary
 


@@ 59,11 65,10 @@ One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht`
                               │           <repos>/~user/<space>
    you ──git push (ssh)───────────────────────┘  ▲
                               │                  │ update + post-receive hooks
-                              ├──►  materialized checkouts (read/index cache)
-                              │           <cache>/~user/<space>/
-                              ├──►  one global bleve index
+                              ├──►  one global bleve index   (cache)
+                              ├──►  render cache, keyed by blob sha  (cache)
                               └──►  Postgres (proposals, comments, agent
-                                              tokens + scopes, ID registry)
+                                              tokens, ID registry)
 
    you ──browser──────────►  read + review UI (no editing)
 ```


@@ 79,25 84,28 @@ small. It is not a complete export of *everything*: review decisions, proposal
 rationale and comments live in Postgres, so a clone gives you the documents and
 who wrote them, not the review record.
 
-### Two-tier storage, and why
-
-- **Bare repo** — the push target and the source of truth. One repo per *space*.
-- **Materialized checkout** of the approved branch — a plain directory of
-  markdown, rebuilt on every merge. This exists so warren's indexer and renderer
-  can work on files (which is what they already do), and so reads never pay
-  object-database costs. It is a **cache**: deletable, rebuildable from the bare
-  repo at any time. Rebuild **atomically** — build into a temp dir and rename
-  into place — or readers observe a torn tree mid-rebuild. Stamp each checkout
-  with the rev it was built from, so staleness is detectable rather than assumed.
-
-**Caveat this tier does not cover:** `?rev=<sha>` pinned reads have no checkout
-to read from, since only the approved head is materialized. They need a second
-read path — blob → renderer, straight from the object database, with no vault
-scan. warren's `vault`/`render` split is file-oriented, so this is a real if
-modest rework rather than pure absorption. It is also an argument for questioning
-the checkout tier altogether: reading blobs at the approved ref plus a render
-cache would make pinned and default reads *the same code path* and delete this
-cache subsystem entirely. Open — see "Open items".
+### One storage tier: git objects, no checkout
+
+**Every read resolves a git tree and reads blobs.** The approved head, a pinned
+`?rev=<sha>`, and a proposal branch are all the same code path with a different
+ref. Nothing is materialized to disk.
+
+Earlier drafts kept a second tier — a checkout of the approved branch, rebuilt on
+each merge — so warren's file-oriented indexer and renderer could work unchanged.
+It is dropped. Pinned reads are in scope (see the read contract), and the review
+UI must render and diff proposal content regardless, so **the git-object read
+path has to exist either way**. Keeping a checkout as well would mean two paths
+that must agree, plus atomic rebuilds, rev stamps, staleness coupling, and a
+cache directory to operate.
+
+**The absorption cost is small and bounded.** `vault.Scan(root string)` touches
+the filesystem in exactly two places (`filepath.WalkDir`, `os.ReadFile`), and the
+seam that replaces it already exists: `vault.FromPages(pages, aliases, assets)`
+builds an `Archive` with no filesystem at all. A git-tree walk feeding
+`FromPages` replaces the scan, and nothing downstream of `Archive` changes.
+
+What remains is a **render cache keyed by blob sha** — content-addressed, so it
+can never go stale, and droppable at any moment.
 
 ## Domain model
 


@@ 224,9 232,16 @@ ignored. Rejection must therefore happen earlier:
 - **`update`** (per-ref, runs before the ref moves, can reject) — enforces the
   refs rule: an agent token may only touch `proposals/*`; the approved branch
   accepts fast-forwards from you only, never a force-update. Also where
-  frontmatter and ID-collision validation rejects a bad push.
-- **`post-receive`** (after the fact, cannot reject) — materialize the checkout,
-  notify, reindex.
+  frontmatter and global-ID-collision validation rejects a bad push.
+
+  **Your own pushes are validated too**, with a `--push-option=skip-validation`
+  escape hatch. The failure mode being guarded is not malice, it is a typo: a
+  duplicated or malformed `id:` corrupts the global registry and silently breaks
+  link resolution and search, and is far cheaper to reject at push time than to
+  find weeks later. The escape hatch exists so that a hook bug or a bad schema
+  can never lock you out of your own repository.
+- **`post-receive`** (after the fact, cannot reject) — notify the daemon so it
+  reindexes the changed documents and updates the space's index rev stamp.
 
 #### The hook is service code, not a shell script
 


@@ 243,10 258,9 @@ unindexed one is a corruption you discover much later. (The reconciler in
 "Consistency and recovery" is the backstop that repairs the other direction.)
 
 **Cost note:** warren's `index.Build` is a **batch full rebuild** (`bleve.New`
-plus a full vault rescan). Per-document incremental upsert/delete is therefore
-**net-new work, not absorption** — see the reuse inventory. Whether it is
-actually needed in v1 depends on write volume; a full rebuild is fine at tens of
-documents a day.
+plus a full vault rescan), with no incremental path. At the confirmed volume —
+tens of documents a day or fewer — a full rebuild per merge is acceptable, so
+this is absorbed as-is rather than rewritten. See the reuse inventory.
 
 Agents keep using the REST/MCP write plane; they never speak git at all. That
 asymmetry is deliberate — it is what makes `If-Match` and provenance trailers


@@ 265,21 279,23 @@ project: ~bigbes/+everything
 spaces:
   - ~bigbes/tarantool-rfcs
   - ~bigbes/home-ops
-#mounts:                      # read-only external corpora — NOT in v1, see below
-#  - {type: dir, path: /Users/blikh/data/home/second-brain}
 ```
 
 The requested **meta-project — "merge all my doc work into one searchable
 thing" — is just a project whose membership is everything.** There is no
 separate aggregate entity, no copying, and no sync job.
 
-**Scope note, and a live tension.** "All my doc work" now means *all spaces owned
-by this service*, not everything on disk: the confirmed boundary is
-agent-authored specs only, with `second-brain` and the Confluence-synced RFCs
-staying independent. So the meta-project unifies what spec.sr.ht owns. The mount
-mechanism stays specified because it is the escape hatch if "searchable in one
-place" later turns out to have meant *literally* everything — but it is not
-built in v1, and the service is deliberately empty until agents fill it.
+**Scope note.** "All my doc work" means *all spaces owned by this service*, not
+everything on disk: the confirmed boundary is agent-authored specs only, with
+`second-brain` and the Confluence-synced RFCs staying independent. The
+meta-project therefore unifies what spec.sr.ht owns, and the service is
+deliberately empty until you and your agents fill it.
+
+**Mounting external corpora read-only is a plausible future, deliberately left
+unspecified.** No syntax, no mount types, no external-ID mapping — designing it
+now would be guessing. The one thing keeping the door open is already decided:
+**document IDs are globally unique**, so a later import cannot collide with what
+is already here. That is the whole of the forward compatibility this needs.
 
 Two consequences that must be designed in from the start rather than retrofitted:
 


@@ 428,14 444,12 @@ whole-document `PUT`, which gives an agent no way to express "delete this" or
 
 ## Consistency and recovery
 
-Four systems are touched by a merge — git refs, the materialized checkout, the
-bleve index, and Postgres — and **none of it is transactional**. Earlier drafts
-only said "the checkout is a rebuildable cache", which covers one of the four.
-The rule that makes the rest tractable:
+Three systems are touched by a merge — git refs, the bleve index, and Postgres —
+and **none of it is transactional**. The rule that makes this tractable:
 
 > **Git refs are the source of truth for whether a proposal exists and whether it
-> merged. Postgres holds metadata that is reconstructable from git. The checkout
-> and the index are pure caches.**
+> merged. Postgres holds metadata that is reconstructable from git. The index
+> and the render cache are pure caches.**
 
 That gives every crash a defined repair, rather than a bespoke recovery per
 failure point:


@@ 444,8 458,7 @@ failure point:
 |---|---|---|
 | branch write and row insert | orphan `proposals/*` ref | reconciler recreates the row from the ref |
 | merge commit and row update | merged ref, row still `open` | reconciler marks merged (ref is truth) |
-| merge and checkout rebuild | stale checkout | rev stamp mismatch → rebuild |
-| checkout and reindex | stale index | index rev stamp mismatch → reindex |
+| merge and reindex | stale index | per-space index rev stamp ≠ approved head → reindex |
 
 A **reconciler runs at startup and periodically**: scan `proposals/*` refs and
 each space's approved head, compare against rows and the index rev stamps, repair


@@ 507,14 520,26 @@ point of the permission model.
 
 Concretely, this deletes from v1: visibility levels (public/unlisted/private),
 approver lists, approval counts, request-changes round-trips, and per-human ACL
-rows. It keeps, and arguably sharpens: **per-agent tokens scoped to a space set
-and a role**, query-time index filtering by that scope, and the refs rule from
-"Two write paths". The unified-login cookie is still needed — not to tell users
-apart, but to tell *you* from an unauthenticated request.
-
-The practical value is blast-radius control: a research agent looping over a
-notes space cannot touch `specs/`, and a compromised or confused token cannot
-reach the approved branch of anything.
+rows. The unified-login cookie is still needed — not to tell users apart, but to
+tell *you* from an unauthenticated request.
+
+**v1 ships one agent token plus mandatory provenance, not per-agent scopes.**
+The distinction worth drawing is between the two boundaries this design has:
+
+- **The refs rule is the boundary that matters, and it is free.** No agent
+  credential can move the approved branch — only `proposals/*`. That is what
+  actually bounds the damage a confused or runaway agent can do, and it holds
+  with a single shared token.
+- **Per-space scoping is the boundary that can wait.** Preventing a
+  notes-writing agent from proposing into `specs/` is real defence in depth, but
+  every agent here is a session you launched yourself, and a bad proposal is
+  visible and rejectable rather than destructive. Adding scopes later is a column
+  on the token row plus a filter clause — not an architectural change.
+
+What is **not** optional is provenance: agent identity and session ID are
+required on every write and recorded in git trailers. One token still yields a
+full audit trail, because the trailers, not the credential, are what identify
+who did what.
 
 ### Provenance
 


@@ 551,16 576,17 @@ consume this: `spec_search`, `spec_read`, `spec_propose`, `spec_comment`,
 **Two absorption gaps, both understated in earlier drafts:**
 
 - **`index.Build` is a batch full rebuild** (`bleve.New` + full vault rescan +
-  chunking). There is no incremental path, so "reindex only the changed
-  documents" is net-new work. Whether v1 needs it is a volume question — a full
-  rebuild is fine at tens of documents a day, and the honest answer is to measure
-  before building incremental upsert/delete.
-- **`vault`/`render` are file-oriented**, so `?rev=` pinned reads need a
-  blob→renderer path that bypasses the vault scan (see "Two-tier storage").
-
-Genuinely net-new: proposals, prose diff, review plane, agent tokens and scoping,
-frontmatter lifecycle, the reconciler, and incremental indexing if volume demands
-it.
+  chunking), with no incremental path. **At the confirmed volume — tens of
+  documents a day or fewer — this is fine and is absorbed as-is.** Incremental
+  per-document upsert/delete is explicitly *not* built. If merges ever get slow
+  enough to notice, that is the signal to revisit; worth instrumenting rebuild
+  duration so the trigger is measured rather than guessed.
+- **`vault.Scan` is file-oriented**, so it is replaced by a git-tree walk feeding
+  the existing `vault.FromPages` seam (see "One storage tier"). Bounded and
+  small; everything downstream of `Archive` is untouched.
+
+Genuinely net-new: proposals, prose diff, review plane, provenance trailers,
+frontmatter lifecycle, and the reconciler.
 
 ## SourceHut integration
 


@@ 587,7 613,7 @@ api-origin=https://spec.srht.bigb.es
 connection-string=postgresql://specsrht@localhost/spec.sr.ht?sslmode=disable
 ; Bare repos, service-owned: <repos>/~<user>/<space>
 repos=/var/lib/spec
-; Materialized checkouts + bleve project indexes. Pure cache; safe to delete.
+; Bleve index + render cache. Pure cache; safe to delete at any time.
 cache=/var/cache/spec
 static-dir=/usr/share/sourcehut/spec.sr.ht/static
 ;bind-address=127.0.0.1:5091


@@ 718,22 744,23 @@ code.
 
 **Phase 1 — core + storage + your push path.** Pure domain (space/doc/rev/ID
 validation, frontmatter parse + schema validation), bare-repo lifecycle,
-materialized checkouts with atomic swap and rev stamps, Postgres schema +
+the git-object read layer (tree walk → `vault.FromPages`), Postgres schema +
 migrations, the reconciler, and the **SSH push path**: an `update` hook that
-validates and enforces the refs rule, plus `post-receive` that materializes. Both
+validates and enforces the refs rule, plus `post-receive` that notifies. Both
 hooks RPC into the daemon. No indexing yet — that arrives with the read plane in
 Phase 2. End state: you can `git push` a space, bad pushes are rejected, and the
-service knows about it.
+service can read what landed.
 
 **Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified
-login, `?rev=` pinning, project index with query-time scope filtering, search,
-and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and
+login, `?rev=` pinning, one global bleve index with query-time project filtering,
+render cache, search, and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and
 `api-origin` in config, federating into `api.sr.ht`. End state: agents can read
 and search everything you have pushed — useful on its own, before any review
 machinery exists.
 
 **Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model,
-agent tokens + scoping + provenance trailers, REST + MCP write tools. MCP tools
+the agent token + provenance trailers, REST + MCP write tools (every write
+response carrying the proposal URL). MCP tools
 and GraphQL resolvers share one service layer — no parallel implementations.
 
 **Phase 4 — review plane.** Proposal pages at stable URLs (returned by every


@@ 744,7 771,8 @@ unreviewed by design, so it must at least be *visible* or it rots silently).
 **Phase 5 — later.** Inline comments (anchoring above), webhooks and
 notifications (`core-go/webhooks`, GraphQL-native — the Phase 2 schema is the
 foundation), GraphQL **mutations** once the proposal state machine has stopped
-moving, vector search, read-only mounts if the boundary ever moves.
+moving, vector search, per-space token scoping and read-only mounts if the
+boundary ever moves.
 
 **Dropped outright:** the web editor (you edit via clone and push), and with it
 the concurrent-web-edit-vs-push conflict problem it would have created.


@@ 783,31 811,27 @@ the concurrent-web-edit-vs-push conflict problem it would have created.
 - **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are
   AGPL/GPL.
 
-### Raised by review, awaiting a decision
-
-Ordered by how much the answer would change the build:
-
-1. **Do agents genuinely need `?rev=` pinning and the approved/draft split?** It
-   is a founding premise of this document, but if every real consumer just wants
-   "current approved text", the pinning machinery, the dual read path, and part
-   of the token-scope model all simplify.
-2. **Is the materialized checkout a requirement or an implementation detail?**
-   Dropping it for blob reads at the approved ref plus a render cache removes the
-   atomic-swap problem, the staleness coupling, and a whole cache tier — at the
-   cost of a deeper rework of warren's file-oriented scan.
-3. **What is the real firehose volume?** Decides whether full-index-rebuild-on-
-   merge is fine (deleting the incremental-indexing work), whether retention
-   matters, and whether merge contention is real or theoretical.
-4. **Scoped per-agent tokens in v1, or one token plus mandatory provenance?** If
-   every agent is a Claude Code session you launched, one token and trailers give
-   full provenance for a fraction of the machinery; scoping can arrive when a
-   genuinely autonomous agent does.
-5. **Should your own pushes be validated at all, or are you trusted absolutely?**
-   If you can push whatever you like, the `update` hook shrinks to "no force-push
-   on approved" and the entire validation stack lives only in the API path, where
-   it is in-process and easy.
-6. **Will existing corpora ever be imported or mounted?** If the honest answer is
-   never, delete mounts from this document entirely.
-7. **Is `spec` the final name?** Low architectural leverage, but it locks the
-   config section, DNS, module path and nav entry on day one, and every example
-   here already hardcodes it.
+### Answered by review round (2026-07-22)
+
+All seven questions raised by the independent review have been decided and folded
+into the sections above:
+
+| Question | Answer |
+|---|---|
+| `?rev=` pinning and approved/draft split? | **Both.** Nearly free — the review UI needs blob→render at arbitrary revs regardless — and it makes `X-Agent-Base` auditable. |
+| Materialized checkout? | **Dropped.** One read path over git objects; render cache keyed by blob sha. |
+| Firehose volume? | **Tens of documents a day or fewer.** Batch index rebuild absorbed as-is; incremental indexing not built; retention a non-issue. |
+| Agent token machinery? | **One token + mandatory provenance.** The refs rule is the boundary that matters; per-space scoping deferred to a column and a filter clause. |
+| Validate your own pushes? | **Yes, with `--push-option=skip-validation`.** Guards typos that corrupt the global ID registry, without a lockout risk. |
+| Import external corpora? | **Probably eventually; left unspecified.** Global IDs are the only forward compatibility required. |
+| Name? | **`spec.sr.ht` at `spec.srht.bigb.es`**, module `sourcecraft.dev/bigbes/sr-ht-spec`, binary `specsrht`. |
+
+### Still open
+
+- **Prose diff quality** — not a question to answer in prose but the Phase 0 gate
+  to run. Everything else assumes it comes out well.
+- **Mixed ru/en per-document language routing** (above): still needs a decision
+  once there is a real corpus to test against.
+- **Attachments and binaries** (above): unresolved, and cheap to defer until a
+  spec actually needs an image.
+- **LICENSE** (above).