From be096fc6b65eea65c2ff497a89860d4d031efb59 Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 04:10:39 +0300 Subject: [PATCH] docs: design proposal for spec.sr.ht Reviewable document storage for the self-hosted SourceHut instance: bots propose, humans review and curate, bots consume the approved text. Records the four confirmed decisions (proposal-first review gate, own bare git repos, absorb warren's read plane, thin full-loop v1) and the projects model that gives cross-space unified search. --- docs/DESIGN.md | 336 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 docs/DESIGN.md diff --git a/docs/DESIGN.md b/docs/DESIGN.md new file mode 100644 index 0000000000000000000000000000000000000000..96f8b3019fbb6f90ec5e01be7a1d271b634b0564 --- /dev/null +++ b/docs/DESIGN.md @@ -0,0 +1,336 @@ +# spec.sr.ht — reviewable document storage for humans and agents + +Status: **design / proposal** — nothing implemented yet. +Author: bigbes (with Claude). Last updated: 2026-07-22. + +## Context + +bigbes wants a third custom service on the self-hosted SourceHut instance +(`*.srht.bigb.es`), after `dolt.sr.ht` (`../sourcehut-dolt`) and `compare.sr.ht` +(`../sourcehut-compare`): **separate storage for specs and documents that is +reviewable by a human, modifiable by a human, and uploaded to and read by bots.** + +Those three verbs are not three features. They are one loop: + +> **bot produces → human curates → bots consume.** + +Everything below follows from taking that loop seriously. Three properties of the +loop are what a plain git repo, a wiki, or upstream `man.sr.ht` each fail to +provide: + +1. **The unit of work is a proposal, not a commit.** Agents produce more text + than a human can read. A store where agents write freely and nobody reviews is + a wiki that rots within a week. The thing you interact with daily is a *review + queue*, not a file tree. +2. **Bots need a stable read contract.** "Give me the approved text of SPEC-0007", + not "HEAD of main, which another agent is halfway through rewriting". Draft and + canonical must be different addresses. +3. **Provenance is review context.** Which agent, which session, on whose behalf, + against which base revision. Reviewing agent output without that is reviewing + anonymous text. + +Same integration model as the two siblings: one Go module, config-driven +`[spec.sr.ht]` section, unified-login cookie, **no upstream SourceHut +modification**. Runs at `https://spec.srht.bigb.es`. + +### Decisions taken (user-confirmed, 2026-07-22) + +| Decision | Choice | +|---|---| +| Review gate | **Proposal-first.** Bots always write to a proposal branch; a human approves before anything reaches the approved revision. Per-space policy may whitelist auto-merge namespaces. | +| Storage substrate | **Own bare git repos, service-owned** (`/~user/`), like dolt.sr.ht owns its NBS stores. | +| `../warren` | **Absorbed.** Its `vault`/`render`/`index`/`search`/`mcpsrv` packages become the read plane of this module. warren remains a standalone tool for local vaults. | +| v1 scope | **Full loop, thin.** Read plane + upload-as-proposal (REST + MCP) + minimal review page (prose diff, approve/reject) + agent tokens. Inline comments and the web editor come later. | +| Aggregation | **Projects** (below) — a named set of spaces with one unified index. The "meta-project" is the degenerate case, not special machinery. | + +## Architecture summary + +One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht` +(plus `specsrht-migrate`, a brant wrapper, exactly as dolt.sr.ht does it). + +``` + agents ──MCP / REST──┐ + ├──► specsrht ──► bare git repos (authoritative) + humans ──web UI──────┘ │ /~user/ + │ + ├──► materialized checkouts (read/index cache) + │ /~user// + ├──► bleve indexes, one per project + └──► Postgres (proposals, comments, ACLs, + agent tokens, ID registry) +``` + +**Git is authoritative for document bodies. Postgres never stores a body.** +That keeps `git clone` a complete export, keeps `git log` a complete provenance +record offline, and keeps the blast radius of a Postgres restore small. + +### Two-tier storage, and why + +- **Bare repo** — the push target and the source of truth. One repo per *space*. +- **Materialized checkout** of the approved branch — a plain directory of + markdown, rebuilt on every merge. This exists so warren's indexer and renderer + can work on files (which is what they already do), and so reads never pay + object-database costs. It is a **cache**: deletable, rebuildable from the bare + repo at any time. + +## Domain model + +### Space + +One bare git repo = one space, addressed `~/`. The unit of +ownership, ACL, and review policy. Contains markdown documents with YAML +frontmatter, plus attachments. + +A versioned `.spec.yml` at the repo root carries the space's own policy — which +means policy changes are themselves reviewable: + +```yaml +review: + required: 1 # approvals needed to merge + approvers: [~bigbes] + auto_merge: [notes/**] # paths that skip the gate +schema: # frontmatter contract, enforced at propose time + required: [id, title, status, owners] + status: [draft, review, approved, superseded] +``` + +### Document + +Markdown + YAML frontmatter, matching the `second-brain` / warren conventions +already in use (wikilinks, `type`/`summary`/`tags`). + +```yaml +id: SPEC-0007 # stable; NEVER changes, including on rename +title: Proposal storage model +status: draft | review | approved | superseded +supersedes: SPEC-0003 +owners: [~bigbes] +tags: [storage, review] +``` + +`id` is the load-bearing field. Paths move; IDs do not. Cross-space links +resolve by ID, comment anchors reference IDs, and bots pin to IDs. The service +enforces ID uniqueness per project via a registry table. + +**Frontmatter is validated at propose time.** A bot that omits `status:` gets a +422, not a silent merge. Schema validation at the door is the cheapest available +defense against agent slop, and it costs almost nothing to implement. + +### Proposal + +A branch `proposals/` plus a Postgres row. Bundles N document edits with a +title and rationale. State machine: + +``` +draft ──► open ──► approved ──► merged + │└──► changes-requested ──┘ + └──► rejected +``` + +### Project + +A named set of spaces plus read-only external mounts, with **one unified index, +one search endpoint, one MCP view, and one wikilink namespace**. Modeled on +hub.sr.ht's project-groups-repos idea. + +```yaml +project: ~bigbes/+everything +spaces: + - ~bigbes/tarantool-rfcs + - ~bigbes/home-ops +mounts: # read-only, indexed, searchable, not proposable + - {type: dir, path: /Users/blikh/data/home/second-brain} + - {type: gitsrht, repo: ~bigbes/warren, subdir: docs/} +``` + +The requested **meta-project — "merge all my doc work into one searchable +thing" — is just a project whose membership is everything.** There is no +separate aggregate entity, no copying, and no sync job. Three consequences that +must be designed in from the start rather than retrofitted: + +1. **ACL filtering happens at query time**, against a single shared index. Never + build per-user indexes — that path ends in N indexes and a stale-permission + bug. Each indexed document carries its space ID; the query layer intersects + with the viewer's readable spaces. An agent token scoped to a project resolves + to the intersection of the spaces it may read. +2. **Cross-space links resolve by ID.** `[[SPEC-0007]]` works project-wide; + relative paths work within a space only. The ID registry rejects a merge that + would collide two IDs inside one project. +3. **Read-only mounts are first class.** `second-brain`, `docs/` directories of + existing git.sr.ht repos, man.sr.ht wikis — all searchable without migrating + anything anywhere. This is likely the fastest route to the service being + useful on day one, since it needs no content to be moved into it. + +Incremental reindex on merge: a merge to a space's approved branch reindexes only +the changed documents, in every project index that includes that space. + +## The three planes + +### 1. Read plane (anonymous-capable, cached) + +`GET /~user/space/path.md` renders. Content negotiation gives `.md` raw, +`.json` metadata+body, `?rev=` pinned to an immutable revision. + +**Reads default to the approved revision**, with a visible "draft is 3 changes +ahead" affordance. This is the plane bots consume; it must be boring and +pinnable. Serving drafts by default would poison every downstream agent context +with unreviewed text — which is the exact failure this whole service exists to +prevent. + +Implementation is warren, absorbed: `vault/` (scan + frontmatter), `render/` +(goldmark + wikilinks), `index/` + `search/` (bleve keyword + optional vector), +`linkcheck/`. + +### 2. Write plane (agents) + +```http +PUT /api/v1/spaces/~bigbes/rfcs/docs/specs/0007-storage.md +If-Match: +X-Proposal: # omit to open a new one +``` + +The body is the whole document. **Whole-document upload, not patches** — that is +how agents actually work, and it makes the merge model trivial (see below). +`If-Match` gives optimistic concurrency: two agents editing the same document +cannot silently clobber each other, and the loser gets a 409 telling it to refetch +and re-propose. That is exactly the right UX for a bot, which can re-derive its +edit cheaply. + +The write always lands on `proposals/`, never on the approved branch. + +### 3. Review plane (humans) + +**The inbox is the product.** The landing page is "N proposals waiting on you", +not a file browser. A proposal page shows the prose diff, per-document, with +approve / request-changes / reject, and (post-v1) inline comments and a web +editor that turns a human edit into a commit on the proposal branch. + +## Merge model: no text merge, ever + +**Verified constraint:** `go-git` v5.19.1 implements only `FastForwardMerge` +(`repository.go:1800`; anything else returns `ErrUnsupportedMergeStrategy`). +There is no three-way merge available in-process, and shelling out to `git` is +against the house pattern established by compare.sr.ht. + +This constraint is a gift, because the whole-document grain makes a text merge +unnecessary. Merging proposal `P` (based on `B`, touching file set `F`) into +approved head `H`: + +``` +for f in F: + if blob(f)@H != blob(f)@B: # someone else changed this doc since B + return 409 stale # refetch and re-propose +newTree = tree(H) with each f replaced by P's blob +commit newTree with parents [H, P.head] +``` + +Pure plumbing — `object.Tree` manipulation plus a commit with two parents, all of +which go-git supports directly. No merge algorithm, no conflict markers, no +conflict-resolution UI, ever. A conflict is always "your base moved, re-propose", +which is trivial for an agent and comprehensible for a human. The real merge +commit keeps the proposal visible in `git log`. + +## The two hard parts + +Naming these now so they are not discovered late. + +### Prose diff, not line diff + +Markdown reflows. A one-word edit renders as a whole-paragraph replace under a +line-oriented differ, which makes reviewing agent output miserable — and +reviewing agent output is the entire product. What is needed is **word-level +intra-paragraph diffing over the rendered block structure**, not +`diff --git` output piped into a viewer. + +This is the single UI decision that determines whether the service is pleasant or +useless, which is why it is the Phase 0 gate below. Note that compare.sr.ht's +`@pierre/diffs` bundle is a *code* differ and is the wrong tool here; the prose +differ is likely net-new (segment into blocks → align blocks → word-diff within +matched blocks). + +### Comment anchoring (post-v1, but design now) + +`../sourcehut-compare/docs/inline-comments.md` already hit this with +`(file, side, line)` against moving refs. In prose it is worse, because line +numbers are meaningless across a reflow. + +Anchor to `(doc id, heading path, block index, block content hash)`. Resolve by +content hash first, fall back to heading-path + block index, and when both fail +mark the comment **outdated** rather than silently relocating it. Anchoring to +`doc id` rather than path is what makes comments survive renames. + +## Agent identity and provenance + +Per-space (or per-project) tokens with a role — `reader` / `proposer` / +`writer` — and a **required agent identity string**. Every commit records it in +a way that survives clone: + +``` +Author: claude-code/spec-writer (for bigbes) +Committer: bigbes + + Add storage model section + + X-Agent-Session: 8fb9c9a4-b078-4af1-89eb-d97c522f9921 + X-Agent-Base: +``` + +Git trailers rather than a Postgres-only audit table, so provenance is visible in +plain `git log` on any clone and cannot drift from the content it describes. + +MCP is a **first-class surface, not a wrapper** — it is how agents will actually +consume this: `spec_search`, `spec_read`, `spec_propose`, `spec_comment`, +`spec_status`. warren's `mcpsrv/` is the starting point. + +## Reuse inventory + +| From | What | Notes | +|---|---|---| +| `../warren` | `vault/`, `render/`, `index/`, `search/`, `mcpsrv/`, `linkcheck/` | The entire read plane and bot retrieval surface, already written. Absorbed into this module. | +| `../sourcehut-compare` | chrome/templates, cookie→identity, GraphQL authorizer + TTL cache, SCSS pipeline, `contrib/` nginx+systemd shape | Closest sibling; copy the integration scaffolding wholesale. | +| `../sourcehut-dolt` | bare-store lifecycle under `/~user/`, brant migration wrapper, config validation | Same storage-root and migration patterns. | +| `sr-ht-core` (fork) | config, crypto, auth, database, server | Pinned to `git.srht.bigb.es/~bigbes/core-go` via `replace`, as in both siblings. Never `go get -u`. | + +Genuinely net-new: proposals, prose diff, review UI, agent tokens, frontmatter +lifecycle, projects/aggregation. + +## Phases + +**Phase 0 — de-risk gate.** Two spikes, both must pass before anything else is +built: + +1. go-git write path in-process: create branch → commit blob → build merged tree + → two-parent commit, on a real bare repo, no shell-out. +2. Prose word-diff rendered against two real revisions of an actual existing spec. + **If the diff does not read well, the design needs rethinking before any + further code.** This is the compare.sr.ht/dolt.sr.ht spike discipline applied + to the riskiest assumption here. + +**Phase 1 — core + storage.** Pure domain (space/doc/rev/ID validation, +frontmatter parse + schema validation), bare-repo lifecycle, materialized +checkouts, Postgres schema + migrations. + +**Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified +login, ACLs, `?rev=` pinning, read-only mounts, project index with query-time ACL +filtering. + +**Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model, +agent tokens + provenance trailers, REST + MCP. + +**Phase 4 — review UI.** Inbox, prose diff, approve / request-changes, status +lifecycle transitions. + +**Phase 5 — later.** Inline comments (anchoring above), web editor, webhooks and +notifications, vector search over the project index. + +## Open items + +- **Naming.** `spec.sr.ht` / `spec.srht.bigb.es` follows the + named-by-function pattern of the two siblings. `docs.sr.ht` collides + conceptually with upstream `man.sr.ht`. +- **Project URL namespace.** `~user/+project` distinguishes projects from spaces + (`~user/space`) in one character; alternatives are `/projects/~user/name` or + reusing hub.sr.ht's own namespace. +- **Port.** compare.sr.ht is on 5090, dolt.sr.ht on 5306–5308. 5091 is free. +- **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are + AGPL/GPL.