# spec.sr.ht — reviewable document storage for humans and agents Status: **design / proposal** — nothing implemented yet. Author: bigbes (with Claude). Last updated: 2026-07-22. ## Context bigbes wants a third custom service on the self-hosted SourceHut instance (`*.srht.bigb.es`), after `dolt.sr.ht` (`../sourcehut-dolt`) and `compare.sr.ht` (`../sourcehut-compare`): **separate storage for specs and documents that is reviewable by a human, modifiable by a human, and uploaded to and read by bots.** Those three verbs are not three features. They are one loop: > **bot produces → human curates → bots consume.** Everything below follows from taking that loop seriously. Three properties of the loop are what a plain git repo, a wiki, or upstream `man.sr.ht` each fail to provide: 1. **The unit of work is a proposal, not a commit.** Agents produce more text than a human can read. A store where agents write freely and nobody reviews is a wiki that rots within a week. The thing you interact with daily is a *review queue*, not a file tree. 2. **Bots need a stable read contract.** "Give me the approved text of SPEC-0007", not "HEAD of main, which another agent is halfway through rewriting". Draft and canonical must be different addresses. 3. **Provenance is review context.** Which agent, which session, on whose behalf, against which base revision. Reviewing agent output without that is reviewing anonymous text. Same integration model as the two siblings: one Go module, config-driven `[spec.sr.ht]` section, unified-login cookie, **no upstream SourceHut modification**. Runs at `https://spec.srht.bigb.es`. ### Decisions taken (user-confirmed, 2026-07-22) | Decision | Choice | |---|---| | Review gate | **Proposal-first.** Bots always write to a proposal branch; a human approves before anything reaches the approved revision. Per-space policy may whitelist auto-merge namespaces. | | Storage substrate | **Own bare git repos, service-owned** (`/~user/`), like dolt.sr.ht owns its NBS stores. | | `../warren` | **Absorbed.** Its `vault`/`render`/`index`/`search`/`mcpsrv` packages become the read plane of this module. warren remains a standalone tool for local vaults. | | v1 scope | **Full loop, thin.** Read plane + upload-as-proposal (REST + MCP) + minimal review page (prose diff, approve/reject) + agent tokens. Inline comments and the web editor come later. | | Aggregation | **Projects** (below) — a named set of spaces with one unified index. The "meta-project" is the degenerate case, not special machinery. | ## Architecture summary One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht` (plus `specsrht-migrate`, a brant wrapper, exactly as dolt.sr.ht does it). ``` agents ──MCP / REST──┐ ├──► specsrht ──► bare git repos (authoritative) humans ──web UI──────┘ │ /~user/ │ ├──► materialized checkouts (read/index cache) │ /~user// ├──► bleve indexes, one per project └──► Postgres (proposals, comments, ACLs, agent tokens, ID registry) ``` **Git is authoritative for document bodies. Postgres never stores a body.** That keeps `git clone` a complete export, keeps `git log` a complete provenance record offline, and keeps the blast radius of a Postgres restore small. ### Two-tier storage, and why - **Bare repo** — the push target and the source of truth. One repo per *space*. - **Materialized checkout** of the approved branch — a plain directory of markdown, rebuilt on every merge. This exists so warren's indexer and renderer can work on files (which is what they already do), and so reads never pay object-database costs. It is a **cache**: deletable, rebuildable from the bare repo at any time. ## Domain model ### Space One bare git repo = one space, addressed `~/`. The unit of ownership, ACL, and review policy. Contains markdown documents with YAML frontmatter, plus attachments. A versioned `.spec.yml` at the repo root carries the space's own policy — which means policy changes are themselves reviewable: ```yaml review: required: 1 # approvals needed to merge approvers: [~bigbes] auto_merge: [notes/**] # paths that skip the gate schema: # frontmatter contract, enforced at propose time required: [id, title, status, owners] status: [draft, review, approved, superseded] ``` ### Document Markdown + YAML frontmatter, matching the `second-brain` / warren conventions already in use (wikilinks, `type`/`summary`/`tags`). ```yaml id: SPEC-0007 # stable; NEVER changes, including on rename title: Proposal storage model status: draft | review | approved | superseded supersedes: SPEC-0003 owners: [~bigbes] tags: [storage, review] ``` `id` is the load-bearing field. Paths move; IDs do not. Cross-space links resolve by ID, comment anchors reference IDs, and bots pin to IDs. The service enforces ID uniqueness per project via a registry table. **Frontmatter is validated at propose time.** A bot that omits `status:` gets a 422, not a silent merge. Schema validation at the door is the cheapest available defense against agent slop, and it costs almost nothing to implement. ### Proposal A branch `proposals/` plus a Postgres row. Bundles N document edits with a title and rationale. State machine: ``` draft ──► open ──► approved ──► merged │└──► changes-requested ──┘ └──► rejected ``` ### Project A named set of spaces plus read-only external mounts, with **one unified index, one search endpoint, one MCP view, and one wikilink namespace**. Modeled on hub.sr.ht's project-groups-repos idea. ```yaml project: ~bigbes/+everything spaces: - ~bigbes/tarantool-rfcs - ~bigbes/home-ops mounts: # read-only, indexed, searchable, not proposable - {type: dir, path: /Users/blikh/data/home/second-brain} - {type: gitsrht, repo: ~bigbes/warren, subdir: docs/} ``` The requested **meta-project — "merge all my doc work into one searchable thing" — is just a project whose membership is everything.** There is no separate aggregate entity, no copying, and no sync job. Three consequences that must be designed in from the start rather than retrofitted: 1. **ACL filtering happens at query time**, against a single shared index. Never build per-user indexes — that path ends in N indexes and a stale-permission bug. Each indexed document carries its space ID; the query layer intersects with the viewer's readable spaces. An agent token scoped to a project resolves to the intersection of the spaces it may read. 2. **Cross-space links resolve by ID.** `[[SPEC-0007]]` works project-wide; relative paths work within a space only. The ID registry rejects a merge that would collide two IDs inside one project. 3. **Read-only mounts are first class.** `second-brain`, `docs/` directories of existing git.sr.ht repos, man.sr.ht wikis — all searchable without migrating anything anywhere. This is likely the fastest route to the service being useful on day one, since it needs no content to be moved into it. Incremental reindex on merge: a merge to a space's approved branch reindexes only the changed documents, in every project index that includes that space. ## The three planes ### 1. Read plane (anonymous-capable, cached) `GET /~user/space/path.md` renders. Content negotiation gives `.md` raw, `.json` metadata+body, `?rev=` pinned to an immutable revision. **Reads default to the approved revision**, with a visible "draft is 3 changes ahead" affordance. This is the plane bots consume; it must be boring and pinnable. Serving drafts by default would poison every downstream agent context with unreviewed text — which is the exact failure this whole service exists to prevent. Implementation is warren, absorbed: `vault/` (scan + frontmatter), `render/` (goldmark + wikilinks), `index/` + `search/` (bleve keyword + optional vector), `linkcheck/`. ### 2. Write plane (agents) ```http PUT /api/v1/spaces/~bigbes/rfcs/docs/specs/0007-storage.md If-Match: X-Proposal: # omit to open a new one ``` The body is the whole document. **Whole-document upload, not patches** — that is how agents actually work, and it makes the merge model trivial (see below). `If-Match` gives optimistic concurrency: two agents editing the same document cannot silently clobber each other, and the loser gets a 409 telling it to refetch and re-propose. That is exactly the right UX for a bot, which can re-derive its edit cheaply. The write always lands on `proposals/`, never on the approved branch. ### 3. Review plane (humans) **The inbox is the product.** The landing page is "N proposals waiting on you", not a file browser. A proposal page shows the prose diff, per-document, with approve / request-changes / reject, and (post-v1) inline comments and a web editor that turns a human edit into a commit on the proposal branch. ## Merge model: no text merge, ever **Verified constraint:** `go-git` v5.19.1 implements only `FastForwardMerge` (`repository.go:1800`; anything else returns `ErrUnsupportedMergeStrategy`). There is no three-way merge available in-process, and shelling out to `git` is against the house pattern established by compare.sr.ht. This constraint is a gift, because the whole-document grain makes a text merge unnecessary. Merging proposal `P` (based on `B`, touching file set `F`) into approved head `H`: ``` for f in F: if blob(f)@H != blob(f)@B: # someone else changed this doc since B return 409 stale # refetch and re-propose newTree = tree(H) with each f replaced by P's blob commit newTree with parents [H, P.head] ``` Pure plumbing — `object.Tree` manipulation plus a commit with two parents, all of which go-git supports directly. No merge algorithm, no conflict markers, no conflict-resolution UI, ever. A conflict is always "your base moved, re-propose", which is trivial for an agent and comprehensible for a human. The real merge commit keeps the proposal visible in `git log`. ## The two hard parts Naming these now so they are not discovered late. ### Prose diff, not line diff Markdown reflows. A one-word edit renders as a whole-paragraph replace under a line-oriented differ, which makes reviewing agent output miserable — and reviewing agent output is the entire product. What is needed is **word-level intra-paragraph diffing over the rendered block structure**, not `diff --git` output piped into a viewer. This is the single UI decision that determines whether the service is pleasant or useless, which is why it is the Phase 0 gate below. Note that compare.sr.ht's `@pierre/diffs` bundle is a *code* differ and is the wrong tool here; the prose differ is likely net-new (segment into blocks → align blocks → word-diff within matched blocks). ### Comment anchoring (post-v1, but design now) `../sourcehut-compare/docs/inline-comments.md` already hit this with `(file, side, line)` against moving refs. In prose it is worse, because line numbers are meaningless across a reflow. Anchor to `(doc id, heading path, block index, block content hash)`. Resolve by content hash first, fall back to heading-path + block index, and when both fail mark the comment **outdated** rather than silently relocating it. Anchoring to `doc id` rather than path is what makes comments survive renames. ## Agent identity and provenance Per-space (or per-project) tokens with a role — `reader` / `proposer` / `writer` — and a **required agent identity string**. Every commit records it in a way that survives clone: ``` Author: claude-code/spec-writer (for bigbes) Committer: bigbes Add storage model section X-Agent-Session: 8fb9c9a4-b078-4af1-89eb-d97c522f9921 X-Agent-Base: ``` Git trailers rather than a Postgres-only audit table, so provenance is visible in plain `git log` on any clone and cannot drift from the content it describes. MCP is a **first-class surface, not a wrapper** — it is how agents will actually consume this: `spec_search`, `spec_read`, `spec_propose`, `spec_comment`, `spec_status`. warren's `mcpsrv/` is the starting point. ## Reuse inventory | From | What | Notes | |---|---|---| | `../warren` | `vault/`, `render/`, `index/`, `search/`, `mcpsrv/`, `linkcheck/` | The entire read plane and bot retrieval surface, already written. Absorbed into this module. | | `../sourcehut-compare` | chrome/templates, cookie→identity, GraphQL authorizer + TTL cache, SCSS pipeline, `contrib/` nginx+systemd shape | Closest sibling; copy the integration scaffolding wholesale. | | `../sourcehut-dolt` | bare-store lifecycle under `/~user/`, brant migration wrapper, config validation | Same storage-root and migration patterns. | | `sr-ht-core` (fork) | config, crypto, auth, database, server | Pinned to `git.srht.bigb.es/~bigbes/core-go` via `replace`, as in both siblings. Never `go get -u`. | Genuinely net-new: proposals, prose diff, review UI, agent tokens, frontmatter lifecycle, projects/aggregation. ## Phases **Phase 0 — de-risk gate.** Two spikes, both must pass before anything else is built: 1. go-git write path in-process: create branch → commit blob → build merged tree → two-parent commit, on a real bare repo, no shell-out. 2. Prose word-diff rendered against two real revisions of an actual existing spec. **If the diff does not read well, the design needs rethinking before any further code.** This is the compare.sr.ht/dolt.sr.ht spike discipline applied to the riskiest assumption here. **Phase 1 — core + storage.** Pure domain (space/doc/rev/ID validation, frontmatter parse + schema validation), bare-repo lifecycle, materialized checkouts, Postgres schema + migrations. **Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified login, ACLs, `?rev=` pinning, read-only mounts, project index with query-time ACL filtering. **Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model, agent tokens + provenance trailers, REST + MCP. **Phase 4 — review UI.** Inbox, prose diff, approve / request-changes, status lifecycle transitions. **Phase 5 — later.** Inline comments (anchoring above), web editor, webhooks and notifications, vector search over the project index. ## Open items - **Naming.** `spec.sr.ht` / `spec.srht.bigb.es` follows the named-by-function pattern of the two siblings. `docs.sr.ht` collides conceptually with upstream `man.sr.ht`. - **Project URL namespace.** `~user/+project` distinguishes projects from spaces (`~user/space`) in one character; alternatives are `/projects/~user/name` or reusing hub.sr.ht's own namespace. - **Port.** compare.sr.ht is on 5090, dolt.sr.ht on 5306–5308. 5091 is free. - **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL.