#spec.sr.ht — reviewable document storage for humans and agents

Status: design / proposal — nothing implemented yet. Author: bigbes (with Claude). Last updated: 2026-07-22.

#Context

bigbes wants a third custom service on the self-hosted SourceHut instance (*.srht.bigb.es), after dolt.sr.ht (../sourcehut-dolt) and compare.sr.ht (../sourcehut-compare): separate storage for specs and documents that is reviewable by a human, modifiable by a human, and uploaded to and read by bots.

Those three verbs are not three features. They are one loop:

bot produces → human curates → bots consume.

Everything below follows from taking that loop seriously. Three properties of the loop are what a plain git repo, a wiki, or upstream man.sr.ht each fail to provide:

1. The unit of work is a proposal, not a commit. Agents produce more text than a human can read. A store where agents write freely and nobody reviews is a wiki that rots within a week. The thing you interact with daily is a review queue, not a file tree.
2. Bots need a stable read contract. "Give me the approved text of SPEC-0007", not "HEAD of main, which another agent is halfway through rewriting". Draft and canonical must be different addresses.
3. Provenance is review context. Which agent, which session, on whose behalf, against which base revision. Reviewing agent output without that is reviewing anonymous text.

Same integration model as the two siblings: one Go module, config-driven [spec.sr.ht] section, unified-login cookie, no upstream SourceHut modification. Runs at https://spec.srht.bigb.es.

#Decisions taken (user-confirmed, 2026-07-22)

#Architecture summary

One Go module sourcecraft.dev/bigbes/sr-ht-spec, one binary specsrht (plus specsrht-migrate, a brant wrapper, exactly as dolt.sr.ht does it).

   agents ──MCP / REST──┐
                        ├──►  specsrht  ──►  bare git repos   (authoritative)
   humans ──web UI──────┘        │           <repos>/~user/<space>
                                 │
                                 ├──►  materialized checkouts (read/index cache)
                                 │           <cache>/~user/<space>/
                                 ├──►  bleve indexes, one per project
                                 └──►  Postgres (proposals, comments, ACLs,
                                                 agent tokens, ID registry)

Git is authoritative for document bodies. Postgres never stores a body. That keeps git clone a complete export, keeps git log a complete provenance record offline, and keeps the blast radius of a Postgres restore small.

#Two-tier storage, and why

• Bare repo — the push target and the source of truth. One repo per space.
• Materialized checkout of the approved branch — a plain directory of markdown, rebuilt on every merge. This exists so warren's indexer and renderer can work on files (which is what they already do), and so reads never pay object-database costs. It is a cache: deletable, rebuildable from the bare repo at any time.

#Domain model

#Space

One bare git repo = one space, addressed ~<user>/<space>. The unit of ownership, ACL, and review policy. Contains markdown documents with YAML frontmatter, plus attachments.

A versioned .spec.yml at the repo root carries the space's own policy — which means policy changes are themselves reviewable:

review:
  required: 1                 # approvals needed to merge
  approvers: [~bigbes]
  auto_merge: [notes/**]      # paths that skip the gate
schema:                       # frontmatter contract, enforced at propose time
  required: [id, title, status, owners]
  status: [draft, review, approved, superseded]

#Document

Markdown + YAML frontmatter, matching the second-brain / warren conventions already in use (wikilinks, type/summary/tags).

id: SPEC-0007                 # stable; NEVER changes, including on rename
title: Proposal storage model
status: draft | review | approved | superseded
supersedes: SPEC-0003
owners: [~bigbes]
tags: [storage, review]

id is the load-bearing field. Paths move; IDs do not. Cross-space links resolve by ID, comment anchors reference IDs, and bots pin to IDs. The service enforces ID uniqueness per project via a registry table.

Frontmatter is validated at propose time. A bot that omits status: gets a 422, not a silent merge. Schema validation at the door is the cheapest available defense against agent slop, and it costs almost nothing to implement.

#Proposal

A branch proposals/<id> plus a Postgres row. Bundles N document edits with a title and rationale. State machine:

draft ──► open ──► approved ──► merged
             │└──► changes-requested ──┘
             └──► rejected

#Project

A named set of spaces plus read-only external mounts, with one unified index, one search endpoint, one MCP view, and one wikilink namespace. Modeled on hub.sr.ht's project-groups-repos idea.

project: ~bigbes/+everything
spaces:
  - ~bigbes/tarantool-rfcs
  - ~bigbes/home-ops
mounts:                       # read-only, indexed, searchable, not proposable
  - {type: dir,  path: /Users/blikh/data/home/second-brain}
  - {type: gitsrht, repo: ~bigbes/warren, subdir: docs/}

The requested meta-project — "merge all my doc work into one searchable thing" — is just a project whose membership is everything. There is no separate aggregate entity, no copying, and no sync job. Three consequences that must be designed in from the start rather than retrofitted:

1. ACL filtering happens at query time, against a single shared index. Never build per-user indexes — that path ends in N indexes and a stale-permission bug. Each indexed document carries its space ID; the query layer intersects with the viewer's readable spaces. An agent token scoped to a project resolves to the intersection of the spaces it may read.
2. Cross-space links resolve by ID. [[SPEC-0007]] works project-wide; relative paths work within a space only. The ID registry rejects a merge that would collide two IDs inside one project.
3. Read-only mounts are first class. second-brain, docs/ directories of existing git.sr.ht repos, man.sr.ht wikis — all searchable without migrating anything anywhere. This is likely the fastest route to the service being useful on day one, since it needs no content to be moved into it.

Incremental reindex on merge: a merge to a space's approved branch reindexes only the changed documents, in every project index that includes that space.

#The three planes

#1. Read plane (anonymous-capable, cached)

GET /~user/space/path.md renders. Content negotiation gives .md raw, .json metadata+body, ?rev=<sha> pinned to an immutable revision.

Reads default to the approved revision, with a visible "draft is 3 changes ahead" affordance. This is the plane bots consume; it must be boring and pinnable. Serving drafts by default would poison every downstream agent context with unreviewed text — which is the exact failure this whole service exists to prevent.

Implementation is warren, absorbed: vault/ (scan + frontmatter), render/ (goldmark + wikilinks), index/ + search/ (bleve keyword + optional vector), linkcheck/.

#2. Write plane (agents)

PUT /api/v1/spaces/~bigbes/rfcs/docs/specs/0007-storage.md
If-Match: <base-rev-sha>
X-Proposal: <id>            # omit to open a new one

The body is the whole document. Whole-document upload, not patches — that is how agents actually work, and it makes the merge model trivial (see below). If-Match gives optimistic concurrency: two agents editing the same document cannot silently clobber each other, and the loser gets a 409 telling it to refetch and re-propose. That is exactly the right UX for a bot, which can re-derive its edit cheaply.

The write always lands on proposals/<id>, never on the approved branch.

#3. Review plane (humans)

The inbox is the product. The landing page is "N proposals waiting on you", not a file browser. A proposal page shows the prose diff, per-document, with approve / request-changes / reject, and (post-v1) inline comments and a web editor that turns a human edit into a commit on the proposal branch.

#Merge model: no text merge, ever

Verified constraint: go-git v5.19.1 implements only FastForwardMerge (repository.go:1800; anything else returns ErrUnsupportedMergeStrategy). There is no three-way merge available in-process, and shelling out to git is against the house pattern established by compare.sr.ht.

This constraint is a gift, because the whole-document grain makes a text merge unnecessary. Merging proposal P (based on B, touching file set F) into approved head H:

for f in F:
    if blob(f)@H != blob(f)@B:      # someone else changed this doc since B
        return 409 stale            # refetch and re-propose
newTree = tree(H) with each f replaced by P's blob
commit newTree with parents [H, P.head]

Pure plumbing — object.Tree manipulation plus a commit with two parents, all of which go-git supports directly. No merge algorithm, no conflict markers, no conflict-resolution UI, ever. A conflict is always "your base moved, re-propose", which is trivial for an agent and comprehensible for a human. The real merge commit keeps the proposal visible in git log.

#The two hard parts

Naming these now so they are not discovered late.

#Prose diff, not line diff

Markdown reflows. A one-word edit renders as a whole-paragraph replace under a line-oriented differ, which makes reviewing agent output miserable — and reviewing agent output is the entire product. What is needed is word-level intra-paragraph diffing over the rendered block structure, not diff --git output piped into a viewer.

This is the single UI decision that determines whether the service is pleasant or useless, which is why it is the Phase 0 gate below. Note that compare.sr.ht's @pierre/diffs bundle is a code differ and is the wrong tool here; the prose differ is likely net-new (segment into blocks → align blocks → word-diff within matched blocks).

#Comment anchoring (post-v1, but design now)

../sourcehut-compare/docs/inline-comments.md already hit this with (file, side, line) against moving refs. In prose it is worse, because line numbers are meaningless across a reflow.

Anchor to (doc id, heading path, block index, block content hash). Resolve by content hash first, fall back to heading-path + block index, and when both fail mark the comment outdated rather than silently relocating it. Anchoring to doc id rather than path is what makes comments survive renames.

#Agent identity and provenance

Per-space (or per-project) tokens with a role — reader / proposer / writer — and a required agent identity string. Every commit records it in a way that survives clone:

Author:  claude-code/spec-writer (for bigbes) <agent@srht.bigb.es>
Committer: bigbes <bigbes@gmail.com>

    Add storage model section

    X-Agent-Session: 8fb9c9a4-b078-4af1-89eb-d97c522f9921
    X-Agent-Base: <sha>

Git trailers rather than a Postgres-only audit table, so provenance is visible in plain git log on any clone and cannot drift from the content it describes.

MCP is a first-class surface, not a wrapper — it is how agents will actually consume this: spec_search, spec_read, spec_propose, spec_comment, spec_status. warren's mcpsrv/ is the starting point.

#Reuse inventory

Genuinely net-new: proposals, prose diff, review UI, agent tokens, frontmatter lifecycle, projects/aggregation.

#SourceHut integration

Recipe B from the sourcehut-custom-service model: pure Go, API side on core-go, web chrome reimplemented (core-go has no HTML templating — the Jinja chrome lives only in Python core.sr.ht). compare.sr.ht already did exactly this, so its nav template, cookie→identity handler, and SCSS entry are the thing to copy rather than rederive.

Integration is entirely config + nginx + DNS. No upstream source is modified.

#Our section

The section name must be the literal spec.sr.ht — the .sr.ht suffix is what puts us in the nav network list (core.sr.ht/srht/app/flask.py::_network) and what other services look us up by.

[spec.sr.ht]
origin=https://spec.srht.bigb.es
; Federated into api.sr.ht from Phase 2 (read side). api.sr.ht fetches
; <api-origin>/query; omit this key and it falls back to <origin>/query.
api-origin=https://spec.srht.bigb.es
connection-string=postgresql://specsrht@localhost/spec.sr.ht?sslmode=disable
; Bare repos, service-owned: <repos>/~<user>/<space>
repos=/var/lib/spec
; Materialized checkouts + bleve project indexes. Pure cache; safe to delete.
cache=/var/cache/spec
static-dir=/usr/share/sourcehut/spec.sr.ht/static
;bind-address=127.0.0.1:5091
migrate-on-upgrade=yes

Canonical key names only — origin, api-origin, connection-string, migrate-on-upgrade are read by the shared accessors (config.GetOrigin, config.GetAPI, server.WithDefaultMiddleware). repos mirrors dolt.sr.ht/git.sr.ht; static-dir and bind-address are the local house convention already used by both siblings. cache is ours.

#Shared keys we read in place (never duplicate)

#Wiring checklist

1. Config — the [spec.sr.ht] origin= line must be visible to every other service's config, not just ours; each service builds its own nav independently. One shared /etc/sr.ht/config.ini makes this automatic — then restart the other services so they pick up the switcher entry.
2. DNS — spec.srht.bigb.es must be under the shared cookie domain (*.srht.bigb.es), or the unified-login cookie is never sent to us and every viewer looks anonymous.
3. nginx — plain proxy_pass http://127.0.0.1:5091, modeled on contrib/compare.sr.ht.conf. Note client_max_body_size needs raising if attachments are allowed.
4. internal-ipnet — same prerequisite as both siblings.
5. Migrations — specsrht-migrate, a brant wrapper, copied from doltsrht-migrate.

#GraphQL federation: read side in Phase 2, writes deferred

Federate the read side, and do it early — space, document, project, search, and read-only proposal listing join api.sr.ht's unified schema in Phase 2. Mutations stay on REST + MCP until the review model has settled.

First, a precision that kills the most tempting argument for federating: federation is not cross-service search. thistle merges schemas and routes each field to its owning service. There is no join engine and no unified index. Federation does not deliver the meta-project — that remains spec.sr.ht's own bleve index over member spaces plus read-only mounts, exactly as specified above. Do not let the gateway create the illusion that aggregation comes for free.

The reasons that actually justify it:

• One endpoint, one token. An agent working this instance already speaks GraphQL to git/todo/builds with a meta PAT. Making specs the one service needing a separate base URL and a separate auth path is friction paid by every agent ever written against it.
• hut. The house CLI is generated against SourceHut GraphQL APIs. Federating is the difference between a hut-shaped workflow being plausible and being impossible.
• Phase 5 pulls gqlgen in regardless. core-go/webhooks is GraphQL-native. Skipping GraphQL is a deferral, not a saving — and a costlier one once a schema has to be retrofitted around an established REST surface.
• The dolt precedent does not generalize. dolt.sr.ht skipped federation because its API is a chunk-store wire protocol with no sane graph to expose. Documents, spaces, proposals and comments are an ordinary CRUD graph — exactly what gqlgen and thistle are good at.

Why writes stay on REST for now, and this is a technical reason rather than scope discipline: the write plane's concurrency story is If-Match: <base-rev>, an HTTP idiom with well-defined 409 semantics that agents get right by default. Modeling base-rev as a mutation argument is perfectly doable, but it is a contract worth designing once, after the proposal state machine has stopped moving. Federation compounds this — once a type is in the gateway it is a consumed contract, so churning the proposal/review types there is expensive. Read types (space, document, project, search) are stable from the start; the review types are not, which is precisely the line drawn above.

MCP and GraphQL are not competitors here. The MCP tools call the same resolver layer, not a parallel implementation.

Opting out, if we ever do, is verified safe. api.sr.ht federates every config section ending in .sr.ht with no allow-list, pointing at api-origin (else origin) + "/query". updateSchema (api.sr.ht/main.go) logs and skips services that are offline or serve an invalid schema, and BuildSchema runs over the healthy ones only — so a non-GraphQL service costs one Unable to update service log line, not a broken gateway. It is also not a hot loop: schema refresh is SIGHUP-driven, not on a ticker (the goroutine selects on signalChan), so the fetch happens at startup and explicit reload only. dolt.sr.ht and compare.sr.ht run with exactly this property today.

Open: whether api.sr.ht is deployed on this instance at all is unconfirmed — sourcehut/sr.ht-nginx/ is the upstream mirror, not our instance config, and no api.srht.bigb.es reference exists in the tree. If the gateway is not running, Phase 2 still builds and serves /query (it is useful directly, and hut can target a service origin), but the federation payoff waits on standing it up.

#Phases

Phase 0 — de-risk gate. Two spikes, both must pass before anything else is built:

1. go-git write path in-process: create branch → commit blob → build merged tree → two-parent commit, on a real bare repo, no shell-out.
2. Prose word-diff rendered against two real revisions of an actual existing spec. If the diff does not read well, the design needs rethinking before any further code. This is the compare.sr.ht/dolt.sr.ht spike discipline applied to the riskiest assumption here.

Phase 1 — core + storage. Pure domain (space/doc/rev/ID validation, frontmatter parse + schema validation), bare-repo lifecycle, materialized checkouts, Postgres schema + migrations.

Phase 2 — read plane. warren absorbed, chrome from compare.sr.ht, unified login, ACLs, ?rev= pinning, read-only mounts, project index with query-time ACL filtering. Plus the read-side GraphQL schema on /query (space, document, project, search, proposal listing) and api-origin in config, federating into api.sr.ht.

Phase 3 — write plane. Proposals, If-Match concurrency, the merge model, agent tokens + provenance trailers, REST + MCP. MCP tools and GraphQL resolvers share one service layer — no parallel implementations.

Phase 4 — review UI. Inbox, prose diff, approve / request-changes, status lifecycle transitions.

Phase 5 — later. Inline comments (anchoring above), web editor, webhooks and notifications (core-go/webhooks, GraphQL-native — the schema from Phase 2 is the foundation), GraphQL mutations once the proposal state machine has stopped moving, vector search over the project index.

#Open items

• Naming. spec.sr.ht / spec.srht.bigb.es follows the named-by-function pattern of the two siblings. docs.sr.ht collides conceptually with upstream man.sr.ht.
• Project URL namespace. ~user/+project distinguishes projects from spaces (~user/space) in one character; alternatives are /projects/~user/name or reusing hub.sr.ht's own namespace.
• Port. compare.sr.ht is on 5090, dolt.sr.ht on 5306–5308. 5091 is free and is what the config block above assumes.
• MCP transport. Streamable HTTP on the same chi router (/mcp) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI.
• LICENSE. Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL.