#spec.sr.ht — reviewable document storage for humans and agents

Status: design / proposal — nothing implemented yet. Author: bigbes (with Claude). Last updated: 2026-07-22.

#Context

bigbes wants a third custom service on the self-hosted SourceHut instance (*.srht.bigb.es), after dolt.sr.ht (../sourcehut-dolt) and compare.sr.ht (../sourcehut-compare): separate storage for specs and documents that is reviewable by a human, modifiable by a human, and uploaded to and read by bots.

Those three verbs are not three features. They are one loop:

bot produces → human curates → bots consume.

Everything below follows from taking that loop seriously. Three properties of the loop are what a plain git repo, a wiki, or upstream man.sr.ht each fail to provide:

1. The unit of work is a proposal, not a commit. Agents produce more text than a human can read. A store where agents write freely and nobody reviews is a wiki that rots within a week. The thing you interact with daily is a review queue, not a file tree.
2. Bots need a stable read contract. "Give me the approved text of SPEC-0007", not "HEAD of main, which another agent is halfway through rewriting". Draft and canonical must be different addresses.
3. Provenance is review context. Which agent, which session, on whose behalf, against which base revision. Reviewing agent output without that is reviewing anonymous text.

Same integration model as the two siblings: one Go module, config-driven [spec.sr.ht] section, unified-login cookie, no upstream SourceHut modification. Runs at https://spec.srht.bigb.es.

#Decisions taken (user-confirmed, 2026-07-22)

#Architecture summary

One Go module sourcecraft.dev/bigbes/sr-ht-spec, one binary specsrht (plus specsrht-migrate, a brant wrapper, exactly as dolt.sr.ht does it).

   agents ──MCP / REST──►  specsrht  ──►  bare git repos   (authoritative)
                              │           <repos>/~user/<space>
   you ──git push (ssh)───────────────────────┘  ▲
                              │                  │ post-receive hook
                              ├──►  materialized checkouts (read/index cache)
                              │           <cache>/~user/<space>/
                              ├──►  bleve indexes, one per project
                              └──►  Postgres (proposals, comments, agent
                                              tokens + scopes, ID registry)

   you ──browser──────────►  read + review UI (no editing)

Note the asymmetry: agents never speak git, and you never go through the write API. Each principal has exactly one write path, which is what makes the refs rule below enforceable.

Git is authoritative for document bodies. Postgres never stores a body. That keeps git clone a complete export, keeps git log a complete provenance record offline, and keeps the blast radius of a Postgres restore small.

#Two-tier storage, and why

• Bare repo — the push target and the source of truth. One repo per space.
• Materialized checkout of the approved branch — a plain directory of markdown, rebuilt on every merge. This exists so warren's indexer and renderer can work on files (which is what they already do), and so reads never pay object-database costs. It is a cache: deletable, rebuildable from the bare repo at any time.

#Domain model

#Space

One bare git repo = one space, addressed ~<user>/<space>. The unit of ownership, ACL, and review policy. Contains markdown documents with YAML frontmatter, plus attachments.

A versioned .spec.yml at the repo root carries the space's own policy — which means policy changes are themselves reviewable:

review:
  # Single-user: the owner is the only approver, so there is no approver list
  # and no approval count. The only real knob is which paths skip the gate.
  auto_merge: [notes/**, reports/**]
schema:                       # frontmatter contract, enforced at propose time
  required: [id, title, status]
  status: [draft, review, approved, superseded]

This one key is what implements the bimodal cadence: specs/ proposals wait for you, notes/ and reports/ land immediately. Two things follow that are easy to miss:

• Auto-merged is not human-approved, and readers must be able to tell. Each merge records approval: human | policy. A bot asking for "the approved text" of a spec should be able to require human approval and get a different answer than for a firehose note. Collapsing the two would quietly launder unreviewed agent output as blessed.
• The firehose still needs a digest. Auto-merged content that never appears in any view is write-only and rots invisibly — the exact failure this service exists to prevent, just relocated. A "what landed since you last looked" feed covering policy-merged content is therefore part of the review plane, not an extra.

#Document

Markdown + YAML frontmatter, matching the second-brain / warren conventions already in use (wikilinks, type/summary/tags).

id: SPEC-0007                 # stable; NEVER changes, including on rename
title: Proposal storage model
status: draft | review | approved | superseded
supersedes: SPEC-0003
owners: [~bigbes]
tags: [storage, review]

id is the load-bearing field. Paths move; IDs do not. Cross-space links resolve by ID, comment anchors reference IDs, and bots pin to IDs. The service enforces ID uniqueness per project via a registry table.

Frontmatter is validated at propose time. A bot that omits status: gets a 422, not a silent merge. Schema validation at the door is the cheapest available defense against agent slop, and it costs almost nothing to implement.

#Proposal

A branch proposals/<id> plus a Postgres row. Bundles N document edits with a title and rationale. State machine:

draft ──► open ──► approved ──► merged
             │└──► changes-requested ──┘
             └──► rejected

#Two write paths: you push, agents propose

This falls directly out of "human edits happen via git clone" plus "single-user", and it is the cleanest rule in the whole design:

The human pushes to the approved branch. Agents may only write proposal branches. Your push is the approval — there is nobody to review it.

Enforced at the receive path, not by convention: an agent token can only update refs matching proposals/*, and only bigbes' own credentials can fast-forward the approved branch. One rule, two principals, no approval UI needed for the human half.

The consequence is a real git remote becomes a v1 requirement, which the earlier drafts did not account for. Options, cheapest first:

1. SSH push straight to the bare repo, plus a post-receive hook — the repos live on the same box you already have SSH to, so git push spec:/var/lib/spec/~bigbes/rfcs main works with zero service code. The hook is where frontmatter validation, ID-collision checks, checkout materialization and reindex happen. Rejecting a bad push is just a non-zero exit from the hook.
2. Smart HTTP via git http-backend — a subprocess, against the house style compare.sr.ht established (no shell-out), and it needs auth plumbing.
3. go-git's transport/server — pure Go and in-house, but server-side support is the weakest part of go-git and this would be net-new risk on the critical path.

Option 1 is the recommendation for v1, precisely because single-user makes it sufficient: there is no multi-tenant credential story to build, and the post-receive hook is a better place for validation than an HTTP handler anyway — it applies to every write path, including anything that bypasses the API. Option 2 or 3 only becomes necessary if the audience ever stops being one person.

Agents keep using the REST/MCP write plane; they never speak git at all. That asymmetry is deliberate — it is what makes If-Match and provenance trailers enforceable, since the service constructs every agent commit itself.

#Project

A named set of spaces, with one unified index, one search endpoint, one MCP view, and one wikilink namespace. Modeled on hub.sr.ht's project-groups-repos idea.

project: ~bigbes/+everything
spaces:
  - ~bigbes/tarantool-rfcs
  - ~bigbes/home-ops
#mounts:                      # read-only external corpora — NOT in v1, see below
#  - {type: dir, path: /Users/blikh/data/home/second-brain}

The requested meta-project — "merge all my doc work into one searchable thing" — is just a project whose membership is everything. There is no separate aggregate entity, no copying, and no sync job.

Scope note, and a live tension. "All my doc work" now means all spaces owned by this service, not everything on disk: the confirmed boundary is agent-authored specs only, with second-brain and the Confluence-synced RFCs staying independent. So the meta-project unifies what spec.sr.ht owns. The mount mechanism stays specified because it is the escape hatch if "searchable in one place" later turns out to have meant literally everything — but it is not built in v1, and the service is deliberately empty until agents fill it.

Two consequences that must be designed in from the start rather than retrofitted:

1. ACL filtering happens at query time, against a single shared index. Never build per-viewer indexes — that path ends in N indexes and a stale-permission bug. Each indexed document carries its space ID; the query layer intersects with the caller's readable spaces. Single-user makes this nearly trivial for you (you can read everything), but it is exactly how agent tokens get scoped to a subset of spaces — which is the boundary that actually matters here (see "Authorization is about agents, not people").
2. Cross-space links resolve by ID. [[SPEC-0007]] works project-wide; relative paths work within a space only. The ID registry rejects a merge that would collide two IDs inside one project.

Incremental reindex on merge: a merge to a space's approved branch reindexes only the changed documents, in every project index that includes that space.

#The three planes

#1. Read plane (anonymous-capable, cached)

GET /~user/space/path.md renders. Content negotiation gives .md raw, .json metadata+body, ?rev=<sha> pinned to an immutable revision.

Reads default to the approved revision, with a visible "draft is 3 changes ahead" affordance. This is the plane bots consume; it must be boring and pinnable. Serving drafts by default would poison every downstream agent context with unreviewed text — which is the exact failure this whole service exists to prevent.

Implementation is warren, absorbed: vault/ (scan + frontmatter), render/ (goldmark + wikilinks), index/ + search/ (bleve keyword + optional vector), linkcheck/.

#2. Write plane (agents)

PUT /api/v1/spaces/~bigbes/rfcs/docs/specs/0007-storage.md
If-Match: <base-rev-sha>
X-Proposal: <id>            # omit to open a new one

The body is the whole document. Whole-document upload, not patches — that is how agents actually work, and it makes the merge model trivial (see below). If-Match gives optimistic concurrency: two agents editing the same document cannot silently clobber each other, and the loser gets a 409 telling it to refetch and re-propose. That is exactly the right UX for a bot, which can re-derive its edit cheaply.

The write always lands on proposals/<id>, never on the approved branch.

#3. Review plane (humans)

The inbox is the product. The landing page is "N proposals waiting on you", not a file browser. A proposal page shows the prose diff, per-document, with approve / request-changes / reject, and (post-v1) inline comments and a web editor that turns a human edit into a commit on the proposal branch.

#Merge model: no text merge, ever

Verified constraint: go-git v5.19.1 implements only FastForwardMerge (repository.go:1800; anything else returns ErrUnsupportedMergeStrategy). There is no three-way merge available in-process, and shelling out to git is against the house pattern established by compare.sr.ht.

This constraint is a gift, because the whole-document grain makes a text merge unnecessary. Merging proposal P (based on B, touching file set F) into approved head H:

for f in F:
    if blob(f)@H != blob(f)@B:      # someone else changed this doc since B
        return 409 stale            # refetch and re-propose
newTree = tree(H) with each f replaced by P's blob
commit newTree with parents [H, P.head]

Pure plumbing — object.Tree manipulation plus a commit with two parents, all of which go-git supports directly. No merge algorithm, no conflict markers, no conflict-resolution UI, ever. A conflict is always "your base moved, re-propose", which is trivial for an agent and comprehensible for a human. The real merge commit keeps the proposal visible in git log.

#The two hard parts

Naming these now so they are not discovered late.

#Prose diff, not line diff

Markdown reflows. A one-word edit renders as a whole-paragraph replace under a line-oriented differ, which makes reviewing agent output miserable — and reviewing agent output is the entire product. What is needed is word-level intra-paragraph diffing over the rendered block structure, not diff --git output piped into a viewer.

This is the single UI decision that determines whether the service is pleasant or useless, which is why it is the Phase 0 gate below. Note that compare.sr.ht's @pierre/diffs bundle is a code differ and is the wrong tool here; the prose differ is likely net-new (segment into blocks → align blocks → word-diff within matched blocks).

#Comment anchoring (post-v1, but design now)

../sourcehut-compare/docs/inline-comments.md already hit this with (file, side, line) against moving refs. In prose it is worse, because line numbers are meaningless across a reflow.

Anchor to (doc id, heading path, block index, block content hash). Resolve by content hash first, fall back to heading-path + block index, and when both fail mark the comment outdated rather than silently relocating it. Anchoring to doc id rather than path is what makes comments survive renames.

#Agent identity and provenance

#Authorization is about agents, not people

Single-user does not mean "no authorization". It relocates it. There is only one human, so no human-vs-human boundary exists — but there are many agents, they are the ones actually writing, and constraining what each may touch is the whole point of the permission model.

Concretely, this deletes from v1: visibility levels (public/unlisted/private), approver lists, approval counts, request-changes round-trips, and per-human ACL rows. It keeps, and arguably sharpens: per-agent tokens scoped to a space set and a role, query-time index filtering by that scope, and the refs rule from "Two write paths". The unified-login cookie is still needed — not to tell users apart, but to tell you from an unauthenticated request.

The practical value is blast-radius control: a research agent looping over a notes space cannot touch specs/, and a compromised or confused token cannot reach the approved branch of anything.

#Provenance

Per-space (or per-project) tokens with a role — reader / proposer / writer — and a required agent identity string. Every commit records it in a way that survives clone:

Author:  claude-code/spec-writer (for bigbes) <agent@srht.bigb.es>
Committer: bigbes <bigbes@gmail.com>

    Add storage model section

    X-Agent-Session: 8fb9c9a4-b078-4af1-89eb-d97c522f9921
    X-Agent-Base: <sha>

Git trailers rather than a Postgres-only audit table, so provenance is visible in plain git log on any clone and cannot drift from the content it describes.

MCP is a first-class surface, not a wrapper — it is how agents will actually consume this: spec_search, spec_read, spec_propose, spec_comment, spec_status. warren's mcpsrv/ is the starting point.

#Reuse inventory

Genuinely net-new: proposals, prose diff, review UI, agent tokens, frontmatter lifecycle, projects/aggregation.

#SourceHut integration

Recipe B from the sourcehut-custom-service model: pure Go, API side on core-go, web chrome reimplemented (core-go has no HTML templating — the Jinja chrome lives only in Python core.sr.ht). compare.sr.ht already did exactly this, so its nav template, cookie→identity handler, and SCSS entry are the thing to copy rather than rederive.

Integration is entirely config + nginx + DNS. No upstream source is modified.

#Our section

The section name must be the literal spec.sr.ht — the .sr.ht suffix is what puts us in the nav network list (core.sr.ht/srht/app/flask.py::_network) and what other services look us up by.

[spec.sr.ht]
origin=https://spec.srht.bigb.es
; Federated into api.sr.ht from Phase 2 (read side). api.sr.ht fetches
; <api-origin>/query; omit this key and it falls back to <origin>/query.
api-origin=https://spec.srht.bigb.es
connection-string=postgresql://specsrht@localhost/spec.sr.ht?sslmode=disable
; Bare repos, service-owned: <repos>/~<user>/<space>
repos=/var/lib/spec
; Materialized checkouts + bleve project indexes. Pure cache; safe to delete.
cache=/var/cache/spec
static-dir=/usr/share/sourcehut/spec.sr.ht/static
;bind-address=127.0.0.1:5091
migrate-on-upgrade=yes

Canonical key names only — origin, api-origin, connection-string, migrate-on-upgrade are read by the shared accessors (config.GetOrigin, config.GetAPI, server.WithDefaultMiddleware). repos mirrors dolt.sr.ht/git.sr.ht; static-dir and bind-address are the local house convention already used by both siblings. cache is ours.

#Shared keys we read in place (never duplicate)

#Wiring checklist

1. Config — the [spec.sr.ht] origin= line must be visible to every other service's config, not just ours; each service builds its own nav independently. One shared /etc/sr.ht/config.ini makes this automatic — then restart the other services so they pick up the switcher entry.
2. DNS — spec.srht.bigb.es must be under the shared cookie domain (*.srht.bigb.es), or the unified-login cookie is never sent to us and every viewer looks anonymous.
3. nginx — plain proxy_pass http://127.0.0.1:5091, modeled on contrib/compare.sr.ht.conf. Note client_max_body_size needs raising if attachments are allowed.
4. internal-ipnet — same prerequisite as both siblings.
5. Migrations — specsrht-migrate, a brant wrapper, copied from doltsrht-migrate.

#GraphQL federation: read side in Phase 2, writes deferred

Federate the read side, and do it early — space, document, project, search, and read-only proposal listing join api.sr.ht's unified schema in Phase 2. Mutations stay on REST + MCP until the review model has settled.

First, a precision that kills the most tempting argument for federating: federation is not cross-service search. thistle merges schemas and routes each field to its owning service. There is no join engine and no unified index. Federation does not deliver the meta-project — that remains spec.sr.ht's own bleve index over member spaces, exactly as specified above. Do not let the gateway create the illusion that aggregation comes for free.

The reasons that actually justify it:

• One endpoint, one token. An agent working this instance already speaks GraphQL to git/todo/builds with a meta PAT. Making specs the one service needing a separate base URL and a separate auth path is friction paid by every agent ever written against it.
• hut. The house CLI is generated against SourceHut GraphQL APIs. Federating is the difference between a hut-shaped workflow being plausible and being impossible.
• Phase 5 pulls gqlgen in regardless. core-go/webhooks is GraphQL-native. Skipping GraphQL is a deferral, not a saving — and a costlier one once a schema has to be retrofitted around an established REST surface.
• The dolt precedent does not generalize. dolt.sr.ht skipped federation because its API is a chunk-store wire protocol with no sane graph to expose. Documents, spaces, proposals and comments are an ordinary CRUD graph — exactly what gqlgen and thistle are good at.

Why writes stay on REST for now, and this is a technical reason rather than scope discipline: the write plane's concurrency story is If-Match: <base-rev>, an HTTP idiom with well-defined 409 semantics that agents get right by default. Modeling base-rev as a mutation argument is perfectly doable, but it is a contract worth designing once, after the proposal state machine has stopped moving. Federation compounds this — once a type is in the gateway it is a consumed contract, so churning the proposal/review types there is expensive. Read types (space, document, project, search) are stable from the start; the review types are not, which is precisely the line drawn above.

MCP and GraphQL are not competitors here. The MCP tools call the same resolver layer, not a parallel implementation.

Opting out, if we ever do, is verified safe. api.sr.ht federates every config section ending in .sr.ht with no allow-list, pointing at api-origin (else origin) + "/query". updateSchema (api.sr.ht/main.go) logs and skips services that are offline or serve an invalid schema, and BuildSchema runs over the healthy ones only — so a non-GraphQL service costs one Unable to update service log line, not a broken gateway. It is also not a hot loop: schema refresh is SIGHUP-driven, not on a ticker (the goroutine selects on signalChan), so the fetch happens at startup and explicit reload only. dolt.sr.ht and compare.sr.ht run with exactly this property today.

Open: whether api.sr.ht is deployed on this instance at all is unconfirmed — sourcehut/sr.ht-nginx/ is the upstream mirror, not our instance config, and no api.srht.bigb.es reference exists in the tree. If the gateway is not running, Phase 2 still builds and serves /query (it is useful directly, and hut can target a service origin), but the federation payoff waits on standing it up.

#Phases

Phase 0 — de-risk gate. Two spikes, both must pass before anything else is built:

1. go-git write path in-process: create branch → commit blob → build merged tree → two-parent commit, on a real bare repo, no shell-out.
2. Prose word-diff rendered against two real revisions of an actual existing spec. If the diff does not read well, the design needs rethinking before any further code. This is the compare.sr.ht/dolt.sr.ht spike discipline applied to the riskiest assumption here.

Ordering constraint. Because the confirmed scope is a fresh silo with no read-only mounts, the store is empty until somebody fills it — a read plane shipped first would have nothing to render. The human push path therefore moves up into Phase 1: it is how content first exists, and it needs almost no service code.

Phase 1 — core + storage + your push path. Pure domain (space/doc/rev/ID validation, frontmatter parse + schema validation), bare-repo lifecycle, materialized checkouts, Postgres schema + migrations, and the SSH push + post-receive hook that validates, materializes and indexes. End state: you can git push a space and the service knows about it.

Phase 2 — read plane. warren absorbed, chrome from compare.sr.ht, unified login, ?rev= pinning, project index with query-time scope filtering, search, and MCP read tools. Plus the read-side GraphQL schema on /query and api-origin in config, federating into api.sr.ht. End state: agents can read and search everything you have pushed — useful on its own, before any review machinery exists.

Phase 3 — write plane. Proposals, If-Match concurrency, the merge model, agent tokens + scoping + provenance trailers, REST + MCP write tools. MCP tools and GraphQL resolvers share one service layer — no parallel implementations.

Phase 4 — review plane. Inbox, prose diff, approve / reject, status lifecycle, and the digest of policy-merged content (the firehose half is unreviewed by design, so it must at least be visible or it rots silently).

Phase 5 — later. Inline comments (anchoring above), webhooks and notifications (core-go/webhooks, GraphQL-native — the Phase 2 schema is the foundation), GraphQL mutations once the proposal state machine has stopped moving, vector search, read-only mounts if the boundary ever moves.

Dropped outright: the web editor (you edit via clone and push), and with it the concurrent-web-edit-vs-push conflict problem it would have created.

#Open items

• Naming. spec.sr.ht / spec.srht.bigb.es follows the named-by-function pattern of the two siblings. docs.sr.ht collides conceptually with upstream man.sr.ht.
• Project URL namespace. ~user/+project distinguishes projects from spaces (~user/space) in one character; alternatives are /projects/~user/name or reusing hub.sr.ht's own namespace.
• Port. compare.sr.ht is on 5090, dolt.sr.ht on 5306–5308. 5091 is free and is what the config block above assumes.
• MCP transport. Streamable HTTP on the same chi router (/mcp) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI.
• Mixed Russian/English search — unresolved and non-trivial. Specs here are written in both (cf. the ru-spec-style skill). bleve applies a per-field analyzer, and the English analyzer's stemmer and stopword list mangle Russian. Options: detect language per document at index time and route to ru/en analyzers on separate fields, querying both; or index a single language-neutral field and lose stemming everywhere. This directly determines whether search is actually usable, and it is not addressed anywhere above.
• Attachments and binaries. Diagrams and images in specs mean binary blobs in git: no useful diff, unbounded repo growth, and a size-cap decision. Mermaid in fenced blocks stays text and diffs properly — possibly worth preferring by convention over checked-in images.
• Agent token distribution. How a Claude Code session actually acquires a scoped token — long-lived value in the environment, or minted per session. Per-session is better for provenance and revocation but needs an issuing flow.
• Retention for the firehose half. Auto-merged notes accumulate forever by default. Whether they expire, get compacted, or are simply never deleted affects repo growth and index size, and is easier to decide now than later.
• LICENSE. Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL.