From 1e7e4b1ecbe8823c0bf45666af3c87ffecfebba2 Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 10:05:02 +0300 Subject: [PATCH] docs: correct errors found by independent review Two claims were plainly wrong and load-bearing: - post-receive cannot reject a push; its exit status is ignored once refs have moved. Validation and the refs rule move to the update hook. The hooks are also not 'zero service code': bleve is single-writer and the daemon holds the index, so both hooks RPC into the daemon, and push fails closed when it is unreachable. - hut builds endpoints from per-service origins, never from api.sr.ht, so federation buys it nothing and 'one endpoint, one token' is hollow. The decision is now a read schema at our own /query; federation is a free config line rather than a motivation. Also resolves an internal contradiction: the doc argued for a single shared index and then specified one bleve index per project. Now one global index with projects as query-time filters, and globally unique document IDs, which removes the per-project collision hole. Further corrections: merge staleness keyed by document ID rather than path, deletion and rename made human-push-only, approval expressed as a property of the branch rather than frontmatter, the proposal state machine collapsed to open/merged/rejected for a single reviewer, If-Match pinned to the approved head, and a new consistency section covering crash repair, the reconciler, per-space mutexes and unix ownership. Records warren's index as a batch full rebuild rather than incremental, and the eight questions from review that still need answers. --- docs/DESIGN.md | 385 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 296 insertions(+), 89 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 10923dbff3989fbe314ba70a9039e72ebcce5ec2..4796062a7ee2ff19a2c3fd181f6635f2843efcf0 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -41,8 +41,8 @@ modification**. Runs at `https://spec.srht.bigb.es`. | Storage substrate | **Own bare git repos, service-owned** (`/~user/`), like dolt.sr.ht owns its NBS stores. | | `../warren` | **Absorbed.** Its `vault`/`render`/`index`/`search`/`mcpsrv` packages become the read plane of this module. warren remains a standalone tool for local vaults. | | v1 scope | **Full loop, thin.** Your git push path + read plane + upload-as-proposal (REST + MCP) + minimal review page (prose diff, approve/reject) + scoped agent tokens. Inline comments come later; the web editor is dropped. | -| Aggregation | **Projects** (below) — a named set of spaces with one unified index. The "meta-project" is the degenerate case, not special machinery. | -| GraphQL federation | **Read side in Phase 2**, joining `api.sr.ht`'s unified schema; mutations stay on REST + MCP until the review model settles. Reversed an earlier "not in v1" call — see the section below for why. | +| Aggregation | **Projects** (below) — a named set of spaces, which is a *saved filter over one global index*, not a container. The "meta-project" is the degenerate case (a filter excluding nothing), not special machinery. | +| GraphQL | **Read schema at our own `/query` in Phase 2**; mutations stay on REST + MCP until the review model settles. Federation into `api.sr.ht` is a free config line, not a motivation — the earlier `hut` and "one endpoint" arguments were wrong and are retracted below. | | Relationship to existing doc homes | **New home for agent-authored specs only.** `second-brain`, the Confluence-synced RFCs, yonote and ultrapack task files stay independent and untouched. Read-only mounts stay in the model as an escape hatch but drop out of v1. | | Audience | **Single-user: bigbes plus his agents.** No other human reads or reviews. Approval collapses to triage; visibility levels, approver lists and approval counts drop from v1. | | Cadence | **Bimodal.** Specs/RFCs get careful review; research notes and reports flow through with light or automatic approval. `.spec.yml` policy carries the split. | @@ -57,10 +57,10 @@ One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht` agents ──MCP / REST──► specsrht ──► bare git repos (authoritative) │ /~user/ you ──git push (ssh)───────────────────────┘ ▲ - │ │ post-receive hook + │ │ update + post-receive hooks ├──► materialized checkouts (read/index cache) │ /~user// - ├──► bleve indexes, one per project + ├──► one global bleve index └──► Postgres (proposals, comments, agent tokens + scopes, ID registry) @@ -72,8 +72,11 @@ write API**. Each principal has exactly one write path, which is what makes the refs rule below enforceable. **Git is authoritative for document bodies. Postgres never stores a body.** -That keeps `git clone` a complete export, keeps `git log` a complete provenance -record offline, and keeps the blast radius of a Postgres restore small. +That keeps `git clone` a complete export of the *content*, keeps authorship +visible in `git log` offline, and keeps the blast radius of a Postgres restore +small. It is not a complete export of *everything*: review decisions, proposal +rationale and comments live in Postgres, so a clone gives you the documents and +who wrote them, not the review record. ### Two-tier storage, and why @@ -82,7 +85,18 @@ record offline, and keeps the blast radius of a Postgres restore small. markdown, rebuilt on every merge. This exists so warren's indexer and renderer can work on files (which is what they already do), and so reads never pay object-database costs. It is a **cache**: deletable, rebuildable from the bare - repo at any time. + repo at any time. Rebuild **atomically** — build into a temp dir and rename + into place — or readers observe a torn tree mid-rebuild. Stamp each checkout + with the rev it was built from, so staleness is detectable rather than assumed. + +**Caveat this tier does not cover:** `?rev=` pinned reads have no checkout +to read from, since only the approved head is materialized. They need a second +read path — blob → renderer, straight from the object database, with no vault +scan. warren's `vault`/`render` split is file-oriented, so this is a real if +modest rework rather than pure absorption. It is also an argument for questioning +the checkout tier altogether: reading blobs at the approved ref plus a render +cache would make pinned and default reads *the same code path* and delete this +cache subsystem entirely. Open — see "Open items". ## Domain model @@ -102,7 +116,7 @@ review: auto_merge: [notes/**, reports/**] schema: # frontmatter contract, enforced at propose time required: [id, title, status] - status: [draft, review, approved, superseded] + status: [draft, review, superseded] ``` This one key is what implements the **bimodal** cadence: `specs/` proposals wait @@ -128,7 +142,7 @@ already in use (wikilinks, `type`/`summary`/`tags`). ```yaml id: SPEC-0007 # stable; NEVER changes, including on rename title: Proposal storage model -status: draft | review | approved | superseded +status: draft | review | superseded # NOT "approved" — see below supersedes: SPEC-0003 owners: [~bigbes] tags: [storage, review] @@ -136,7 +150,15 @@ tags: [storage, review] `id` is the load-bearing field. Paths move; IDs do not. Cross-space links resolve by ID, comment anchors reference IDs, and bots pin to IDs. The service -enforces ID uniqueness per project via a registry table. +enforces **global** ID uniqueness via a registry table. + +**"Approved" is a property of the branch, not of the frontmatter.** A document +reachable from the approved ref is approved; that is the whole definition. Had +`approved` stayed in the `status` enum, either the approved branch would +permanently carry `status: draft`, or the merge would have to rewrite frontmatter +nobody authored — which would also silently invalidate the agent's `If-Match` +base on its next read. `status` stays *authored* metadata (is this a rough draft, +is it superseded); approval state is read from git. **Frontmatter is validated at propose time.** A bot that omits `status:` gets a 422, not a silent merge. Schema validation at the door is the cheapest available @@ -148,11 +170,16 @@ A branch `proposals/` plus a Postgres row. Bundles N document edits with a title and rationale. State machine: ``` -draft ──► open ──► approved ──► merged - │└──► changes-requested ──┘ - └──► rejected +open ──► merged + └───► rejected ``` +Collapsed from a five-state machine for the single-user case: with nobody else in +the loop, "approve" *is* "merge now", and there is no one to request changes +*from* — a proposal you dislike is rejected and the agent proposes again. Keeping +`approved` and `merged` as separate states, or a `changes-requested` cycle, would +have been machinery serving a review conversation that has no second party. + ### Two write paths: you push, agents propose This falls directly out of "human edits happen via `git clone`" plus @@ -169,23 +196,56 @@ human half. The consequence is **a real git remote becomes a v1 requirement**, which the earlier drafts did not account for. Options, cheapest first: -1. **SSH push straight to the bare repo, plus a `post-receive` hook** — the repos - live on the same box you already have SSH to, so - `git push spec:/var/lib/spec/~bigbes/rfcs main` works with **zero service - code**. The hook is where frontmatter validation, ID-collision checks, - checkout materialization and reindex happen. Rejecting a bad push is just a - non-zero exit from the hook. -2. **Smart HTTP** via `git http-backend` — a subprocess, against the house style - compare.sr.ht established (no shell-out), and it needs auth plumbing. +1. **SSH push straight to the bare repo, plus receive hooks** — the repos live on + the same box you already have SSH to, so + `git push spec:/var/lib/spec/~bigbes/rfcs main` needs no transport code. +2. **Smart HTTP** via `git http-backend` — needs auth plumbing we would otherwise + not write, and a second code path to the same refs. 3. **go-git's `transport/server`** — pure Go and in-house, but server-side support is the weakest part of go-git and this would be net-new risk on the critical path. -**Option 1 is the recommendation for v1**, precisely because single-user makes it -sufficient: there is no multi-tenant credential story to build, and the -`post-receive` hook is a better place for validation than an HTTP handler anyway -— it applies to *every* write path, including anything that bypasses the API. -Option 2 or 3 only becomes necessary if the audience ever stops being one person. +**Option 1 is the recommendation for v1**: single-user means there is no +multi-tenant credential story to build, and the hook applies to *every* write +path, including anything that bypasses the API. Option 2 or 3 only becomes +necessary if the audience stops being one person. + +> **Correction.** Earlier drafts of this document put validation in +> `post-receive` and claimed a non-zero exit would reject the push, at "zero +> service code". **Both claims were wrong**, and since this is the mechanism the +> whole human write path rests on, the corrected version is spelled out below. + +#### Which hook does what + +`post-receive` runs **after** refs have already been updated; its exit status is +ignored. Rejection must therefore happen earlier: + +- **`update`** (per-ref, runs before the ref moves, can reject) — enforces the + refs rule: an agent token may only touch `proposals/*`; the approved branch + accepts fast-forwards from you only, never a force-update. Also where + frontmatter and ID-collision validation rejects a bad push. +- **`post-receive`** (after the fact, cannot reject) — materialize the checkout, + notify, reindex. + +#### The hook is service code, not a shell script + +bleve is **single-writer** and the running `specsrht` daemon holds the index +open, so a separate hook process cannot reindex — and validation needs the same +schema/policy logic the API uses. Both hooks are therefore **thin shims that RPC +into the daemon** over a localhost socket, calling internal +`ValidatePush` / `Materialize` / `Reindex` entry points. + +That makes daemon availability part of the push path, so it needs a stated +behavior: **fail closed** — if the daemon is unreachable, `update` rejects the +push. A rejected push is recoverable in one command; a silently unvalidated, +unindexed one is a corruption you discover much later. (The reconciler in +"Consistency and recovery" is the backstop that repairs the other direction.) + +**Cost note:** warren's `index.Build` is a **batch full rebuild** (`bleve.New` +plus a full vault rescan). Per-document incremental upsert/delete is therefore +**net-new work, not absorption** — see the reuse inventory. Whether it is +actually needed in v1 depends on write volume; a full rebuild is fine at tens of +documents a day. Agents keep using the REST/MCP write plane; they never speak git at all. That asymmetry is deliberate — it is what makes `If-Match` and provenance trailers @@ -193,9 +253,11 @@ enforceable, since the service constructs every agent commit itself. ### Project -A named set of spaces, with **one unified index, one search endpoint, one MCP -view, and one wikilink namespace**. Modeled on hub.sr.ht's project-groups-repos -idea. +A named set of spaces sharing **one search scope, one MCP view, and one +wikilink namespace**. Modeled on hub.sr.ht's project-groups-repos idea. + +A project is **pure metadata — a saved filter, not a container.** It owns no +index and no storage. ```yaml project: ~bigbes/+everything @@ -220,19 +282,28 @@ built in v1, and the service is deliberately empty until agents fill it. Two consequences that must be designed in from the start rather than retrofitted: -1. **ACL filtering happens at query time**, against a single shared index. Never - build per-viewer indexes — that path ends in N indexes and a stale-permission - bug. Each indexed document carries its space ID; the query layer intersects - with the caller's readable spaces. Single-user makes this nearly trivial for - *you* (you can read everything), but it is exactly how **agent** tokens get - scoped to a subset of spaces — which is the boundary that actually matters - here (see "Authorization is about agents, not people"). -2. **Cross-space links resolve by ID.** `[[SPEC-0007]]` works project-wide; - relative paths work within a space only. The ID registry rejects a merge that - would collide two IDs inside one project. - -Incremental reindex on merge: a merge to a space's approved branch reindexes only -the changed documents, in every project index that includes that space. +1. **One global index, filtered at query time.** Every indexed document carries + its space ID; a project query is that index filtered to the project's spaces, + and an agent's scope is the same filter intersected with its token. + + > **Correction.** An earlier draft stated the principle "never build + > per-viewer indexes" and then specified *one bleve index per project* — with + > the `+everything` meta-project as a full second copy of the corpus. That is + > the same disease renamed: every merge fans out to N indexes and adding a + > space to a project forces a rebuild. One global index removes it, and makes + > the meta-project genuinely degenerate (a filter that excludes nothing). + +2. **Document IDs are globally unique**, not per-project. Per-project uniqueness + had a hole with no good answer: projects are edited *after* merges, so adding + a space could juxtapose two already-merged documents sharing an ID with no + merge left to reject. Single user, single instance — a global registry costs + nothing and deletes the problem. `[[SPEC-0007]]` then resolves the same way + everywhere; relative paths stay space-local. + +Reindex on merge touches **one index**, not N: a merge updates the changed +documents in the single global index, and every project that contains the space +sees the change for free. Whether that update is incremental or a full rebuild is +a volume question, not an architectural one (see the reuse inventory). ## The three planes @@ -259,6 +330,21 @@ If-Match: X-Proposal: # omit to open a new one ``` +**`If-Match` is the space's approved-head sha at the time the agent read the +document** — not the blob sha and not the proposal branch head. One value, one +meaning, identical across REST and MCP: + +- **Opening a proposal** — the value becomes the proposal's base `B`, and the + branch is cut from it. Rejected with 409 if it is not an ancestor of the + current approved head. +- **Adding to an existing proposal** — validated against `B`, which does *not* + move as the proposal accumulates edits. An agent revising its own proposal + therefore keeps sending the same value, and only a change to the *approved* + branch under it produces a 409. + +Leaving this ambiguous is how REST and MCP end up with quietly different 409 +semantics, so it is pinned here rather than in the implementation. + The body is the whole document. **Whole-document upload, not patches** — that is how agents actually work, and it makes the merge model trivial (see below). `If-Match` gives optimistic concurrency: two agents editing the same document @@ -272,25 +358,26 @@ The write always lands on `proposals/`, never on the approved branch. **The inbox is the product.** The landing page is "N proposals waiting on you", not a file browser. A proposal page shows the prose diff, per-document, with -approve / request-changes / reject, and (post-v1) inline comments and a web -editor that turns a human edit into a commit on the proposal branch. +approve (which merges immediately) or reject, and (post-v1) inline comments. +There is no request-changes cycle and no web editor: with one reviewer, changes +are made by rejecting and re-proposing, or by editing locally and pushing. ## Merge model: no text merge, ever **Verified constraint:** `go-git` v5.19.1 implements only `FastForwardMerge` (`repository.go:1800`; anything else returns `ErrUnsupportedMergeStrategy`). -There is no three-way merge available in-process, and shelling out to `git` is -against the house pattern established by compare.sr.ht. +There is no three-way merge available in-process. This constraint is a gift, because the whole-document grain makes a text merge -unnecessary. Merging proposal `P` (based on `B`, touching file set `F`) into +unnecessary — which is the real justification, standing on its own without any +appeal to a no-shell-out rule. Merging proposal `P` (based on `B`, touching file set `F`) into approved head `H`: ``` -for f in F: - if blob(f)@H != blob(f)@B: # someone else changed this doc since B - return 409 stale # refetch and re-propose -newTree = tree(H) with each f replaced by P's blob +for d in F: # F is a set of document IDs, not paths + if blob(path(d)@H) != blob(path(d)@B): # changed under us since B + return 409 stale # refetch and re-propose +newTree = tree(H) with each d's blob replaced (at its path in H) commit newTree with parents [H, P.head] ``` @@ -300,6 +387,65 @@ conflict-resolution UI, ever. A conflict is always "your base moved, re-propose" which is trivial for an agent and comprehensible for a human. The real merge commit keeps the proposal visible in `git log`. +**Staleness is keyed by document ID, not path.** An earlier draft compared +`blob(f)@H` to `blob(f)@B` by path, inside a system whose entire premise is +"paths move, IDs do not". A rename between `B` and `H` would then surface as a +baffling 409 — or worse, a proposal re-adding the old path would silently +resurrect a document that had been moved. Resolving each ID to its path *in `H`* +before comparing costs one frontmatter parse of the changed set and removes both +failure modes. + +**Deletion and rename need an explicit surface.** The write plane is +whole-document `PUT`, which gives an agent no way to express "delete this" or +"move this". Two coherent answers; **the second is the v1 recommendation**: + +1. Add `DELETE` and a move operation to the proposal API. +2. **Deletion and rename are human-push-only.** They are rare, destructive, and + trivially expressed by the person who already has a git remote and full push + rights to the approved branch. An agent that thinks a document should go + proposes `status: superseded` instead, which is reviewable and reversible. + +## Consistency and recovery + +Four systems are touched by a merge — git refs, the materialized checkout, the +bleve index, and Postgres — and **none of it is transactional**. Earlier drafts +only said "the checkout is a rebuildable cache", which covers one of the four. +The rule that makes the rest tractable: + +> **Git refs are the source of truth for whether a proposal exists and whether it +> merged. Postgres holds metadata that is reconstructable from git. The checkout +> and the index are pure caches.** + +That gives every crash a defined repair, rather than a bespoke recovery per +failure point: + +| Crash between | Symptom | Repair | +|---|---|---| +| branch write and row insert | orphan `proposals/*` ref | reconciler recreates the row from the ref | +| merge commit and row update | merged ref, row still `open` | reconciler marks merged (ref is truth) | +| merge and checkout rebuild | stale checkout | rev stamp mismatch → rebuild | +| checkout and reindex | stale index | index rev stamp mismatch → reindex | + +A **reconciler runs at startup and periodically**: scan `proposals/*` refs and +each space's approved head, compare against rows and the index rev stamps, repair +divergence. It is perhaps a hundred lines and it is what lets every other +component crash without ceremony. + +### Concurrency and ownership + +- **Two writers, one repo.** Human pushes go through native `receive-pack` + (spawned by sshd); agent proposals and merges are go-git in-process. Two + independent ref-locking implementations on the same loose refs and + `packed-refs`. go-git's locking is not verified to interoperate with native + git's, so **the daemon takes a per-space mutex** for all its git writes, and a + merge that loses a ref CAS to a concurrent human push **retries** rather than + failing. +- **Unix ownership must be stated, not assumed.** Repos are owned by the service + user; your pushes go over SSH **as the service user** with a forced command, + with identity established by SSH key rather than unix account. This keeps a + single owner on every file, and gives the forced command the hook context it + needs to apply the refs rule. + ## The two hard parts Naming these now so they are not discovered late. @@ -376,13 +522,24 @@ consume this: `spec_search`, `spec_read`, `spec_propose`, `spec_comment`, | From | What | Notes | |---|---|---| -| `../warren` | `vault/`, `render/`, `index/`, `search/`, `mcpsrv/`, `linkcheck/` | The entire read plane and bot retrieval surface, already written. Absorbed into this module. | +| `../warren` | `vault/`, `render/`, `index/`, `search/`, `mcpsrv/`, `linkcheck/` | Most of the read plane, already written. Absorbed — but see the two gaps below; this is not free. | | `../sourcehut-compare` | chrome/templates, cookie→identity, GraphQL authorizer + TTL cache, SCSS pipeline, `contrib/` nginx+systemd shape | Closest sibling; copy the integration scaffolding wholesale. | | `../sourcehut-dolt` | bare-store lifecycle under `/~user/`, brant migration wrapper, config validation | Same storage-root and migration patterns. | | `sr-ht-core` (fork) | config, crypto, auth, database, server, **gqlgen scaffolding + `webhooks`** | Pinned to `git.srht.bigb.es/~bigbes/core-go` via `replace`, as in both siblings. Never `go get -u`. The gqlgen path is the blessed one and is what Phase 2's `/query` is built on. | -Genuinely net-new: proposals, prose diff, review UI, agent tokens, frontmatter -lifecycle, projects/aggregation. +**Two absorption gaps, both understated in earlier drafts:** + +- **`index.Build` is a batch full rebuild** (`bleve.New` + full vault rescan + + chunking). There is no incremental path, so "reindex only the changed + documents" is net-new work. Whether v1 needs it is a volume question — a full + rebuild is fine at tens of documents a day, and the honest answer is to measure + before building incremental upsert/delete. +- **`vault`/`render` are file-oriented**, so `?rev=` pinned reads need a + blob→renderer path that bypasses the vault scan (see "Two-tier storage"). + +Genuinely net-new: proposals, prose diff, review plane, agent tokens and scoping, +frontmatter lifecycle, the reconciler, and incremental indexing if volume demands +it. ## SourceHut integration @@ -450,35 +607,45 @@ convention already used by both siblings. `cache` is ours. 5. **Migrations** — `specsrht-migrate`, a brant wrapper, copied from `doltsrht-migrate`. -### GraphQL federation: read side in Phase 2, writes deferred +### GraphQL: a read schema at our own `/query` in Phase 2 + +**Serve a read-side GraphQL schema — `space`, `document`, `project`, `search`, +proposal listing — at `https://spec.srht.bigb.es/query` in Phase 2. Mutations +stay on REST + MCP** until the review model has settled. Federation into +`api.sr.ht` is then a single config line (`api-origin=`) that we may as well set, +but it is **not** a reason to do any of this. -**Federate the read side, and do it early** — `space`, `document`, `project`, -`search`, and read-only proposal listing join `api.sr.ht`'s unified schema in -Phase 2. **Mutations stay on REST + MCP** until the review model has settled. +> **Correction.** An earlier draft argued for federation on the grounds of +> "one endpoint, one token" and `hut` ergonomics. **Both arguments were wrong** +> and are retracted here. -First, a precision that kills the most tempting argument *for* federating: -**federation is not cross-service search.** thistle merges schemas and routes -each field to its owning service. There is no join engine and no unified index. -Federation does **not** deliver the meta-project — that remains spec.sr.ht's own -bleve index over member spaces, exactly as specified above. -Do not let the gateway create the illusion that aggregation comes for free. +Two tempting arguments that do **not** survive checking: -The reasons that actually justify it: +- **Federation is not cross-service search.** thistle merges schemas and routes + each field to its owning service. There is no join engine and no unified index. + Federation does **not** deliver the meta-project — that remains spec.sr.ht's + own bleve index, exactly as specified above. +- **`hut` does not go through the gateway.** `hut/client.go` builds its endpoint + from the *per-service* origin (`inst.Services[service].Origin + "/query"`), not + from `api.sr.ht`. So federating buys `hut` nothing; what `hut` needs is a + `/query` at our own origin — and realistically also a fork, since its + subcommands are generated per service. For the same reason "one endpoint, one + token" is hollow: a meta PAT already validates against each service's own + `/query` directly. + +What actually justifies serving GraphQL at all: -- **One endpoint, one token.** An agent working this instance already speaks - GraphQL to git/todo/builds with a meta PAT. Making specs the one service - needing a separate base URL and a separate auth path is friction paid by every - agent ever written against it. -- **`hut`.** The house CLI is generated against SourceHut GraphQL APIs. - Federating is the difference between a `hut`-shaped workflow being plausible - and being impossible. - **Phase 5 pulls gqlgen in regardless.** `core-go/webhooks` is GraphQL-native. Skipping GraphQL is a deferral, not a saving — and a costlier one once a schema has to be retrofitted around an established REST surface. -- **The dolt precedent does not generalize.** dolt.sr.ht skipped federation - because its API *is* a chunk-store wire protocol with no sane graph to expose. - Documents, spaces, proposals and comments are an ordinary CRUD graph — exactly - what gqlgen and thistle are good at. +- **The dolt precedent does not generalize.** dolt.sr.ht skipped GraphQL because + its API *is* a chunk-store wire protocol with no sane graph to expose. + Documents, spaces, proposals and comments are an ordinary CRUD graph. +- **It is the instance-native read surface.** Anything on this instance that + already speaks SourceHut GraphQL can consume specs without a bespoke client. + +That is a decent case for a schema and a weak case for the gateway. If it turns +out `api.sr.ht` is not deployed here (see below), nothing about Phase 2 changes. **Why writes stay on REST for now**, and this is a technical reason rather than scope discipline: the write plane's concurrency story is `If-Match: `, @@ -505,9 +672,10 @@ only. dolt.sr.ht and compare.sr.ht run with exactly this property today. **Open:** whether `api.sr.ht` is deployed on this instance at all is unconfirmed — `sourcehut/sr.ht-nginx/` is the upstream mirror, not our instance config, and no -`api.srht.bigb.es` reference exists in the tree. If the gateway is not running, -Phase 2 still builds and serves `/query` (it is useful directly, and `hut` can -target a service origin), but the federation payoff waits on standing it up. +`api.srht.bigb.es` reference exists in the tree. Given the retraction above this +barely matters: Phase 2 serves `/query` at our own origin either way, and `hut` +targets that origin directly. Federation is a config line to set if the gateway +happens to exist. ## Phases @@ -529,9 +697,12 @@ code. **Phase 1 — core + storage + your push path.** Pure domain (space/doc/rev/ID validation, frontmatter parse + schema validation), bare-repo lifecycle, -materialized checkouts, Postgres schema + migrations, and the **SSH push + -`post-receive` hook** that validates, materializes and indexes. End state: you -can `git push` a space and the service knows about it. +materialized checkouts with atomic swap and rev stamps, Postgres schema + +migrations, the reconciler, and the **SSH push path**: an `update` hook that +validates and enforces the refs rule, plus `post-receive` that materializes. Both +hooks RPC into the daemon. No indexing yet — that arrives with the read plane in +Phase 2. End state: you can `git push` a space, bad pushes are rejected, and the +service knows about it. **Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified login, `?rev=` pinning, project index with query-time scope filtering, search, @@ -569,13 +740,14 @@ the concurrent-web-edit-vs-push conflict problem it would have created. - **MCP transport.** Streamable HTTP on the same chi router (`/mcp`) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI. -- **Mixed Russian/English search — unresolved and non-trivial.** Specs here are - written in both (cf. the `ru-spec-style` skill). bleve applies a **per-field - analyzer**, and the English analyzer's stemmer and stopword list mangle - Russian. Options: detect language per document at index time and route to - `ru`/`en` analyzers on separate fields, querying both; or index a single - language-neutral field and lose stemming everywhere. This directly determines - whether search is actually usable, and it is not addressed anywhere above. +- **Mixed Russian/English search — partially solved, not finished.** warren's + `search/keyword.go` already wires both the `lang/en` and `lang/ru` analyzers + with per-index selection, so the starting point is better than an earlier draft + claimed. What is still missing is **per-document** language routing: specs here + are written in both (cf. the `ru-spec-style` skill), and one analyzer per index + mangles whichever language it was not chosen for. Options: detect language at + index time and write to `ru`/`en` fields, querying both; or accept degraded + stemming on the minority language. - **Attachments and binaries.** Diagrams and images in specs mean binary blobs in git: no useful diff, unbounded repo growth, and a size-cap decision. Mermaid in fenced blocks stays text and diffs properly — possibly worth *preferring* by @@ -588,3 +760,38 @@ the concurrent-web-edit-vs-push conflict problem it would have created. affects repo growth and index size, and is easier to decide now than later. - **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL. + +### Raised by review, awaiting a decision + +Ordered by how much the answer would change the build: + +1. **Would review happen in a browser, or at a terminal?** `git fetch origin + proposals/42 && git diff` with a local word-diff is a workflow that exists + today. If terminal review is acceptable even initially, the prose-diff + renderer and most of the review plane — the largest net-new component and the + Phase 0 gate — leave v1, shrinking it to storage + hooks + read plane + + proposal API. +2. **Do agents genuinely need `?rev=` pinning and the approved/draft split?** It + is a founding premise of this document, but if every real consumer just wants + "current approved text", the pinning machinery, the dual read path, and part + of the token-scope model all simplify. +3. **Is the materialized checkout a requirement or an implementation detail?** + Dropping it for blob reads at the approved ref plus a render cache removes the + atomic-swap problem, the staleness coupling, and a whole cache tier — at the + cost of a deeper rework of warren's file-oriented scan. +4. **What is the real firehose volume?** Decides whether full-index-rebuild-on- + merge is fine (deleting the incremental-indexing work), whether retention + matters, and whether merge contention is real or theoretical. +5. **Scoped per-agent tokens in v1, or one token plus mandatory provenance?** If + every agent is a Claude Code session you launched, one token and trailers give + full provenance for a fraction of the machinery; scoping can arrive when a + genuinely autonomous agent does. +6. **Should your own pushes be validated at all, or are you trusted absolutely?** + If you can push whatever you like, the `update` hook shrinks to "no force-push + on approved" and the entire validation stack lives only in the API path, where + it is in-process and easy. +7. **Will existing corpora ever be imported or mounted?** If the honest answer is + never, delete mounts from this document entirely. +8. **Is `spec` the final name?** Low architectural leverage, but it locks the + config section, DNS, module path and nav entry on day one, and every example + here already hardcodes it.