From fc49924382cf83716bc6ae929578363dd3fe7f1a Mon Sep 17 00:00:00 2001 From: bigbes Date: Wed, 22 Jul 2026 04:29:39 +0300 Subject: [PATCH] docs: fold in scope, audience, cadence and edit-path decisions Four confirmed answers reshape the design rather than just the backlog: - Agent-authored specs only. Read-only mounts leave v1; the meta-project now unifies owned spaces. The store is a fresh silo, so it is empty until filled, which reorders the phases. - Single-user. Visibility levels, approver lists, approval counts and request-changes cycles drop. Authorization relocates rather than disappearing: it is about scoping agents, not separating humans. - Bimodal cadence. Records approval as human or policy so auto-merged notes are not laundered as reviewed, and adds a digest so the firehose half stays visible. - Human edits via clone and push. Adds the git remote as a v1 requirement and drops the web editor. Humans push to the approved branch, agents may only write proposals/*; SSH plus a post-receive hook needs no service code and validates every write path. Also records mixed ru/en search analyzers as an unresolved question that determines whether search is usable at all. --- docs/DESIGN.md | 211 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 165 insertions(+), 46 deletions(-) diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 372bb1c016e58bccfc316d20f129c0a80695b573..10923dbff3989fbe314ba70a9039e72ebcce5ec2 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -40,9 +40,13 @@ modification**. Runs at `https://spec.srht.bigb.es`. | Review gate | **Proposal-first.** Bots always write to a proposal branch; a human approves before anything reaches the approved revision. Per-space policy may whitelist auto-merge namespaces. | | Storage substrate | **Own bare git repos, service-owned** (`/~user/`), like dolt.sr.ht owns its NBS stores. | | `../warren` | **Absorbed.** Its `vault`/`render`/`index`/`search`/`mcpsrv` packages become the read plane of this module. warren remains a standalone tool for local vaults. | -| v1 scope | **Full loop, thin.** Read plane + upload-as-proposal (REST + MCP) + minimal review page (prose diff, approve/reject) + agent tokens. Inline comments and the web editor come later. | +| v1 scope | **Full loop, thin.** Your git push path + read plane + upload-as-proposal (REST + MCP) + minimal review page (prose diff, approve/reject) + scoped agent tokens. Inline comments come later; the web editor is dropped. | | Aggregation | **Projects** (below) — a named set of spaces with one unified index. The "meta-project" is the degenerate case, not special machinery. | | GraphQL federation | **Read side in Phase 2**, joining `api.sr.ht`'s unified schema; mutations stay on REST + MCP until the review model settles. Reversed an earlier "not in v1" call — see the section below for why. | +| Relationship to existing doc homes | **New home for agent-authored specs only.** `second-brain`, the Confluence-synced RFCs, yonote and ultrapack task files stay independent and untouched. Read-only mounts stay in the model as an escape hatch but drop out of v1. | +| Audience | **Single-user: bigbes plus his agents.** No other human reads or reviews. Approval collapses to triage; visibility levels, approver lists and approval counts drop from v1. | +| Cadence | **Bimodal.** Specs/RFCs get careful review; research notes and reports flow through with light or automatic approval. `.spec.yml` policy carries the split. | +| Human edit path | **`git clone`, edit locally, push.** The web editor is not v1 and may never be. This makes a real git remote a v1 requirement — see "Two write paths" below. | ## Architecture summary @@ -50,17 +54,23 @@ One Go module **`sourcecraft.dev/bigbes/sr-ht-spec`**, one binary `specsrht` (plus `specsrht-migrate`, a brant wrapper, exactly as dolt.sr.ht does it). ``` - agents ──MCP / REST──┐ - ├──► specsrht ──► bare git repos (authoritative) - humans ──web UI──────┘ │ /~user/ - │ - ├──► materialized checkouts (read/index cache) - │ /~user// - ├──► bleve indexes, one per project - └──► Postgres (proposals, comments, ACLs, - agent tokens, ID registry) + agents ──MCP / REST──► specsrht ──► bare git repos (authoritative) + │ /~user/ + you ──git push (ssh)───────────────────────┘ ▲ + │ │ post-receive hook + ├──► materialized checkouts (read/index cache) + │ /~user// + ├──► bleve indexes, one per project + └──► Postgres (proposals, comments, agent + tokens + scopes, ID registry) + + you ──browser──────────► read + review UI (no editing) ``` +Note the asymmetry: **agents never speak git**, and **you never go through the +write API**. Each principal has exactly one write path, which is what makes the +refs rule below enforceable. + **Git is authoritative for document bodies. Postgres never stores a body.** That keeps `git clone` a complete export, keeps `git log` a complete provenance record offline, and keeps the blast radius of a Postgres restore small. @@ -87,14 +97,29 @@ means policy changes are themselves reviewable: ```yaml review: - required: 1 # approvals needed to merge - approvers: [~bigbes] - auto_merge: [notes/**] # paths that skip the gate + # Single-user: the owner is the only approver, so there is no approver list + # and no approval count. The only real knob is which paths skip the gate. + auto_merge: [notes/**, reports/**] schema: # frontmatter contract, enforced at propose time - required: [id, title, status, owners] + required: [id, title, status] status: [draft, review, approved, superseded] ``` +This one key is what implements the **bimodal** cadence: `specs/` proposals wait +for you, `notes/` and `reports/` land immediately. Two things follow that are +easy to miss: + +- **Auto-merged is not human-approved, and readers must be able to tell.** Each + merge records `approval: human | policy`. A bot asking for "the approved text" + of a spec should be able to require human approval and get a different answer + than for a firehose note. Collapsing the two would quietly launder unreviewed + agent output as blessed. +- **The firehose still needs a digest.** Auto-merged content that never appears + in any view is write-only and rots invisibly — the exact failure this service + exists to prevent, just relocated. A "what landed since you last looked" feed + covering policy-merged content is therefore part of the review plane, not an + extra. + ### Document Markdown + YAML frontmatter, matching the `second-brain` / warren conventions @@ -128,39 +153,83 @@ draft ──► open ──► approved ──► merged └──► rejected ``` +### Two write paths: you push, agents propose + +This falls directly out of "human edits happen via `git clone`" plus +"single-user", and it is the cleanest rule in the whole design: + +> **The human pushes to the approved branch. Agents may only write proposal +> branches.** Your push *is* the approval — there is nobody to review it. + +Enforced at the receive path, not by convention: an agent token can only update +refs matching `proposals/*`, and only bigbes' own credentials can fast-forward +the approved branch. One rule, two principals, no approval UI needed for the +human half. + +The consequence is **a real git remote becomes a v1 requirement**, which the +earlier drafts did not account for. Options, cheapest first: + +1. **SSH push straight to the bare repo, plus a `post-receive` hook** — the repos + live on the same box you already have SSH to, so + `git push spec:/var/lib/spec/~bigbes/rfcs main` works with **zero service + code**. The hook is where frontmatter validation, ID-collision checks, + checkout materialization and reindex happen. Rejecting a bad push is just a + non-zero exit from the hook. +2. **Smart HTTP** via `git http-backend` — a subprocess, against the house style + compare.sr.ht established (no shell-out), and it needs auth plumbing. +3. **go-git's `transport/server`** — pure Go and in-house, but server-side + support is the weakest part of go-git and this would be net-new risk on the + critical path. + +**Option 1 is the recommendation for v1**, precisely because single-user makes it +sufficient: there is no multi-tenant credential story to build, and the +`post-receive` hook is a better place for validation than an HTTP handler anyway +— it applies to *every* write path, including anything that bypasses the API. +Option 2 or 3 only becomes necessary if the audience ever stops being one person. + +Agents keep using the REST/MCP write plane; they never speak git at all. That +asymmetry is deliberate — it is what makes `If-Match` and provenance trailers +enforceable, since the service constructs every agent commit itself. + ### Project -A named set of spaces plus read-only external mounts, with **one unified index, -one search endpoint, one MCP view, and one wikilink namespace**. Modeled on -hub.sr.ht's project-groups-repos idea. +A named set of spaces, with **one unified index, one search endpoint, one MCP +view, and one wikilink namespace**. Modeled on hub.sr.ht's project-groups-repos +idea. ```yaml project: ~bigbes/+everything spaces: - ~bigbes/tarantool-rfcs - ~bigbes/home-ops -mounts: # read-only, indexed, searchable, not proposable - - {type: dir, path: /Users/blikh/data/home/second-brain} - - {type: gitsrht, repo: ~bigbes/warren, subdir: docs/} +#mounts: # read-only external corpora — NOT in v1, see below +# - {type: dir, path: /Users/blikh/data/home/second-brain} ``` The requested **meta-project — "merge all my doc work into one searchable thing" — is just a project whose membership is everything.** There is no -separate aggregate entity, no copying, and no sync job. Three consequences that -must be designed in from the start rather than retrofitted: +separate aggregate entity, no copying, and no sync job. + +**Scope note, and a live tension.** "All my doc work" now means *all spaces owned +by this service*, not everything on disk: the confirmed boundary is +agent-authored specs only, with `second-brain` and the Confluence-synced RFCs +staying independent. So the meta-project unifies what spec.sr.ht owns. The mount +mechanism stays specified because it is the escape hatch if "searchable in one +place" later turns out to have meant *literally* everything — but it is not +built in v1, and the service is deliberately empty until agents fill it. + +Two consequences that must be designed in from the start rather than retrofitted: 1. **ACL filtering happens at query time**, against a single shared index. Never - build per-user indexes — that path ends in N indexes and a stale-permission + build per-viewer indexes — that path ends in N indexes and a stale-permission bug. Each indexed document carries its space ID; the query layer intersects - with the viewer's readable spaces. An agent token scoped to a project resolves - to the intersection of the spaces it may read. + with the caller's readable spaces. Single-user makes this nearly trivial for + *you* (you can read everything), but it is exactly how **agent** tokens get + scoped to a subset of spaces — which is the boundary that actually matters + here (see "Authorization is about agents, not people"). 2. **Cross-space links resolve by ID.** `[[SPEC-0007]]` works project-wide; relative paths work within a space only. The ID registry rejects a merge that would collide two IDs inside one project. -3. **Read-only mounts are first class.** `second-brain`, `docs/` directories of - existing git.sr.ht repos, man.sr.ht wikis — all searchable without migrating - anything anywhere. This is likely the fastest route to the service being - useful on day one, since it needs no content to be moved into it. Incremental reindex on merge: a merge to a space's approved branch reindexes only the changed documents, in every project index that includes that space. @@ -262,6 +331,26 @@ mark the comment **outdated** rather than silently relocating it. Anchoring to ## Agent identity and provenance +### Authorization is about agents, not people + +Single-user does **not** mean "no authorization". It relocates it. There is only +one human, so no human-vs-human boundary exists — but there are many agents, they +are the ones actually writing, and constraining what each may touch is the whole +point of the permission model. + +Concretely, this deletes from v1: visibility levels (public/unlisted/private), +approver lists, approval counts, request-changes round-trips, and per-human ACL +rows. It keeps, and arguably sharpens: **per-agent tokens scoped to a space set +and a role**, query-time index filtering by that scope, and the refs rule from +"Two write paths". The unified-login cookie is still needed — not to tell users +apart, but to tell *you* from an unauthenticated request. + +The practical value is blast-radius control: a research agent looping over a +notes space cannot touch `specs/`, and a compromised or confused token cannot +reach the approved branch of anything. + +### Provenance + Per-space (or per-project) tokens with a role — `reader` / `proposer` / `writer` — and a **required agent identity string**. Every commit records it in a way that survives clone: @@ -343,7 +432,7 @@ convention already used by both siblings. `cache` is ours. | `[sr.ht] internal-ipnet` | this host must fall inside it or internal GraphQL calls are rejected | | `[webhooks] private-key` | **`crypto.InitCrypto` fatally requires it even though v1 emits no webhooks** | | `[meta.sr.ht] origin` | login/logout redirects, profile fetch, PAT validation | -| `[git.sr.ht] repos` / `api-origin` | only for read-only mounts of `docs/` dirs in existing git.sr.ht repos | +| `[git.sr.ht] repos` / `api-origin` | only if read-only mounts of `docs/` dirs in git.sr.ht repos are ever enabled (not v1) | ### Wiring checklist @@ -371,7 +460,7 @@ First, a precision that kills the most tempting argument *for* federating: **federation is not cross-service search.** thistle merges schemas and routes each field to its owning service. There is no join engine and no unified index. Federation does **not** deliver the meta-project — that remains spec.sr.ht's own -bleve index over member spaces plus read-only mounts, exactly as specified above. +bleve index over member spaces, exactly as specified above. Do not let the gateway create the illusion that aggregation comes for free. The reasons that actually justify it: @@ -432,27 +521,40 @@ built: further code.** This is the compare.sr.ht/dolt.sr.ht spike discipline applied to the riskiest assumption here. -**Phase 1 — core + storage.** Pure domain (space/doc/rev/ID validation, -frontmatter parse + schema validation), bare-repo lifecycle, materialized -checkouts, Postgres schema + migrations. +**Ordering constraint.** Because the confirmed scope is a *fresh silo* with no +read-only mounts, the store is **empty until somebody fills it** — a read plane +shipped first would have nothing to render. The human push path therefore moves +up into Phase 1: it is how content first exists, and it needs almost no service +code. + +**Phase 1 — core + storage + your push path.** Pure domain (space/doc/rev/ID +validation, frontmatter parse + schema validation), bare-repo lifecycle, +materialized checkouts, Postgres schema + migrations, and the **SSH push + +`post-receive` hook** that validates, materializes and indexes. End state: you +can `git push` a space and the service knows about it. **Phase 2 — read plane.** warren absorbed, chrome from compare.sr.ht, unified -login, ACLs, `?rev=` pinning, read-only mounts, project index with query-time ACL -filtering. Plus the **read-side GraphQL schema on `/query`** (space, document, -project, search, proposal listing) and `api-origin` in config, federating into -`api.sr.ht`. +login, `?rev=` pinning, project index with query-time scope filtering, search, +and **MCP read tools**. Plus the **read-side GraphQL schema on `/query`** and +`api-origin` in config, federating into `api.sr.ht`. End state: agents can read +and search everything you have pushed — useful on its own, before any review +machinery exists. **Phase 3 — write plane.** Proposals, `If-Match` concurrency, the merge model, -agent tokens + provenance trailers, REST + MCP. MCP tools and GraphQL resolvers -share one service layer — no parallel implementations. +agent tokens + scoping + provenance trailers, REST + MCP write tools. MCP tools +and GraphQL resolvers share one service layer — no parallel implementations. + +**Phase 4 — review plane.** Inbox, prose diff, approve / reject, status +lifecycle, and the **digest of policy-merged content** (the firehose half is +unreviewed by design, so it must at least be *visible* or it rots silently). -**Phase 4 — review UI.** Inbox, prose diff, approve / request-changes, status -lifecycle transitions. +**Phase 5 — later.** Inline comments (anchoring above), webhooks and +notifications (`core-go/webhooks`, GraphQL-native — the Phase 2 schema is the +foundation), GraphQL **mutations** once the proposal state machine has stopped +moving, vector search, read-only mounts if the boundary ever moves. -**Phase 5 — later.** Inline comments (anchoring above), web editor, webhooks and -notifications (`core-go/webhooks`, GraphQL-native — the schema from Phase 2 is -the foundation), GraphQL **mutations** once the proposal state machine has -stopped moving, vector search over the project index. +**Dropped outright:** the web editor (you edit via clone and push), and with it +the concurrent-web-edit-vs-push conflict problem it would have created. ## Open items @@ -467,5 +569,22 @@ stopped moving, vector search over the project index. - **MCP transport.** Streamable HTTP on the same chi router (`/mcp`) keeps it to one listener and one nginx block; a second port is only needed if MCP ends up wanting different timeouts than the web UI. +- **Mixed Russian/English search — unresolved and non-trivial.** Specs here are + written in both (cf. the `ru-spec-style` skill). bleve applies a **per-field + analyzer**, and the English analyzer's stemmer and stopword list mangle + Russian. Options: detect language per document at index time and route to + `ru`/`en` analyzers on separate fields, querying both; or index a single + language-neutral field and lose stemming everywhere. This directly determines + whether search is actually usable, and it is not addressed anywhere above. +- **Attachments and binaries.** Diagrams and images in specs mean binary blobs in + git: no useful diff, unbounded repo growth, and a size-cap decision. Mermaid in + fenced blocks stays text and diffs properly — possibly worth *preferring* by + convention over checked-in images. +- **Agent token distribution.** How a Claude Code session actually acquires a + scoped token — long-lived value in the environment, or minted per session. + Per-session is better for provenance and revocation but needs an issuing flow. +- **Retention for the firehose half.** Auto-merged notes accumulate forever by + default. Whether they expire, get compacted, or are simply never deleted + affects repo growth and index size, and is easier to decide now than later. - **LICENSE.** Unchosen, same as compare.sr.ht. SourceHut's own services are AGPL/GPL.