fix: anchor installed data paths to ASSETS, and install schema.sql
Two install-time bugs that only show up on a packaged deploy, never in a
checkout.
MIGRATIONDIR derived from PREFIX, so a default build installed migrations
to /usr/local/share/sourcehut/migrations while specsrht-migrate looked
under [sr.ht]assets, default /usr/share/sourcehut. The two agreed only at
PREFIX=/usr. Every installed data path now anchors to ASSETS, which is the
runtime lookup root.
schema.sql was never installed, so specsrht-migrate init could not find
the schema it applies wholesale on a fresh database. sourcehut-dolt has
the same latent path mismatch.
Static assets are guarded so install works before web/ exists.
feat: specsrht-migrate — brant wrapper for the spec.sr.ht schema
Single-service wrapper around git.sr.ht/~bitfehler/brant, a close sibling of
doltsrht-migrate: the brant subcommands (up, down, current, list, stamp,
validate, ping) plus an extra `init` that applies schema.sql wholesale and
stamps to head for a fresh install.
The DSN comes from [spec.sr.ht]connection-string unless --dsn overrides it;
a missing or empty value is a fatal error rather than a default connection.
Migrations load from ./migrations in a checkout, else from
<[sr.ht]assets>/migrations/spec.sr.ht. -a honours [spec.sr.ht]migrate-on-upgrade
and exits early when it is off. lib/pq's "postgres" driver replaces brant's
pgx default, which this module does not link.
Tests cover flag parsing, DSN precedence and the migrations-directory
resolution order without a database; the init/up round trips against a scratch
schema skip unless SPECSRHT_TEST_PG is set.
docs: record the Phase 0 verdict and its Phase 4 requirement
The gate passed on real data: rewrapping this document from 80 to 58
columns without changing a word gives 1563 changed lines under a line
differ and zero under the prose differ.
The verdict carries a requirement rather than just a pass. Thirteen
percent of real prose modifications shred into interleaved fragments
because those paragraphs genuinely were rewritten sentence by sentence,
so it cannot be fixed in the differ. Every such block scores similarity
at or below 0.73, so the review UI must switch to a two-column old/new
view below ~0.75; building only the inline renderer would make one review
in eight unreadable.
Also records that the differ already emits the comment-anchor tuple, with
heading path excluded from the hash so renaming a section does not orphan
every comment beneath it.
chore: promote fernet-go and go-ini to direct dependencies
authn/ imports both directly to decrypt the unified-login cookie and read
the instance ini, so the toolchain reclassifies them. No version changed
and no module was added.
Deliberately not running go mod tidy yet: bleve, chi, brant and the MCP
SDK have no importer until phases 2 and 3, and tidy would drop them from
go.mod, reintroducing it into every later parallel wave's file set.
feat: prosediff — word-level prose diff over markdown block structure
The Phase 0 de-risk gate. Segments a document into blocks with goldmark
(headings, paragraphs, list items, code fences, table rows, block quotes,
frontmatter), aligns the two block sequences with Myers over content
hashes, and diffs word-by-word inside modified prose blocks and
line-by-line inside modified code fences.
Whitespace and line wrapping alone produce no diff in prose, and always
do in code — that split is the whole point. Moves are detected by
verbatim anchor and grown over their neighbours, so a relocated section
does not explode into add+remove; a move that also bridges one edited
block is recognised, a section rewritten while moving is not, and that
limit is pinned by a test rather than papered over.
SPIKE.md reports the verdict against twelve real revisions of
docs/DESIGN.md: rewrapping the whole 1139-line document produces 1563
changed lines for git and zero changes here; 77% of real prose
modifications read as small edits; 13% shred and want a two-column
fallback in the web layer, which BlockChange.Similarity already gates.
Verdict: the approach works, build the review UI on it.
docs: correct the merge model against implementation
Resolves a contradiction: the Space section called policy changes
reviewable, but the merge is keyed by document ID and .spec.yml has none,
so a proposal could never express one. Policy joins deletion and rename as
human-push-only, which costs nothing with a single reviewer.
Corrects the claimed cost of ID-keyed staleness. It needs an ID-to-path
index over both whole trees, not one parse of the changed set; free at our
volume, two full-tree reads at ten thousand documents.
Adds the three staleness cases the two-line rule omitted, each of which is
silent corruption if unhandled, and records that an already-merged
proposal needs an ancestry check because the staleness rule reports a
confusing 409 instead.
Records that malformed documents on the approved branch are tolerated
rather than fatal, since skip-validation guarantees they can exist and
failing the index build would turn one typo into a space-wide outage.
feat: gitx — bare space repos, the git-object read path, and the id-keyed merge
One storage tier: there is no checkout on disk, so every read resolves a tree
and reads blobs. The approved head, a pinned ?rev= and a proposal branch are
the same code path with a different revision.
Lifecycle: Create makes a bare repo at <root>/~<owner>/<space> with HEAD on the
approved branch and one empty initial commit, so nothing downstream has to
special-case an unborn branch. Open reads the approved branch back from HEAD.
Merge: tree-splice, exactly as designed. Staleness is keyed by document ID, not
path — each changed document is resolved to its path on the approved head
through its frontmatter id before its blob is compared, so a rename between the
base and the head neither invents a conflict nor resurrects the moved path. On
staleness a *StaleError carries the current head for the caller's 409.
Otherwise the approved tree is rewritten with each document's blob at its path
in the head and committed with two explicit parents [approvedHead,
proposalHead]. Repository.Merge is never called; go-git v5 supports only
FastForwardMerge and this is pure object.Tree plumbing.
Refs rule: CheckRefUpdate is a pure predicate over (principal, ref, ancestry) —
agents may only touch proposals/*, the approved branch takes fast-forwards from
the human only and is never force-updated or deleted. hooks/ calls it without a
repository, which is why ancestry is an input rather than a lookup.
Concurrency: a process-wide per-space mutex keyed by repository directory guards
every write, and every ref move is a compare-and-swap that rebuilds and retries
when it loses to a concurrent native receive-pack push.
Bounded throughout: context deadlines on every operation, a per-blob cap and
per-walk byte and entry budgets. Nothing is truncated — a partially read
document would be indexed and served as though whole, so it fails instead.
Tests build every fixture programmatically through this package's own plumbing,
never the git binary, and cover the rename between base and head, each
staleness case, the two-parent merge shape, the refs rule for both principals
including force-update rejection, oversized blobs, and the CAS retry.
docs: fix an unimplementable crash repair
The repair table claimed the reconciler recreates a proposal row from an
orphan ref. It cannot: the id is a Postgres serial, and title, rationale,
base_rev, agent and agent_session exist nowhere in a ref, since trailers
live on commits and title lives nowhere in git.
Splits the truth rule instead. Refs stay authoritative for merged-ness;
the row is authoritative for existence and metadata. Row-first ordering is
already forced by deriving the branch name from the serial, and it is also
the safe order, so the unrecoverable case is designed out rather than
repaired. Orphan refs and contentless rows are deleted, which is safe
because an agent still holds the content and can re-propose.
Also records the schema-enforced invariants, notably that merged implies a
non-null approval so policy merges cannot be laundered as human ones.
feat: Postgres schema, initial migration and the db/ query layer
The six tables of the design's Postgres schema — space, document_id,
proposal, agent_token, index_stamp, digest_mark — plus a typed query layer
over them. project and comment stay deliberately absent.
Two invariants are enforced in SQL rather than in Go, because a
check-then-write in Go has a window and this does not:
- Global document ID uniqueness is the doc_id PRIMARY KEY, and the merge
path's multi-row upsert guards its ON CONFLICT branch on the owning
space, so a batch claiming another space's ID leaves that row untouched
and surfaces as a CollisionError naming where the ID really lives.
- Proposal transitions run as UPDATE ... WHERE state = 'open'; the current
state is read back only to name it in the error. CHECK constraints make
an unknown state, a merged row without an approval kind or merged_rev,
and empty provenance unwritable by anything, including SQL that does not
go through this package.
MergeProposal wraps the transition and the registry re-pointing in one
transaction: a merged proposal whose documents are still registered at their
pre-merge paths would break link resolution and the next staleness check.
Agent tokens store only a SHA-256 hash; the lookup boundary re-verifies with
subtle.ConstantTimeCompare and rejects revoked tokens distinctly.
Tests skip when SPECSRHT_TEST_PG is unset, so the pure-logic half — batch
duplicate detection, token comparison, argument guards, and the agreement
between schema.sql and migrations/0001_initial.sql — still runs with no
database available.
docs: align provenance section with the one-token decision
The section still described per-space tokens with reader/proposer/writer
roles, which the decisions table and the authorization section had already
retracted in favour of one token plus mandatory provenance.
Also records what implementation had to settle: the agent mailbox derives
from our own origin rather than the bare cookie domain, X-Agent-Base is
mandatory and must be an object name, identity travels as X-Agent and
X-Agent-Session with the base coming only from If-Match, a non-owner human
resolves to anonymous, and cookie expiry is not enforceable so network-key
rotation is the only logout-everywhere lever.
feat(authn): unified-login identity, agent token validation, provenance trailers
Cookie identity is fail-open to anonymous: a missing, tampered or
foreign-key cookie yields no user rather than an error, so anonymous
browsing never breaks on a bad cookie.
Agent tokens are the opposite. A bad bearer is a permanent 401, a store
outage a transient 503, and revoked rows are returned so operators can
distinguish a killed token from one that never existed.
TokenStore is declared here rather than imported from db/, so the two
packages can be built in parallel and service/ wires them together.
docs: correct the core-go dependency pin
The doc repeated dolt.sr.ht's README claim that core-go is wired via a
replace directive to the instance fork. Neither sibling's go.mod contains
one; both require sourcecraft.dev/bigbes/sr-ht-core directly at
dd418a200152, which is the module path the fork declares from that commit
onward. A replace against the older c2c2f38 is rejected outright, since
its go.mod still declared git.sr.ht/~sircmpwn/core-go.
feat: foundation — go.mod with every dependency, and the core/ domain
Phase 1 foundation commit. Two things, so that later parallel waves write
disjoint directories and never touch go.mod:
- go.mod / go.sum carrying every external dependency the whole module will
need (go-git, bleve, goldmark, chi, lib/pq, yaml.v3, the MCP SDK, brant,
auxilia, testify, and the sr-ht-core fork). Populated by building a
throwaway blank-import file, which is then deleted; `go mod tidy` runs
once, at the very end of the build-out.
- core/, the pure domain: owner/space names, safe relative paths, the
globally-unique document ID grammar, frontmatter parsing and schema
validation, `.spec.yml` policy with auto_merge glob matching, and the
proposal state machine. Standard library plus yaml.v3, nothing else.
Two design invariants are enforced here rather than documented and hoped for:
"approved" is not a status (it is a property of the branch a document is
reachable from), and the proposal machine has exactly open/merged/rejected.
A per-space `.spec.yml` cannot reintroduce either.
Note on the sr-ht-core pin: the design calls for a `replace` onto
git.srht.bigb.es/~bigbes/core-go at c2c2f38, but that commit's go.mod still
declares `module git.sr.ht/~sircmpwn/core-go`, so Go rejects the replacement.
Both siblings pin the later dd418a20 under the canonical path with no
replace; this does the same.
docs: add repo layout, schema, implementation plan and verification
Extends the design with the sections the sibling services carry: package
tree with a strict downward dependency rule, the Postgres schema, a
wave-based implementation plan, an end-to-end verification checklist and
the open risks.
Work is dispatched per the parallel-implementer convention: foundation
commit carries every external dependency and the core package, then
siblings importing it write disjoint directories in parallel. Phase 0
runs the prose-diff spike first because it is the assumption with no
fallback.
Projects and comments are deliberately absent from the schema: a project
is a saved filter that is speculative before there are spaces to filter,
and comment anchoring should be settled against a built review UI rather
than committed to a schema first.
docs: settle the seven questions raised by review
- Read contract keeps pinned ?rev= and the approved/draft split. The
review UI needs blob->render at arbitrary revs regardless, so pinning is
nearly free and makes X-Agent-Base auditable.
- Materialized checkout dropped. One read path over git objects for
approved head, pinned revs and proposal branches alike, which removes
atomic swap, rev stamps, a cache directory and two crash-repair rows.
Cost is bounded: vault.Scan touches the filesystem twice and
vault.FromPages already accepts pre-loaded pages.
- Volume is tens of documents a day, so warren's batch index rebuild is
absorbed unchanged and incremental indexing is explicitly not built.
- One agent token plus mandatory provenance. The refs rule is the boundary
that bounds damage; per-space scoping defers to a column and a filter.
- Human pushes are validated too, with a skip-validation push option. The
risk is a typo corrupting the global ID registry, not malice.
- External corpora left unspecified; global IDs are the only forward
compatibility needed.
- Name confirmed as spec.sr.ht.
docs: confirm browser review, and make the link the entry point
Review happens in a browser, so the prose differ stays in v1 and remains
the Phase 0 gate.
The link, not the inbox, is the normal way in: you are usually already
talking to an agent when it proposes, so every write response must carry
the proposal URL for the agent to surface. Proposal URLs are stable past
merge or rejection. The inbox demotes to a backstop for unattended work,
sharing a page with the policy-merged digest, which is invisible for the
same reason.
docs: correct errors found by independent review
Two claims were plainly wrong and load-bearing:
- post-receive cannot reject a push; its exit status is ignored once refs
have moved. Validation and the refs rule move to the update hook. The
hooks are also not 'zero service code': bleve is single-writer and the
daemon holds the index, so both hooks RPC into the daemon, and push
fails closed when it is unreachable.
- hut builds endpoints from per-service origins, never from api.sr.ht, so
federation buys it nothing and 'one endpoint, one token' is hollow. The
decision is now a read schema at our own /query; federation is a free
config line rather than a motivation.
Also resolves an internal contradiction: the doc argued for a single
shared index and then specified one bleve index per project. Now one
global index with projects as query-time filters, and globally unique
document IDs, which removes the per-project collision hole.
Further corrections: merge staleness keyed by document ID rather than
path, deletion and rename made human-push-only, approval expressed as a
property of the branch rather than frontmatter, the proposal state machine
collapsed to open/merged/rejected for a single reviewer, If-Match pinned
to the approved head, and a new consistency section covering crash repair,
the reconciler, per-space mutexes and unix ownership.
Records warren's index as a batch full rebuild rather than incremental,
and the eight questions from review that still need answers.
docs: fold in scope, audience, cadence and edit-path decisions
Four confirmed answers reshape the design rather than just the backlog:
- Agent-authored specs only. Read-only mounts leave v1; the meta-project
now unifies owned spaces. The store is a fresh silo, so it is empty
until filled, which reorders the phases.
- Single-user. Visibility levels, approver lists, approval counts and
request-changes cycles drop. Authorization relocates rather than
disappearing: it is about scoping agents, not separating humans.
- Bimodal cadence. Records approval as human or policy so auto-merged
notes are not laundered as reviewed, and adds a digest so the firehose
half stays visible.
- Human edits via clone and push. Adds the git remote as a v1 requirement
and drops the web editor. Humans push to the approved branch, agents may
only write proposals/*; SSH plus a post-receive hook needs no service
code and validates every write path.
Also records mixed ru/en search analyzers as an unresolved question that
determines whether search is usable at all.
docs: federate the read side into api.sr.ht from Phase 2
Reverses the earlier no-federation call. The deciding arguments are one
endpoint plus one meta PAT for agents already querying git/todo/builds,
hut ergonomics, and core-go/webhooks being GraphQL-native so Phase 5
pulls gqlgen in regardless. The dolt precedent does not generalize: its
API is a chunk-store protocol, whereas documents and proposals are an
ordinary CRUD graph.
Mutations stay on REST + MCP. If-Match optimistic concurrency is an HTTP
idiom with well-defined 409 semantics, and a federated type is a consumed
contract, so the unsettled proposal types stay out of the gateway.
Also records that federation is not cross-service search: thistle merges
schemas and routes fields, so the meta-project still needs our own index.
docs: add SourceHut integration section to the spec.sr.ht design
Records the config-driven wiring (Recipe B: pure Go, chrome copied from
compare.sr.ht), the canonical [spec.sr.ht] keys, and the shared keys read
in place.
Documents why skipping GraphQL federation is safe: api.sr.ht federates
every .sr.ht config section with no allow-list, but updateSchema skips
services that fail to serve /query, and the refresh is SIGHUP-driven
rather than a ticker.