@@ 0,0 1,377 @@
+# dolt.sr.ht — "DoltLab for SourceHut" (self-hosted Dolt database hosting service)
+
+## Context
+
+bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance
+(`*.srht.bigb.es`): a new service `dolt.sr.ht` that hosts Dolt databases the way
+git.sr.ht hosts git repos — `dolt clone/push/pull` over HTTPS plus an integrated web
+UI (shared nav, unified login via the `sr.ht.unified-login.v1` cookie, shared
+Bootstrap theme). Pure Go, single new repo at `/Users/blikh/data/home/sourcehut-dolt`
+(currently empty). No upstream SourceHut modification — integration is config-driven
+(a `[dolt.sr.ht]` config section puts us in every service's nav). DoltLab itself is
+closed-source; everything we need is in Apache-2.0 `dolthub/dolt` packages.
+
+**v1 scope (user-confirmed):** create/list/browse databases + clone/push over the
+remotesapi, with **both** auth flows: username + meta personal access token
+(`--user` + `DOLT_REMOTE_PASSWORD`, Basic) **and** dolt's Ed25519 keypair flow
+(`dolt creds`/`dolt login`, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL
+federation, no SSH transport, no webhooks, no issues/PRs.
+
+All findings below were verified against dolthub/dolt `main` and the SourceHut mirror
+at `/Users/blikh/data/home/sourcehut` (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/).
+
+## Architecture summary
+
+One Go module **`go.bigb.es/sourcehut-dolt`**, two binaries:
+- **`doltsrht`** — one process, three listeners:
+ - web UI (chi) on `-b localhost:5307`
+ - remotesapi (gRPC `ChunkStoreService` + HTTP chunk data plane, h2c-multiplexed
+ single port) on `127.0.0.1:5306` via importable
+ `github.com/dolthub/dolt/go/libraries/doltcore/remotesrv`
+ - a small separate gRPC server for `CredentialsService.WhoAmI` (keypair login) on
+ `127.0.0.1:5308` — nginx path-routes it, so no need to touch remotesrv internals
+- **`doltsrht-migrate`** — brant migrations (`git.sr.ht/~bitfehler/brant`), modeled on
+ `sourcehut-migrate/cmd/sourcehut-migrate/main.go`.
+
+Storage: **bare NBS chunk-store dirs** (no `.dolt/`, no working set) at
+`<repos>/~<user>/<name>` — exactly what `remotesrv` serves and `file://` remotes use.
+Postgres holds metadata (user mirror + repository + access + dolt_key).
+Clone URL: `dolt clone https://dolt.srht.bigb.es/~user/db`.
+
+Key verified facts that shape the design:
+- `server.New` (core-go) works **without** `WithSchema` (gqlgen-only). But
+ `WithDefaultMiddleware` attaches `auth.Middleware`, which 401s requests with no
+ cookie/header — kills anonymous browsing. → don't call it; assemble our own
+ middleware group on `srv.AnonRouter()`.
+- core-go token validation trio (mirrors `auth.OAuth2`): `auth.DecodeBearerToken`
+ (offline HMAC via `[webhooks]private-key`-derived key) + `auth.LookupUser` (local
+ `"user"` table, lazily populated by `FetchMetaProfile` internal-GraphQL to meta) +
+ `auth.LookupTokenRevocation` (GraphQL `tokenRevocationStatus` on meta). Requires
+ shared `[sr.ht]network-key`, `[webhooks]private-key`, and our host inside meta's
+ `internal-ipnet`.
+- dolt CLI credential selection (`go/libraries/doltcore/env/grpc_dial_provider.go`):
+ `--user` ⇒ `authorization: Basic b64(user:pass)` with pass from
+ `DOLT_REMOTE_PASSWORD`; no `--user` but `dolt creds`/`dolt login` key present ⇒
+ `authorization: Bearer <EdDSA JWT>` (aud = our host, exp = now+30s, kid =
+ base32(SHA-512/224(pubkey)) custom alphabet `0123456789abcdefghijklmnopqrstuv`,
+ iss hardcoded `dolt-client.dolthub.com` — do NOT reject on iss); no creds at all ⇒
+ no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully
+ host-agnostic: `dolt login --auth-endpoint <host:443> --login-url <url>`, defaults
+ settable via `dolt config --global` keys `remotes.default_host`, `creds.add_url`.
+- Upstream `remotesrv.ServerInterceptor` drops the authenticated ctx and never sees
+ the repo path ⇒ per-repo ACLs impossible with it. → we write our own
+ unary+stream interceptors (keep upstream's method classification: write =
+ {GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny).
+- Data plane is collision-free for nginx: gRPC under
+ `/dolt.services.remotesapi.v1alpha1.*/`, sealed chunk URLs under
+ `/single_symmetric_key_sealed_request/` (AES-GCM sealed, 15-min expiry, possession
+ = authorization), web UI owns the rest. `getScheme()` honors
+ `x-forwarded-proto` — nginx must set it on grpc_pass.
+- Empty-db init primitive: `doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)`
+ + `ddb.WriteEmptyRepo(ctx, "main", name, email)`.
+- Bare stores **cannot** be opened by the sqle engine / embedded driver (they expect
+ working sets). Browse UI uses low-level read-only APIs instead (see §Browse).
+- dolt CLI v2.1.10 installed at `/opt/homebrew/bin/dolt` (integration tests).
+- `github.com/dolthub/dolt/go` is a separate module pinned via pseudo-versions;
+ heavy deps, CGO (gozstd). Apache-2.0.
+
+## Repo layout
+
+```
+go.mod Makefile schema.sql migrations/ config.example.ini README.md LICENSE
+contrib/dolt.sr.ht.conf # nginx
+scss/main.scss # @import "base" against core.sr.ht shared scss
+static/ # logo.svg + main.min.<sha8>.css build output
+cmd/doltsrht/main.go
+cmd/doltsrht-migrate/main.go
+core/ # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go
+db/ # postgres: repos.go, access.go, keys.go (dolt_key CRUD)
+authn/ # ctx.go (Caller), cookie.go (optional unified-login middleware),
+ # token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify)
+storage/ # init.go (InitStore/DeleteStore via WriteEmptyRepo), dbcache.go (remotesrv.DBCache)
+remoteapi/ # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server)
+browse/ # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores
+web/ # router.go, handlers_*.go, chrome.go, templates.go, templates/*.html
+```
+
+Dependency direction: `core` ← all; `db`/`authn`/`storage`/`browse` mutually
+independent; `remoteapi` ← {core, db, authn, storage}; `web` ← {core, db, authn,
+browse, storage}; `cmd` ← all.
+
+## Postgres schema (schema.sql, brant `-- +brant Up` migrations)
+
+```sql
+CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED');
+CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED');
+CREATE TYPE access_mode AS ENUM ('RO', 'RW');
+
+CREATE TABLE "user" ( -- exact UserMixin shape; core-go FetchMetaProfile inserts rows
+ id integer PRIMARY KEY, -- meta's user id, inserted explicitly (not serial)
+ username varchar(256) UNIQUE,
+ created timestamp NOT NULL, updated timestamp NOT NULL,
+ email varchar(256) NOT NULL UNIQUE,
+ user_type user_type NOT NULL,
+ url varchar(256), location varchar(256), bio varchar(4096),
+ suspension_notice varchar(4096)
+);
+CREATE INDEX ix_user_username ON "user"(username);
+
+CREATE TABLE repository (
+ id serial PRIMARY KEY,
+ created timestamp NOT NULL, updated timestamp NOT NULL,
+ name varchar(64) NOT NULL,
+ description varchar(1024),
+ owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
+ path varchar(1024) NOT NULL UNIQUE, -- absolute on-disk store dir
+ visibility visibility NOT NULL,
+ CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name)
+);
+
+CREATE TABLE access (
+ id serial PRIMARY KEY,
+ created timestamp NOT NULL, updated timestamp NOT NULL,
+ repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE,
+ user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
+ mode access_mode NOT NULL,
+ CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id)
+);
+
+CREATE TABLE dolt_key ( -- Ed25519 pubkeys for dolt creds auth
+ id serial PRIMARY KEY,
+ created timestamp NOT NULL,
+ user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
+ kid varchar(64) NOT NULL UNIQUE, -- base32(SHA-512/224(pubkey)), dolt alphabet
+ pubkey bytea NOT NULL, -- 32-byte ed25519 public key
+ comment varchar(256),
+ last_used timestamp
+);
+```
+
+DB name rule: `^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$`, max 64 (names double as
+SQL database identifiers; reject `.`/`..`).
+
+## Main wiring (`cmd/doltsrht/main.go`)
+
+```go
+conf := config.LoadConfig()
+srv := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto
+db := sql.Open("postgres", cfg[dolt.sr.ht]connection-string)
+srv.AnonRouter().Group(func(r chi.Router) {
+ r.Use(chimw.RealIP, chimw.Recoverer)
+ r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db))
+ r.Use(authn.OptionalCookieMiddleware()) // never 401s; anonymous stays anonymous
+ web.Register(r)
+})
+rsrv := remoteapi.New(...) // remotesapi on 127.0.0.1:5306 (grpc+http one port)
+csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308
+go rsrv.Serve(); go csrv.Serve(); srv.Run()
+```
+
+No redis, no email, no `WithSchema` in v1.
+
+**Routes** (web/router.go): `GET /` dashboard; `GET,POST /create` (cookie required);
+`GET /~{user}`; `GET /~{user}/{db}` overview (branches, latest commits, clone box
+with both auth variants); `GET /~{user}/{db}/log?branch=&from=`;
+`GET /~{user}/{db}/commit/{hash}`; `GET /~{user}/{db}/tree/{ref}`;
+`GET /~{user}/{db}/table/{ref}/{table}?page=`; `GET,POST /~{user}/{db}/settings`
+(owner only: description/visibility/ACLs/delete); `GET,POST /settings/keys`
+(dolt-key association page — reads `#<pubkey-base32>` URL fragment via a few lines of
+inline JS to prefill the form, since `dolt login` appends the pubkey as a fragment);
+`GET /static/*`.
+
+**Chrome** (web/chrome.go + templates): Go `html/template` port of core.sr.ht
+`layout.html`/`nav.html` — `<nav class="container navbar navbar-light
+navbar-expand-sm">`, brand `[sr.ht]site-name` + `<span class="text-danger">dolt</span>`,
+network list = config sections ending `.sr.ht` (minus paste/pages) linking
+`config.GetOrigin(conf, site, true)`, `active` on self; login block →
+`{meta-origin}/login?return_to=...` / logout; environment banner when
+`[sr.ht]environment != "production"`; stylesheet href globbed from
+`static-dir/main.min.*.css` at startup. CSS: `sassc -I $(ASSETS)/scss scss/main.scss`
+→ minify → `main.min.<sha256[:8]>.css` (Makefile cloned from git.sr.ht); logo.svg
+copied from core.sr.ht.
+
+## Access matrix (core/access.go — pure, table-driven tests)
+
+Ops: OpBrowse, OpCloneRead, OpPush, OpAdmin.
+
+| Caller \ Visibility | PUBLIC | UNLISTED | PRIVATE |
+|---|---|---|---|
+| anonymous / any user | browse, clone | browse, clone (unlisted) | — |
+| ACL RO | browse, clone | browse, clone | browse, clone |
+| ACL RW | + push | + push | browse, clone, push |
+| owner | all | all | all |
+
+PRIVATE + unauthorized ⇒ NotFound (don't leak existence). Suspended users: reads
+allowed, push/admin denied. Token grants: `dolt.sr.ht/repos:RO` for reads, `:RW` for
+push (empty-grant personal tokens pass via `HasAll`).
+
+## remotesapi subsystem
+
+- **Interceptors** (remoteapi/interceptors.go): our own
+ `grpc.ChainUnaryInterceptor`/`ChainStreamInterceptor` via `ServerArgs.Options`.
+ Per RPC: classify method → extract `authorization` metadata →
+ `authn.ResolveGRPC(ctx, header)`:
+ - `Basic` → username + pass; `auth.DecodeBearerToken(pass)` (offline HMAC) →
+ require token owner == presented username → parallel `LookupUser` +
+ `LookupTokenRevocation` (like `auth.OAuth2`) → positive results cached 60 s
+ in-process keyed by `sha512(pass)` (a push issues many RPCs; bounds revocation
+ lag), negatives uncached.
+ - `Bearer` → parse JWS (go-jose, EdDSA): kid header → `dolt_key` row → verify
+ signature with stored pubkey; check `aud == our host`, `exp`,
+ `sub == "doltClientCredentials/<kid>"`; ignore iss. Local-only, no cache needed
+ (30 s tokens); update `last_used`. Reuse `creds.PubKeyToKIDStr` from dolt.
+ - absent → anonymous.
+ Then repo path from `GetRepoPath()`/`RepoId` → `core.ParseRepoPath` (trim `/`,
+ optional `~`, reject `..`) → load repo row → `core.Allowed(caller, repo, op)` →
+ `handler(authedCtx, req)`. Stream wrapper overrides `Context()` and checks each
+ `RecvMsg`. Base ctx carries `config.Context` + `database.Context`.
+- **DBCache** (storage/dbcache.go): `Get(ctx, path, nbfVer)` → require existing
+ repository row (no push-to-create in v1; creation is explicit via web) →
+ memoized `nbs.NewLocalStore(ctx, nbfVer, absPath, 128MiB,
+ nbs.NewUnlimitedMemQuotaProvider(), false)` (same as upstream LocalCSCache);
+ evict + close on repo delete.
+- **Assembly** (remoteapi/server.go): `remotesrv.NewServer(ServerArgs{HttpHost:
+ "dolt.srht.bigb.es", HttpListenAddr == GrpcListenAddr: "127.0.0.1:5306", FS:
+ filesys.LocalFS, DBCache, Options: ourInterceptors, ConcurrencyControl:
+ PUSH_CONCURRENCY_CONTROL_IGNORE_WORKING_SET, ...})`. Sealed-URL data plane needs
+ no extra auth. Per-process sealer key ⇒ in-flight transfers break on restart
+ (client retries; acceptable).
+- **CredentialsService** (remoteapi/credsvc.go): own `grpc.NewServer` on `:5308`
+ registering `remotesapi.RegisterCredentialsServiceServer`; `WhoAmI` verifies the
+ Bearer JWT exactly as above and returns Username/EmailAddress/DisplayName from the
+ user row. `dolt login` polls WhoAmI until the key is associated via the web page.
+- **Create/delete flow** (storage/init.go + web): validate name → tx INSERT
+ repository → `doltdb.LoadDoltDB(ctx, types.Format_Default,
+ earl.FileUrlFromPath(absPath), filesys.LocalFS)` + `ddb.WriteEmptyRepo(ctx,
+ "main", [sr.ht]owner-name, owner-email)`; remove dir on tx failure. Delete: tx
+ delete row → `os.RemoveAll` → DBCache evict.
+
+**Keypair UX shipped in v1** (user-confirmed): users run
+`dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url
+https://dolt.srht.bigb.es/settings/keys` (or set `remotes.default_host` /
+`creds.add_url` once via `dolt config --global`); the settings page associates the
+posted pubkey with the logged-in SourceHut user; thereafter plain
+`dolt clone/push` works with no `--user` and no env var — like git SSH keys.
+
+## Browse subsystem (browse/) — riskiest code, isolated
+
+No SQL engine (can't open bare stores). Per request, read-only, closed after:
+`doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, filesys.LocalFS)` — NBS
+manifest-based readers are safe alongside the push writer.
+- Branches: `ddb.GetBranches(ctx)`; default `main`.
+- Log: `ResolveCommitRef` + `commitwalk.GetTopologicalOrderIterator` (from
+ `go/libraries/doltcore/env/actions/commitwalk`), paginated by count + start hash.
+- Tables/schema: commit → `RootValue` → `GetTableNames` / `GetTable(...).GetSchema()`.
+- Rows: `table.GetRowData` → `durable.ProllyMapFromIndex` →
+ `prolly.Map.IterOrdinalRange(offset, offset+limit)`, rendered via schema types.
+- Commit page: diff **summaries** only in v1 (per-table added/dropped/schema-changed
+ + row-count delta via the diff package); full row diffs deferred.
+
+Fallback if prolly iteration proves fragile across versions: keep a lightweight
+materialized clone per db (real `.dolt`, refreshed post-push) opened via
+`dolthub/driver` — swap contained inside `browse/`.
+
+## Config + nginx
+
+`config.example.ini` (goes into the single shared instance config.ini; existing
+services need only the `[dolt.sr.ht] origin=` line + restart to show us in nav):
+
+```ini
+[dolt.sr.ht]
+origin=https://dolt.srht.bigb.es
+connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
+repos=/var/lib/dolt
+remotesapi-listen=127.0.0.1:5306
+credsapi-listen=127.0.0.1:5308
+static-dir=/usr/share/sourcehut/dolt.sr.ht/static
+migrate-on-upgrade=yes
+```
+
+Reused in place: `[sr.ht]` network-key / internal-ipnet / owner-* / site-name /
+environment; `[webhooks]private-key`; `[meta.sr.ht]origin`. Deployment checklist:
+our host must be in **meta's** `internal-ipnet` (FetchMetaProfile +
+LookupTokenRevocation use internal auth).
+
+nginx (`contrib/dolt.sr.ht.conf`, sr.ht-nginx style):
+
+```nginx
+server_name dolt.srht.bigb.es; # TLS/http2 via sourcehut.conf + port443.conf
+location /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ {
+ grpc_pass grpc://127.0.0.1:5306;
+ grpc_set_header X-Forwarded-Proto https;
+ grpc_read_timeout 600s; grpc_send_timeout 600s; client_max_body_size 0;
+}
+location /dolt.services.remotesapi.v1alpha1.CredentialsService/ {
+ grpc_pass grpc://127.0.0.1:5308;
+ grpc_set_header X-Forwarded-Proto https;
+}
+location /single_symmetric_key_sealed_request/ { # chunk data plane GET+PUT
+ proxy_pass http://127.0.0.1:5306;
+ client_max_body_size 0; proxy_request_buffering off; proxy_read_timeout 600s;
+}
+location / { proxy_pass http://127.0.0.1:5307; } # web UI + /static
+```
+
+## Implementation phases (parallel-implementer convention from CLAUDE.md)
+
+**Phase 0 — foundation (serial, commit first):** go.mod; pin ALL deps via `go get`
+(+ throwaway smoke-import build to populate go.sum): core-go@mirror-version,
+`github.com/dolthub/dolt/go@latest`-resolved pseudo-version (record it),
+go-jose, grpc, chi/v5, lib/pq, brant, logrus. Complete `core/` package + unit
+tests; schema.sql + migrations/0001; config.example.ini; Makefile; nginx conf;
+README. **Spike test** (`storage/spike_test.go`, build-tagged): WriteEmptyRepo →
+remotesrv on localhost → `dolt clone http://127.0.0.1:PORT/x/y` with CLI v2.1.10 —
+proves format/RPC compat before any parallel work. Gate: `go build ./... && go test
+./core/...`.
+
+**Phase 1 — parallel wave A** (disjoint packages; each implementer builds/tests only
+its own package, never `./...`):
+- 1a `db/` — repository/access/dolt_key queries.
+- 1b `authn/` — cookie + Basic-token + Bearer-JWT resolution; tests forge tokens via
+ `auth.BearerToken.Encode()` with synthesized ini (random fernet key + ed25519
+ seed); revocation stubbed behind an interface; JWT tests sign with
+ ephemeral ed25519 keys.
+- 1c `storage/` — InitStore/DeleteStore/Cache (repo lookup injected as func).
+- 1d `browse/` — fixtures built programmatically via doltdb API.
+
+**Phase 2 — parallel wave B:**
+- 2a `remoteapi/` — interceptors, server assembly, credsvc; integration test
+ (`-tags integration`) driving real dolt CLI: clone → sql insert → commit → push →
+ re-clone → verify; ACL denial; anonymous PUBLIC clone; Bearer-JWT clone after key
+ association; PRIVATE anonymous ⇒ NotFound. (needs 1b, 1c; 1a via interface)
+- 2b `web/` — router/handlers/templates/chrome; httptest with stubbed interfaces.
+ (needs 1a, 1b, 1d)
+
+**Phase 3 — serial finish:** `cmd/doltsrht`, `cmd/doltsrht-migrate`, startup smoke
+test (/healthz), **`go mod tidy` only now**, CSS build against instance scss, README
+deployment section.
+
+## Verification (post-deploy, end-to-end)
+
+1. Add `[dolt.sr.ht]` section to shared config.ini, `doltsrht-migrate init`, deploy
+ nginx conf + DNS `dolt.srht.bigb.es`, start `doltsrht`.
+2. Browser: log into meta → visit dolt.srht.bigb.es → nav shows all services +
+ "Logged in as bigbes"; nav of git/meta pages now shows "dolt".
+3. Create db `demo` (PUBLIC) via /create.
+4. Token flow: `export DOLT_REMOTE_PASSWORD=<meta PAT>`; `dolt clone --user bigbes
+ https://dolt.srht.bigb.es/~bigbes/demo`; create table, insert, `dolt commit`,
+ `dolt push origin main`.
+5. Web: overview shows branch + commits; /log, /table/main/t shows the row;
+ /commit/<hash> shows table summary.
+6. Keypair flow: `dolt creds new`; `dolt login --auth-endpoint
+ dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys`;
+ associate key in browser; `dolt clone https://dolt.srht.bigb.es/~bigbes/demo`
+ (no --user) + push succeed.
+7. Negative: anonymous clone of PUBLIC works; flip to PRIVATE → anonymous clone ⇒
+ not found; user2 token push ⇒ denied; grant RO → clone ok, push denied; revoke
+ PAT on meta → push blocked within ≤60 s.
+
+## Open risks
+
+- dolt CLI v2.1.10 ⟷ pinned `dolt/go` module compat — de-risked by Phase-0 spike;
+ worst case pin different module version or require newer CLI.
+- prolly-level browse code is version-fragile — isolated in `browse/` with
+ materialized-clone fallback design.
+- 60 s revocation-cache window for Basic tokens — accepted, documented.
+- internal-ipnet / shared-key misconfig symptom: "Temporary error" on first
+ login/push — deployment checklist item.