# dolt.sr.ht — "DoltLab for SourceHut" (self-hosted Dolt database hosting service)

## Context

bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance
(`*.srht.bigb.es`): a new service `dolt.sr.ht` that hosts Dolt databases the way
git.sr.ht hosts git repos — `dolt clone/push/pull` over HTTPS plus an integrated web
UI (shared nav, unified login via the `sr.ht.unified-login.v1` cookie, shared
Bootstrap theme). Pure Go, single new repo at `/Users/blikh/data/home/sourcehut-dolt`
(currently empty). No upstream SourceHut modification — integration is config-driven
(a `[dolt.sr.ht]` config section puts us in every service's nav). DoltLab itself is
closed-source; everything we need is in Apache-2.0 `dolthub/dolt` packages.

**v1 scope (user-confirmed):** create/list/browse databases + clone/push over the
remotesapi, with **both** auth flows: username + meta personal access token
(`--user` + `DOLT_REMOTE_PASSWORD`, Basic) **and** dolt's Ed25519 keypair flow
(`dolt creds`/`dolt login`, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL
federation, no SSH transport, no webhooks, no issues/PRs.

All findings below were verified against dolthub/dolt `main` and the SourceHut mirror
at `/Users/blikh/data/home/sourcehut` (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/).

## Architecture summary

One Go module **`go.bigb.es/sourcehut-dolt`**, two binaries:
- **`doltsrht`** — one process, three listeners:
  - web UI (chi) on `-b localhost:5307`
  - remotesapi (gRPC `ChunkStoreService` + HTTP chunk data plane, h2c-multiplexed
    single port) on `127.0.0.1:5306` via importable
    `github.com/dolthub/dolt/go/libraries/doltcore/remotesrv`
  - a small separate gRPC server for `CredentialsService.WhoAmI` (keypair login) on
    `127.0.0.1:5308` — nginx path-routes it, so no need to touch remotesrv internals
- **`doltsrht-migrate`** — brant migrations (`git.sr.ht/~bitfehler/brant`), modeled on
  `sourcehut-migrate/cmd/sourcehut-migrate/main.go`.

Storage: **bare NBS chunk-store dirs** (no `.dolt/`, no working set) at
`<repos>/~<user>/<name>` — exactly what `remotesrv` serves and `file://` remotes use.
Postgres holds metadata (user mirror + repository + access + dolt_key).
Clone URL: `dolt clone https://dolt.srht.bigb.es/~user/db`.

Key verified facts that shape the design:
- `server.New` (core-go) works **without** `WithSchema` (gqlgen-only). But
  `WithDefaultMiddleware` attaches `auth.Middleware`, which 401s requests with no
  cookie/header — kills anonymous browsing. → don't call it; assemble our own
  middleware group on `srv.AnonRouter()`.
- core-go token validation trio (mirrors `auth.OAuth2`): `auth.DecodeBearerToken`
  (offline HMAC via `[webhooks]private-key`-derived key) + `auth.LookupUser` (local
  `"user"` table, lazily populated by `FetchMetaProfile` internal-GraphQL to meta) +
  `auth.LookupTokenRevocation` (GraphQL `tokenRevocationStatus` on meta). Requires
  shared `[sr.ht]network-key`, `[webhooks]private-key`, and our host inside meta's
  `internal-ipnet`.
- dolt CLI credential selection (`go/libraries/doltcore/env/grpc_dial_provider.go`):
  `--user` ⇒ `authorization: Basic b64(user:pass)` with pass from
  `DOLT_REMOTE_PASSWORD`; no `--user` but `dolt creds`/`dolt login` key present ⇒
  `authorization: Bearer <EdDSA JWT>` (aud = our host, exp = now+30s, kid =
  base32(SHA-512/224(pubkey)) custom alphabet `0123456789abcdefghijklmnopqrstuv`,
  iss hardcoded `dolt-client.dolthub.com` — do NOT reject on iss); no creds at all ⇒
  no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully
  host-agnostic: `dolt login --auth-endpoint <host:443> --login-url <url>`, defaults
  settable via `dolt config --global` keys `remotes.default_host`, `creds.add_url`.
- Upstream `remotesrv.ServerInterceptor` drops the authenticated ctx and never sees
  the repo path ⇒ per-repo ACLs impossible with it. → we write our own
  unary+stream interceptors (keep upstream's method classification: write =
  {GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny).
- Data plane is collision-free for nginx: gRPC under
  `/dolt.services.remotesapi.v1alpha1.*/`, sealed chunk URLs under
  `/single_symmetric_key_sealed_request/` (AES-GCM sealed, 15-min expiry, possession
  = authorization), web UI owns the rest. `getScheme()` honors
  `x-forwarded-proto` — nginx must set it on grpc_pass.
- Empty-db init primitive: `doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)`
  + `ddb.WriteEmptyRepo(ctx, "main", name, email)`.
- Bare stores **cannot** be opened by the sqle engine / embedded driver (they expect
  working sets). Browse UI uses low-level read-only APIs instead (see §Browse).
- dolt CLI v2.1.10 installed at `/opt/homebrew/bin/dolt` (integration tests).
- `github.com/dolthub/dolt/go` is a separate module pinned via pseudo-versions;
  heavy deps, CGO (gozstd). Apache-2.0.

## Repo layout

```
go.mod  Makefile  schema.sql  migrations/  config.example.ini  README.md  LICENSE
contrib/dolt.sr.ht.conf        # nginx
scss/main.scss                 # @import "base" against core.sr.ht shared scss
static/                        # logo.svg + main.min.<sha8>.css build output
cmd/doltsrht/main.go
cmd/doltsrht-migrate/main.go
core/       # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go
db/         # postgres: repos.go, access.go, keys.go (dolt_key CRUD)
authn/      # ctx.go (Caller), cookie.go (optional unified-login middleware),
            # token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify)
storage/    # init.go (InitStore/DeleteStore via WriteEmptyRepo), dbcache.go (remotesrv.DBCache)
remoteapi/  # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server)
browse/     # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores
web/        # router.go, handlers_*.go, chrome.go, templates.go, templates/*.html
```

Dependency direction: `core` ← all; `db`/`authn`/`storage`/`browse` mutually
independent; `remoteapi` ← {core, db, authn, storage}; `web` ← {core, db, authn,
browse, storage}; `cmd` ← all.

## Postgres schema (schema.sql, brant `-- +brant Up` migrations)

```sql
CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED');
CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED');
CREATE TYPE access_mode AS ENUM ('RO', 'RW');

CREATE TABLE "user" (            -- exact UserMixin shape; core-go FetchMetaProfile inserts rows
    id integer PRIMARY KEY,      -- meta's user id, inserted explicitly (not serial)
    username varchar(256) UNIQUE,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    email varchar(256) NOT NULL UNIQUE,
    user_type user_type NOT NULL,
    url varchar(256), location varchar(256), bio varchar(4096),
    suspension_notice varchar(4096)
);
CREATE INDEX ix_user_username ON "user"(username);

CREATE TABLE repository (
    id serial PRIMARY KEY,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    name varchar(64) NOT NULL,
    description varchar(1024),
    owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    path varchar(1024) NOT NULL UNIQUE,          -- absolute on-disk store dir
    visibility visibility NOT NULL,
    CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name)
);

CREATE TABLE access (
    id serial PRIMARY KEY,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE,
    user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    mode access_mode NOT NULL,
    CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id)
);

CREATE TABLE dolt_key (                          -- Ed25519 pubkeys for dolt creds auth
    id serial PRIMARY KEY,
    created timestamp NOT NULL,
    user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    kid varchar(64) NOT NULL UNIQUE,             -- base32(SHA-512/224(pubkey)), dolt alphabet
    pubkey bytea NOT NULL,                       -- 32-byte ed25519 public key
    comment varchar(256),
    last_used timestamp
);
```

DB name rule: `^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$`, max 64 (names double as
SQL database identifiers; reject `.`/`..`).

## Main wiring (`cmd/doltsrht/main.go`)

```go
conf := config.LoadConfig()
srv  := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto
db   := sql.Open("postgres", cfg[dolt.sr.ht]connection-string)
srv.AnonRouter().Group(func(r chi.Router) {
    r.Use(chimw.RealIP, chimw.Recoverer)
    r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db))
    r.Use(authn.OptionalCookieMiddleware())   // never 401s; anonymous stays anonymous
    web.Register(r)
})
rsrv := remoteapi.New(...)   // remotesapi on 127.0.0.1:5306 (grpc+http one port)
csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308
go rsrv.Serve(); go csrv.Serve(); srv.Run()
```

No redis, no email, no `WithSchema` in v1.

**Routes** (web/router.go): `GET /` dashboard; `GET,POST /create` (cookie required);
`GET /~{user}`; `GET /~{user}/{db}` overview (branches, latest commits, clone box
with both auth variants); `GET /~{user}/{db}/log?branch=&from=`;
`GET /~{user}/{db}/commit/{hash}`; `GET /~{user}/{db}/tree/{ref}`;
`GET /~{user}/{db}/table/{ref}/{table}?page=`; `GET,POST /~{user}/{db}/settings`
(owner only: description/visibility/ACLs/delete); `GET,POST /settings/keys`
(dolt-key association page — reads `#<pubkey-base32>` URL fragment via a few lines of
inline JS to prefill the form, since `dolt login` appends the pubkey as a fragment);
`GET /static/*`.

**Chrome** (web/chrome.go + templates): Go `html/template` port of core.sr.ht
`layout.html`/`nav.html` — `<nav class="container navbar navbar-light
navbar-expand-sm">`, brand `[sr.ht]site-name` + `<span class="text-danger">dolt</span>`,
network list = config sections ending `.sr.ht` (minus paste/pages) linking
`config.GetOrigin(conf, site, true)`, `active` on self; login block →
`{meta-origin}/login?return_to=...` / logout; environment banner when
`[sr.ht]environment != "production"`; stylesheet href globbed from
`static-dir/main.min.*.css` at startup. CSS: `sassc -I $(ASSETS)/scss scss/main.scss`
→ minify → `main.min.<sha256[:8]>.css` (Makefile cloned from git.sr.ht); logo.svg
copied from core.sr.ht.

## Access matrix (core/access.go — pure, table-driven tests)

Ops: OpBrowse, OpCloneRead, OpPush, OpAdmin.

| Caller \ Visibility | PUBLIC | UNLISTED | PRIVATE |
|---|---|---|---|
| anonymous / any user | browse, clone | browse, clone (unlisted) | — |
| ACL RO | browse, clone | browse, clone | browse, clone |
| ACL RW | + push | + push | browse, clone, push |
| owner | all | all | all |

PRIVATE + unauthorized ⇒ NotFound (don't leak existence). Suspended users: reads
allowed, push/admin denied. Token grants: `dolt.sr.ht/repos:RO` for reads, `:RW` for
push (empty-grant personal tokens pass via `HasAll`).

## remotesapi subsystem

- **Interceptors** (remoteapi/interceptors.go): our own
  `grpc.ChainUnaryInterceptor`/`ChainStreamInterceptor` via `ServerArgs.Options`.
  Per RPC: classify method → extract `authorization` metadata →
  `authn.ResolveGRPC(ctx, header)`:
  - `Basic` → username + pass; `auth.DecodeBearerToken(pass)` (offline HMAC) →
    require token owner == presented username → parallel `LookupUser` +
    `LookupTokenRevocation` (like `auth.OAuth2`) → positive results cached 60 s
    in-process keyed by `sha512(pass)` (a push issues many RPCs; bounds revocation
    lag), negatives uncached.
  - `Bearer` → parse JWS (go-jose, EdDSA): kid header → `dolt_key` row → verify
    signature with stored pubkey; check `aud == our host`, `exp`,
    `sub == "doltClientCredentials/<kid>"`; ignore iss. Local-only, no cache needed
    (30 s tokens); update `last_used`. Reuse `creds.PubKeyToKIDStr` from dolt.
  - absent → anonymous.
  Then repo path from `GetRepoPath()`/`RepoId` → `core.ParseRepoPath` (trim `/`,
  optional `~`, reject `..`) → load repo row → `core.Allowed(caller, repo, op)` →
  `handler(authedCtx, req)`. Stream wrapper overrides `Context()` and checks each
  `RecvMsg`. Base ctx carries `config.Context` + `database.Context`.
- **DBCache** (storage/dbcache.go): `Get(ctx, path, nbfVer)` → require existing
  repository row (no push-to-create in v1; creation is explicit via web) →
  memoized `nbs.NewLocalStore(ctx, nbfVer, absPath, 128MiB,
  nbs.NewUnlimitedMemQuotaProvider(), false)` (same as upstream LocalCSCache);
  evict + close on repo delete.
- **Assembly** (remoteapi/server.go): `remotesrv.NewServer(ServerArgs{HttpHost:
  "dolt.srht.bigb.es", HttpListenAddr == GrpcListenAddr: "127.0.0.1:5306", FS:
  filesys.LocalFS, DBCache, Options: ourInterceptors, ConcurrencyControl:
  PUSH_CONCURRENCY_CONTROL_IGNORE_WORKING_SET, ...})`. Sealed-URL data plane needs
  no extra auth. Per-process sealer key ⇒ in-flight transfers break on restart
  (client retries; acceptable).
- **CredentialsService** (remoteapi/credsvc.go): own `grpc.NewServer` on `:5308`
  registering `remotesapi.RegisterCredentialsServiceServer`; `WhoAmI` verifies the
  Bearer JWT exactly as above and returns Username/EmailAddress/DisplayName from the
  user row. `dolt login` polls WhoAmI until the key is associated via the web page.
- **Create/delete flow** (storage/init.go + web): validate name → tx INSERT
  repository → `doltdb.LoadDoltDB(ctx, types.Format_Default,
  earl.FileUrlFromPath(absPath), filesys.LocalFS)` + `ddb.WriteEmptyRepo(ctx,
  "main", [sr.ht]owner-name, owner-email)`; remove dir on tx failure. Delete: tx
  delete row → `os.RemoveAll` → DBCache evict.

**Keypair UX shipped in v1** (user-confirmed): users run
`dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url
https://dolt.srht.bigb.es/settings/keys` (or set `remotes.default_host` /
`creds.add_url` once via `dolt config --global`); the settings page associates the
posted pubkey with the logged-in SourceHut user; thereafter plain
`dolt clone/push` works with no `--user` and no env var — like git SSH keys.

## Browse subsystem (browse/) — riskiest code, isolated

No SQL engine (can't open bare stores). Per request, read-only, closed after:
`doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, filesys.LocalFS)` — NBS
manifest-based readers are safe alongside the push writer.
- Branches: `ddb.GetBranches(ctx)`; default `main`.
- Log: `ResolveCommitRef` + `commitwalk.GetTopologicalOrderIterator` (from
  `go/libraries/doltcore/env/actions/commitwalk`), paginated by count + start hash.
- Tables/schema: commit → `RootValue` → `GetTableNames` / `GetTable(...).GetSchema()`.
- Rows: `table.GetRowData` → `durable.ProllyMapFromIndex` →
  `prolly.Map.IterOrdinalRange(offset, offset+limit)`, rendered via schema types.
- Commit page: diff **summaries** only in v1 (per-table added/dropped/schema-changed
  + row-count delta via the diff package); full row diffs deferred.

Fallback if prolly iteration proves fragile across versions: keep a lightweight
materialized clone per db (real `.dolt`, refreshed post-push) opened via
`dolthub/driver` — swap contained inside `browse/`.

## Config + nginx

`config.example.ini` (goes into the single shared instance config.ini; existing
services need only the `[dolt.sr.ht] origin=` line + restart to show us in nav):

```ini
[dolt.sr.ht]
origin=https://dolt.srht.bigb.es
connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
repos=/var/lib/dolt
remotesapi-listen=127.0.0.1:5306
credsapi-listen=127.0.0.1:5308
static-dir=/usr/share/sourcehut/dolt.sr.ht/static
migrate-on-upgrade=yes
```

Reused in place: `[sr.ht]` network-key / internal-ipnet / owner-* / site-name /
environment; `[webhooks]private-key`; `[meta.sr.ht]origin`. Deployment checklist:
our host must be in **meta's** `internal-ipnet` (FetchMetaProfile +
LookupTokenRevocation use internal auth).

nginx (`contrib/dolt.sr.ht.conf`, sr.ht-nginx style):

```nginx
server_name dolt.srht.bigb.es;  # TLS/http2 via sourcehut.conf + port443.conf
location /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ {
    grpc_pass grpc://127.0.0.1:5306;
    grpc_set_header X-Forwarded-Proto https;
    grpc_read_timeout 600s; grpc_send_timeout 600s; client_max_body_size 0;
}
location /dolt.services.remotesapi.v1alpha1.CredentialsService/ {
    grpc_pass grpc://127.0.0.1:5308;
    grpc_set_header X-Forwarded-Proto https;
}
location /single_symmetric_key_sealed_request/ {   # chunk data plane GET+PUT
    proxy_pass http://127.0.0.1:5306;
    client_max_body_size 0; proxy_request_buffering off; proxy_read_timeout 600s;
}
location / { proxy_pass http://127.0.0.1:5307; }   # web UI + /static
```

## Implementation phases (parallel-implementer convention from CLAUDE.md)

**Phase 0 — foundation (serial, commit first):** go.mod; pin ALL deps via `go get`
(+ throwaway smoke-import build to populate go.sum): core-go@mirror-version,
`github.com/dolthub/dolt/go@latest`-resolved pseudo-version (record it),
go-jose, grpc, chi/v5, lib/pq, brant, logrus. Complete `core/` package + unit
tests; schema.sql + migrations/0001; config.example.ini; Makefile; nginx conf;
README. **Spike test** (`storage/spike_test.go`, build-tagged): WriteEmptyRepo →
remotesrv on localhost → `dolt clone http://127.0.0.1:PORT/x/y` with CLI v2.1.10 —
proves format/RPC compat before any parallel work. Gate: `go build ./... && go test
./core/...`.

**Phase 1 — parallel wave A** (disjoint packages; each implementer builds/tests only
its own package, never `./...`):
- 1a `db/` — repository/access/dolt_key queries.
- 1b `authn/` — cookie + Basic-token + Bearer-JWT resolution; tests forge tokens via
  `auth.BearerToken.Encode()` with synthesized ini (random fernet key + ed25519
  seed); revocation stubbed behind an interface; JWT tests sign with
  ephemeral ed25519 keys.
- 1c `storage/` — InitStore/DeleteStore/Cache (repo lookup injected as func).
- 1d `browse/` — fixtures built programmatically via doltdb API.

**Phase 2 — parallel wave B:**
- 2a `remoteapi/` — interceptors, server assembly, credsvc; integration test
  (`-tags integration`) driving real dolt CLI: clone → sql insert → commit → push →
  re-clone → verify; ACL denial; anonymous PUBLIC clone; Bearer-JWT clone after key
  association; PRIVATE anonymous ⇒ NotFound. (needs 1b, 1c; 1a via interface)
- 2b `web/` — router/handlers/templates/chrome; httptest with stubbed interfaces.
  (needs 1a, 1b, 1d)

**Phase 3 — serial finish:** `cmd/doltsrht`, `cmd/doltsrht-migrate`, startup smoke
test (/healthz), **`go mod tidy` only now**, CSS build against instance scss, README
deployment section.

## Verification (post-deploy, end-to-end)

1. Add `[dolt.sr.ht]` section to shared config.ini, `doltsrht-migrate init`, deploy
   nginx conf + DNS `dolt.srht.bigb.es`, start `doltsrht`.
2. Browser: log into meta → visit dolt.srht.bigb.es → nav shows all services +
   "Logged in as bigbes"; nav of git/meta pages now shows "dolt".
3. Create db `demo` (PUBLIC) via /create.
4. Token flow: `export DOLT_REMOTE_PASSWORD=<meta PAT>`; `dolt clone --user bigbes
   https://dolt.srht.bigb.es/~bigbes/demo`; create table, insert, `dolt commit`,
   `dolt push origin main`.
5. Web: overview shows branch + commits; /log, /table/main/t shows the row;
   /commit/<hash> shows table summary.
6. Keypair flow: `dolt creds new`; `dolt login --auth-endpoint
   dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys`;
   associate key in browser; `dolt clone https://dolt.srht.bigb.es/~bigbes/demo`
   (no --user) + push succeed.
7. Negative: anonymous clone of PUBLIC works; flip to PRIVATE → anonymous clone ⇒
   not found; user2 token push ⇒ denied; grant RO → clone ok, push denied; revoke
   PAT on meta → push blocked within ≤60 s.

## Open risks

- dolt CLI v2.1.10 ⟷ pinned `dolt/go` module compat — de-risked by Phase-0 spike;
  worst case pin different module version or require newer CLI.
- prolly-level browse code is version-fragile — isolated in `browse/` with
  materialized-clone fallback design.
- 60 s revocation-cache window for Basic tokens — accepted, documented.
- internal-ipnet / shared-key misconfig symptom: "Temporary error" on first
  login/push — deployment checklist item.