#dolt.sr.ht — "DoltLab for SourceHut" (self-hosted Dolt database hosting service)

#Context

bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance (*.srht.bigb.es): a new service dolt.sr.ht that hosts Dolt databases the way git.sr.ht hosts git repos — dolt clone/push/pull over HTTPS plus an integrated web UI (shared nav, unified login via the sr.ht.unified-login.v1 cookie, shared Bootstrap theme). Pure Go, single new repo at /Users/blikh/data/home/sourcehut-dolt (currently empty). No upstream SourceHut modification — integration is config-driven (a [dolt.sr.ht] config section puts us in every service's nav). DoltLab itself is closed-source; everything we need is in Apache-2.0 dolthub/dolt packages.

v1 scope (user-confirmed): create/list/browse databases + clone/push over the remotesapi, with both auth flows: username + meta personal access token (--user + DOLT_REMOTE_PASSWORD, Basic) and dolt's Ed25519 keypair flow (dolt creds/dolt login, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL federation, no SSH transport, no webhooks, no issues/PRs.

All findings below were verified against dolthub/dolt main and the SourceHut mirror at /Users/blikh/data/home/sourcehut (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/).

#Architecture summary

One Go module sourcecraft.dev/bigbes/sr-ht-dolt, two binaries:

• doltsrht — one process, three listeners:
  • web UI (chi) on -b localhost:5307
  • remotesapi (gRPC ChunkStoreService + HTTP chunk data plane, h2c-multiplexed single port) on 127.0.0.1:5306 via importable github.com/dolthub/dolt/go/libraries/doltcore/remotesrv
  • a small separate gRPC server for CredentialsService.WhoAmI (keypair login) on 127.0.0.1:5308 — nginx path-routes it, so no need to touch remotesrv internals
• doltsrht-migrate — brant migrations (git.sr.ht/~bitfehler/brant), modeled on sourcehut-migrate/cmd/sourcehut-migrate/main.go.

Storage: bare NBS chunk-store dirs (no .dolt/, no working set) at <repos>/~<user>/<name> — exactly what remotesrv serves and file:// remotes use. Postgres holds metadata (user mirror + repository + access + dolt_key). Clone URL: dolt clone https://dolt.srht.bigb.es/~user/db.

Key verified facts that shape the design:

• server.New (core-go) works without WithSchema (gqlgen-only). But WithDefaultMiddleware attaches auth.Middleware, which 401s requests with no cookie/header — kills anonymous browsing. → don't call it; assemble our own middleware group on srv.AnonRouter().
• core-go token validation trio (mirrors auth.OAuth2): auth.DecodeBearerToken (offline HMAC via [webhooks]private-key-derived key) + auth.LookupUser (local "user" table, lazily populated by FetchMetaProfile internal-GraphQL to meta) + auth.LookupTokenRevocation (GraphQL tokenRevocationStatus on meta). Requires shared [sr.ht]network-key, [webhooks]private-key, and our host inside meta's internal-ipnet.
• dolt CLI credential selection (go/libraries/doltcore/env/grpc_dial_provider.go): --user ⇒ authorization: Basic b64(user:pass) with pass from DOLT_REMOTE_PASSWORD; no --user but dolt creds/dolt login key present ⇒ authorization: Bearer <EdDSA JWT> (aud = our host, exp = now+30s, kid = base32(SHA-512/224(pubkey)) custom alphabet 0123456789abcdefghijklmnopqrstuv, iss hardcoded dolt-client.dolthub.com — do NOT reject on iss); no creds at all ⇒ no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully host-agnostic: dolt login --auth-endpoint <host:443> --login-url <url>, defaults settable via dolt config --global keys remotes.default_host, creds.add_url.
• Upstream remotesrv.ServerInterceptor drops the authenticated ctx and never sees the repo path ⇒ per-repo ACLs impossible with it. → we write our own unary+stream interceptors (keep upstream's method classification: write = {GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny).
• Data plane is collision-free for nginx: gRPC under /dolt.services.remotesapi.v1alpha1.*/, sealed chunk URLs under /single_symmetric_key_sealed_request/ (AES-GCM sealed, 15-min expiry, possession = authorization), web UI owns the rest. getScheme() honors x-forwarded-proto — nginx must set it on grpc_pass.
• Empty-db init primitive: doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)
  • ddb.WriteEmptyRepo(ctx, "main", name, email).
• Bare stores cannot be opened by the sqle engine / embedded driver (they expect working sets). Browse UI uses low-level read-only APIs instead (see §Browse).
• dolt CLI v2.1.10 installed at /opt/homebrew/bin/dolt (integration tests).
• github.com/dolthub/dolt/go is a separate module pinned via pseudo-versions; heavy deps, CGO (gozstd). Apache-2.0.

#Repo layout

go.mod  Makefile  schema.sql  migrations/  config.example.ini  README.md  LICENSE
contrib/dolt.sr.ht.conf        # nginx
scss/main.scss                 # @import "base" against core.sr.ht shared scss
static/                        # logo.svg + main.min.<sha8>.css build output
cmd/doltsrht/main.go
cmd/doltsrht-migrate/main.go
core/       # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go
db/         # postgres: repos.go, access.go, keys.go (dolt_key CRUD)
authn/      # ctx.go (Caller), cookie.go (optional unified-login middleware),
            # token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify)
storage/    # init.go (InitStore/DeleteStore via WriteEmptyRepo), dbcache.go (remotesrv.DBCache)
remoteapi/  # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server)
browse/     # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores
web/        # router.go, handlers_*.go, chrome.go, templates.go, templates/*.html

Dependency direction: core ← all; db/authn/storage/browse mutually independent; remoteapi ← {core, db, authn, storage}; web ← {core, db, authn, browse, storage}; cmd ← all.

#Postgres schema (schema.sql, brant -- +brant Up migrations)

CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED');
CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED');
CREATE TYPE access_mode AS ENUM ('RO', 'RW');

CREATE TABLE "user" (            -- exact UserMixin shape; core-go FetchMetaProfile inserts rows
    id integer PRIMARY KEY,      -- meta's user id, inserted explicitly (not serial)
    username varchar(256) UNIQUE,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    email varchar(256) NOT NULL UNIQUE,
    user_type user_type NOT NULL,
    url varchar(256), location varchar(256), bio varchar(4096),
    suspension_notice varchar(4096)
);
CREATE INDEX ix_user_username ON "user"(username);

CREATE TABLE repository (
    id serial PRIMARY KEY,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    name varchar(64) NOT NULL,
    description varchar(1024),
    owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    path varchar(1024) NOT NULL UNIQUE,          -- absolute on-disk store dir
    visibility visibility NOT NULL,
    CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name)
);

CREATE TABLE access (
    id serial PRIMARY KEY,
    created timestamp NOT NULL, updated timestamp NOT NULL,
    repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE,
    user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    mode access_mode NOT NULL,
    CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id)
);

CREATE TABLE dolt_key (                          -- Ed25519 pubkeys for dolt creds auth
    id serial PRIMARY KEY,
    created timestamp NOT NULL,
    user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
    kid varchar(64) NOT NULL UNIQUE,             -- base32(SHA-512/224(pubkey)), dolt alphabet
    pubkey bytea NOT NULL,                       -- 32-byte ed25519 public key
    comment varchar(256),
    last_used timestamp
);

DB name rule: ^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$, max 64 (names double as SQL database identifiers; reject ./..).

#Main wiring (cmd/doltsrht/main.go)

conf := config.LoadConfig()
srv  := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto
db   := sql.Open("postgres", cfg[dolt.sr.ht]connection-string)
srv.AnonRouter().Group(func(r chi.Router) {
    r.Use(chimw.RealIP, chimw.Recoverer)
    r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db))
    r.Use(authn.OptionalCookieMiddleware())   // never 401s; anonymous stays anonymous
    web.Register(r)
})
rsrv := remoteapi.New(...)   // remotesapi on 127.0.0.1:5306 (grpc+http one port)
csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308
go rsrv.Serve(); go csrv.Serve(); srv.Run()

No redis, no email, no WithSchema in v1.

Routes (web/router.go): GET / dashboard; GET,POST /create (cookie required); GET /~{user}; GET /~{user}/{db} overview (branches, latest commits, clone box with both auth variants); GET /~{user}/{db}/log?branch=&from=; GET /~{user}/{db}/commit/{hash}; GET /~{user}/{db}/tree/{ref}; GET /~{user}/{db}/table/{ref}/{table}?page=; GET,POST /~{user}/{db}/settings (owner only: description/visibility/ACLs/delete); GET,POST /settings/keys (dolt-key association page — reads #<pubkey-base32> URL fragment via a few lines of inline JS to prefill the form, since dolt login appends the pubkey as a fragment); GET /static/*.

Chrome (web/chrome.go + templates): Go html/template port of core.sr.ht layout.html/nav.html — <nav class="container navbar navbar-light navbar-expand-sm">, brand [sr.ht]site-name + <span class="text-danger">dolt</span>, network list = config sections ending .sr.ht (minus paste/pages) linking config.GetOrigin(conf, site, true), active on self; login block → {meta-origin}/login?return_to=... / logout; environment banner when [sr.ht]environment != "production"; stylesheet href globbed from static-dir/main.min.*.css at startup. CSS: sassc -I $(ASSETS)/scss scss/main.scss → minify → main.min.<sha256[:8]>.css (Makefile cloned from git.sr.ht); logo.svg copied from core.sr.ht.

#Access matrix (core/access.go — pure, table-driven tests)

Ops: OpBrowse, OpCloneRead, OpPush, OpAdmin.

PRIVATE + unauthorized ⇒ NotFound (don't leak existence). Suspended users: reads allowed, push/admin denied. Token grants: dolt.sr.ht/repos:RO for reads, :RW for push (empty-grant personal tokens pass via HasAll).

#remotesapi subsystem

• Interceptors (remoteapi/interceptors.go): our own grpc.ChainUnaryInterceptor/ChainStreamInterceptor via ServerArgs.Options. Per RPC: classify method → extract authorization metadata → authn.ResolveGRPC(ctx, header):
  • Basic → username + pass; auth.DecodeBearerToken(pass) (offline HMAC) → require token owner == presented username → parallel LookupUser + LookupTokenRevocation (like auth.OAuth2) → positive results cached 60 s in-process keyed by sha512(pass) (a push issues many RPCs; bounds revocation lag), negatives uncached.
  • Bearer → parse JWS (go-jose, EdDSA): kid header → dolt_key row → verify signature with stored pubkey; check aud == our host, exp, sub == "doltClientCredentials/<kid>"; ignore iss. Local-only, no cache needed (30 s tokens); update last_used. Reuse creds.PubKeyToKIDStr from dolt.
  • absent → anonymous. Then repo path from GetRepoPath()/RepoId → core.ParseRepoPath (trim /, optional ~, reject ..) → load repo row → core.Allowed(caller, repo, op) → handler(authedCtx, req). Stream wrapper overrides Context() and checks each RecvMsg. Base ctx carries config.Context + database.Context.
• DBCache (storage/dbcache.go): Get(ctx, path, nbfVer) → require existing repository row (no push-to-create in v1; creation is explicit via web) → memoized nbs.NewLocalStore(ctx, nbfVer, absPath, 128MiB, nbs.NewUnlimitedMemQuotaProvider(), false) (same as upstream LocalCSCache); evict + close on repo delete.
• Assembly (remoteapi/server.go): remotesrv.NewServer(ServerArgs{HttpHost: "dolt.srht.bigb.es", HttpListenAddr == GrpcListenAddr: "127.0.0.1:5306", FS: filesys.LocalFS, DBCache, Options: ourInterceptors, ConcurrencyControl: PUSH_CONCURRENCY_CONTROL_IGNORE_WORKING_SET, ...}). Sealed-URL data plane needs no extra auth. Per-process sealer key ⇒ in-flight transfers break on restart (client retries; acceptable).
• CredentialsService (remoteapi/credsvc.go): own grpc.NewServer on :5308 registering remotesapi.RegisterCredentialsServiceServer; WhoAmI verifies the Bearer JWT exactly as above and returns Username/EmailAddress/DisplayName from the user row. dolt login polls WhoAmI until the key is associated via the web page.
• Create/delete flow (storage/init.go + web): validate name → tx INSERT repository → doltdb.LoadDoltDB(ctx, types.Format_Default, earl.FileUrlFromPath(absPath), filesys.LocalFS) + ddb.WriteEmptyRepo(ctx, "main", [sr.ht]owner-name, owner-email); remove dir on tx failure. Delete: tx delete row → os.RemoveAll → DBCache evict.

Keypair UX shipped in v1 (user-confirmed): users run dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys (or set remotes.default_host / creds.add_url once via dolt config --global); the settings page associates the posted pubkey with the logged-in SourceHut user; thereafter plain dolt clone/push works with no --user and no env var — like git SSH keys.

#Browse subsystem (browse/) — riskiest code, isolated

No SQL engine (can't open bare stores). Per request, read-only, closed after: doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, filesys.LocalFS) — NBS manifest-based readers are safe alongside the push writer.

• Branches: ddb.GetBranches(ctx); default main.
• Log: ResolveCommitRef + commitwalk.GetTopologicalOrderIterator (from go/libraries/doltcore/env/actions/commitwalk), paginated by count + start hash.
• Tables/schema: commit → RootValue → GetTableNames / GetTable(...).GetSchema().
• Rows: table.GetRowData → durable.ProllyMapFromIndex → prolly.Map.IterOrdinalRange(offset, offset+limit), rendered via schema types.
• Commit page: diff summaries only in v1 (per-table added/dropped/schema-changed
  • row-count delta via the diff package); full row diffs deferred.

Fallback if prolly iteration proves fragile across versions: keep a lightweight materialized clone per db (real .dolt, refreshed post-push) opened via dolthub/driver — swap contained inside browse/.

#Config + nginx

config.example.ini (goes into the single shared instance config.ini; existing services need only the [dolt.sr.ht] origin= line + restart to show us in nav):

[dolt.sr.ht]
origin=https://dolt.srht.bigb.es
connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
repos=/var/lib/dolt
remotesapi-listen=127.0.0.1:5306
credsapi-listen=127.0.0.1:5308
static-dir=/usr/share/sourcehut/dolt.sr.ht/static
migrate-on-upgrade=yes

Reused in place: [sr.ht] network-key / internal-ipnet / owner-* / site-name / environment; [webhooks]private-key; [meta.sr.ht]origin. Deployment checklist: our host must be in meta's internal-ipnet (FetchMetaProfile + LookupTokenRevocation use internal auth).

nginx (contrib/dolt.sr.ht.conf, sr.ht-nginx style):

server_name dolt.srht.bigb.es;  # TLS/http2 via sourcehut.conf + port443.conf
location /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ {
    grpc_pass grpc://127.0.0.1:5306;
    grpc_set_header X-Forwarded-Proto https;
    grpc_read_timeout 600s; grpc_send_timeout 600s; client_max_body_size 0;
}
location /dolt.services.remotesapi.v1alpha1.CredentialsService/ {
    grpc_pass grpc://127.0.0.1:5308;
    grpc_set_header X-Forwarded-Proto https;
}
location /single_symmetric_key_sealed_request/ {   # chunk data plane GET+PUT
    proxy_pass http://127.0.0.1:5306;
    client_max_body_size 0; proxy_request_buffering off; proxy_read_timeout 600s;
}
location / { proxy_pass http://127.0.0.1:5307; }   # web UI + /static

#Implementation phases (parallel-implementer convention from CLAUDE.md)

Phase 0 — foundation (serial, commit first): go.mod; pin ALL deps via go get (+ throwaway smoke-import build to populate go.sum): core-go@mirror-version, github.com/dolthub/dolt/go@latest-resolved pseudo-version (record it), go-jose, grpc, chi/v5, lib/pq, brant, logrus. Complete core/ package + unit tests; schema.sql + migrations/0001; config.example.ini; Makefile; nginx conf; README. Spike test (storage/spike_test.go, build-tagged): WriteEmptyRepo → remotesrv on localhost → dolt clone http://127.0.0.1:PORT/x/y with CLI v2.1.10 — proves format/RPC compat before any parallel work. Gate: go build ./... && go test ./core/....

Phase 1 — parallel wave A (disjoint packages; each implementer builds/tests only its own package, never ./...):

• 1a db/ — repository/access/dolt_key queries.
• 1b authn/ — cookie + Basic-token + Bearer-JWT resolution; tests forge tokens via auth.BearerToken.Encode() with synthesized ini (random fernet key + ed25519 seed); revocation stubbed behind an interface; JWT tests sign with ephemeral ed25519 keys.
• 1c storage/ — InitStore/DeleteStore/Cache (repo lookup injected as func).
• 1d browse/ — fixtures built programmatically via doltdb API.

Phase 2 — parallel wave B:

• 2a remoteapi/ — interceptors, server assembly, credsvc; integration test (-tags integration) driving real dolt CLI: clone → sql insert → commit → push → re-clone → verify; ACL denial; anonymous PUBLIC clone; Bearer-JWT clone after key association; PRIVATE anonymous ⇒ NotFound. (needs 1b, 1c; 1a via interface)
• 2b web/ — router/handlers/templates/chrome; httptest with stubbed interfaces. (needs 1a, 1b, 1d)

Phase 3 — serial finish: cmd/doltsrht, cmd/doltsrht-migrate, startup smoke test (/healthz), go mod tidy only now, CSS build against instance scss, README deployment section.

#Verification (post-deploy, end-to-end)

1. Add [dolt.sr.ht] section to shared config.ini, doltsrht-migrate init, deploy nginx conf + DNS dolt.srht.bigb.es, start doltsrht.
2. Browser: log into meta → visit dolt.srht.bigb.es → nav shows all services + "Logged in as bigbes"; nav of git/meta pages now shows "dolt".
3. Create db demo (PUBLIC) via /create.
4. Token flow: export DOLT_REMOTE_PASSWORD=<meta PAT>; dolt clone --user bigbes https://dolt.srht.bigb.es/~bigbes/demo; create table, insert, dolt commit, dolt push origin main.
5. Web: overview shows branch + commits; /log, /table/main/t shows the row; /commit/ shows table summary.
6. Keypair flow: dolt creds new; dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys; associate key in browser; dolt clone https://dolt.srht.bigb.es/~bigbes/demo (no --user) + push succeed.
7. Negative: anonymous clone of PUBLIC works; flip to PRIVATE → anonymous clone ⇒ not found; user2 token push ⇒ denied; grant RO → clone ok, push denied; revoke PAT on meta → push blocked within ≤60 s.

#Open risks

• dolt CLI v2.1.10 ⟷ pinned dolt/go module compat — de-risked by Phase-0 spike; worst case pin different module version or require newer CLI.
• prolly-level browse code is version-fragile — isolated in browse/ with materialized-clone fallback design.
• 60 s revocation-cache window for Basic tokens — accepted, documented.
• internal-ipnet / shared-key misconfig symptom: "Temporary error" on first login/push — deployment checklist item.