bigbes wants a DoltLab-equivalent integrated into the self-hosted SourceHut instance
(*.srht.bigb.es): a new service dolt.sr.ht that hosts Dolt databases the way
git.sr.ht hosts git repos — dolt clone/push/pull over HTTPS plus an integrated web
UI (shared nav, unified login via the sr.ht.unified-login.v1 cookie, shared
Bootstrap theme). Pure Go, single new repo at /Users/blikh/data/home/sourcehut-dolt
(currently empty). No upstream SourceHut modification — integration is config-driven
(a [dolt.sr.ht] config section puts us in every service's nav). DoltLab itself is
closed-source; everything we need is in Apache-2.0 dolthub/dolt packages.
v1 scope (user-confirmed): create/list/browse databases + clone/push over the
remotesapi, with both auth flows: username + meta personal access token
(--user + DOLT_REMOTE_PASSWORD, Basic) and dolt's Ed25519 keypair flow
(dolt creds/dolt login, Bearer EdDSA JWT — the git-SSH-key-like UX). No GraphQL
federation, no SSH transport, no webhooks, no issues/PRs.
All findings below were verified against dolthub/dolt main and the SourceHut mirror
at /Users/blikh/data/home/sourcehut (core-go/, core.sr.ht/, git.sr.ht/, sr.ht-nginx/).
One Go module sourcecraft.dev/bigbes/sr-ht-dolt, two binaries:
doltsrht — one process, three listeners:
-b localhost:5307ChunkStoreService + HTTP chunk data plane, h2c-multiplexed
single port) on 127.0.0.1:5306 via importable
github.com/dolthub/dolt/go/libraries/doltcore/remotesrvCredentialsService.WhoAmI (keypair login) on
127.0.0.1:5308 — nginx path-routes it, so no need to touch remotesrv internalsdoltsrht-migrate — brant migrations (git.sr.ht/~bitfehler/brant), modeled on
sourcehut-migrate/cmd/sourcehut-migrate/main.go.Storage: bare NBS chunk-store dirs (no .dolt/, no working set) at
<repos>/~<user>/<name> — exactly what remotesrv serves and file:// remotes use.
Postgres holds metadata (user mirror + repository + access + dolt_key).
Clone URL: dolt clone https://dolt.srht.bigb.es/~user/db.
Key verified facts that shape the design:
server.New (core-go) works without WithSchema (gqlgen-only). But
WithDefaultMiddleware attaches auth.Middleware, which 401s requests with no
cookie/header — kills anonymous browsing. → don't call it; assemble our own
middleware group on srv.AnonRouter().auth.OAuth2): auth.DecodeBearerToken
(offline HMAC via [webhooks]private-key-derived key) + auth.LookupUser (local
"user" table, lazily populated by FetchMetaProfile internal-GraphQL to meta) +
auth.LookupTokenRevocation (GraphQL tokenRevocationStatus on meta). Requires
shared [sr.ht]network-key, [webhooks]private-key, and our host inside meta's
internal-ipnet.go/libraries/doltcore/env/grpc_dial_provider.go):
--user ⇒ authorization: Basic b64(user:pass) with pass from
DOLT_REMOTE_PASSWORD; no --user but dolt creds/dolt login key present ⇒
authorization: Bearer <EdDSA JWT> (aud = our host, exp = now+30s, kid =
base32(SHA-512/224(pubkey)) custom alphabet 0123456789abcdefghijklmnopqrstuv,
iss hardcoded dolt-client.dolthub.com — do NOT reject on iss); no creds at all ⇒
no metadata ⇒ anonymous (must work for PUBLIC clones). Client is fully
host-agnostic: dolt login --auth-endpoint <host:443> --login-url <url>, defaults
settable via dolt config --global keys remotes.default_host, creds.add_url.remotesrv.ServerInterceptor drops the authenticated ctx and never sees
the repo path ⇒ per-repo ACLs impossible with it. → we write our own
unary+stream interceptors (keep upstream's method classification: write =
{GetUploadLocations, AddTableFiles, Commit}; read = the 8 others; unknown ⇒ deny)./dolt.services.remotesapi.v1alpha1.*/, sealed chunk URLs under
/single_symmetric_key_sealed_request/ (AES-GCM sealed, 15-min expiry, possession
= authorization), web UI owns the rest. getScheme() honors
x-forwarded-proto — nginx must set it on grpc_pass.doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, fs)
ddb.WriteEmptyRepo(ctx, "main", name, email). This is the opt-in path, not
the default one: dolt decides fast-forward on the client, so the initial commit
it writes makes the first push from any database with a history of its own a
non-fast-forward the server cannot forgive. Every creation path (web form,
/internal/repos, push-to-create) therefore defaults to InitEmptyStore — a bare
NBS store with no commits and no branches — and only the create form's
"initialize with an empty commit" checkbox reaches WriteEmptyRepo. The cost of an
empty store is that it cannot be cloned (ErrNoDataAtRemote), which is why the
overview of a database with no branches teaches push instead of clone./opt/homebrew/bin/dolt (integration tests).github.com/dolthub/dolt/go is a separate module pinned via pseudo-versions;
heavy deps, CGO (gozstd). Apache-2.0.go.mod Makefile schema.sql migrations/ config.example.ini README.md LICENSE
contrib/dolt.sr.ht.conf # nginx
scss/main.scss # @import "base" against core.sr.ht shared scss
static/ # logo.svg + main.min.<sha8>.css build output
cmd/doltsrht/main.go
cmd/doltsrht-migrate/main.go
core/ # PURE domain: names.go (validate, ParseRepoPath), access.go (matrix), models.go
db/ # postgres: repos.go, access.go, keys.go (dolt_key CRUD)
authn/ # ctx.go (Caller), cookie.go (optional unified-login middleware),
# token.go (Basic: PAT trio + 60s cache), jwt.go (Bearer: EdDSA verify)
storage/ # init.go (InitEmptyStore default / InitStore opt-in / Delete/MoveStore), dbcache.go (remotesrv.DBCache)
remoteapi/ # server.go (remotesrv assembly), interceptors.go, credsvc.go (WhoAmI grpc server)
browse/ # open.go, log.go, tables.go, diff.go — read-only doltdb over bare stores
web/ # router.go, handlers_*.go, templates.go, templates/*.html
# (nav/brand/login chrome comes from sr-ht-ecore/chrome)
Dependency direction: core ← all; db/authn/storage/browse mutually
independent; remoteapi ← {core, db, authn, storage}; web ← {core, db, authn,
browse, storage}; cmd ← all.
-- +brant Up migrations)CREATE TYPE visibility AS ENUM ('PUBLIC', 'PRIVATE', 'UNLISTED');
CREATE TYPE user_type AS ENUM ('PENDING', 'USER', 'ADMIN', 'SUSPENDED');
CREATE TYPE access_mode AS ENUM ('RO', 'RW');
CREATE TABLE "user" ( -- exact UserMixin shape; core-go FetchMetaProfile inserts rows
id integer PRIMARY KEY, -- meta's user id, inserted explicitly (not serial)
username varchar(256) UNIQUE,
created timestamp NOT NULL, updated timestamp NOT NULL,
email varchar(256) NOT NULL UNIQUE,
user_type user_type NOT NULL,
url varchar(256), location varchar(256), bio varchar(4096),
suspension_notice varchar(4096)
);
CREATE INDEX ix_user_username ON "user"(username);
CREATE TABLE repository (
id serial PRIMARY KEY,
created timestamp NOT NULL, updated timestamp NOT NULL,
name varchar(64) NOT NULL,
description varchar(1024),
owner_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
path varchar(1024) NOT NULL UNIQUE, -- absolute on-disk store dir
visibility visibility NOT NULL,
CONSTRAINT uq_repo_owner_id_name UNIQUE (owner_id, name)
);
CREATE TABLE access (
id serial PRIMARY KEY,
created timestamp NOT NULL, updated timestamp NOT NULL,
repo_id integer NOT NULL REFERENCES repository(id) ON DELETE CASCADE,
user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
mode access_mode NOT NULL,
CONSTRAINT uq_access_user_id_repo_id UNIQUE (user_id, repo_id)
);
CREATE TABLE dolt_key ( -- Ed25519 pubkeys for dolt creds auth
id serial PRIMARY KEY,
created timestamp NOT NULL,
user_id integer NOT NULL REFERENCES "user"(id) ON DELETE CASCADE,
kid varchar(64) NOT NULL UNIQUE, -- base32(SHA-512/224(pubkey)), dolt alphabet
pubkey bytea NOT NULL, -- 32-byte ed25519 public key
comment varchar(256),
last_used timestamp
);
DB name rule: ^[a-zA-Z0-9](?:[a-zA-Z0-9_-]*[a-zA-Z0-9])?$, max 64 (names double as
SQL database identifiers; reject ./..).
cmd/doltsrht/main.go)conf := config.LoadConfig()
srv := server.New("dolt.sr.ht", "localhost:5307", conf, os.Args[1:]) // runs crypto.InitCrypto
db := sql.Open("postgres", cfg[dolt.sr.ht]connection-string)
srv.AnonRouter().Group(func(r chi.Router) {
r.Use(chimw.RealIP, chimw.Recoverer)
r.Use(config.Middleware(conf, "dolt.sr.ht"), database.Middleware(db))
r.Use(authn.OptionalCookieMiddleware()) // never 401s; anonymous stays anonymous
web.Register(r)
})
rsrv := remoteapi.New(...) // remotesapi on 127.0.0.1:5306 (grpc+http one port)
csrv := remoteapi.NewCredSvc(...) // CredentialsService.WhoAmI on 127.0.0.1:5308
go rsrv.Serve(); go csrv.Serve(); srv.Run()
No redis, no email, no WithSchema in v1.
Routes (web/router.go): GET / dashboard; GET,POST /create (cookie required);
GET /~{user}; GET /~{user}/{db} overview (branches, latest commits, clone box
with both auth variants); GET /~{user}/{db}/log?branch=&from=;
GET /~{user}/{db}/commit/{hash}; GET /~{user}/{db}/tree/{ref};
GET /~{user}/{db}/table/{ref}/{table}?page=; GET,POST /~{user}/{db}/settings
(owner only: description/visibility/ACLs/delete); GET,POST /settings/keys
(dolt-key association page — reads #<pubkey-base32> URL fragment via a few lines of
inline JS to prefill the form, since dolt login appends the pubkey as a fragment);
GET /static/*.
Chrome (sourcecraft.dev/bigbes/sr-ht-ecore/chrome): the brand, the service
switcher, the login block, the environment banner and the database listing are
the shared package's, not ours — one chrome.Service built at startup from the
instance config with our section (dolt.sr.ht), one chrome.Page per request
embedded in each handler's view struct, and the srht-nav / srht-env-banner /
srht-repo-list partials attached to every template set by chrome.Attach. The
policy is ecore's: the switcher renders for authenticated viewers only, paste,
pages and hub never appear in it, and the profile link prefers hub's ~username
page. Ours is the layout document itself, the page-width choice
(Page.ContainerClass, container-fluid on the row browser) and the stylesheet
href, globbed from static-dir/main.min.*.css at startup because its name
carries a build hash. CSS: sassc -I $(ASSETS)/scss scss/main.scss → minify →
main.min.<sha256[:8]>.css (Makefile cloned from git.sr.ht); logo.svg copied
from core.sr.ht.
Ops: OpBrowse, OpCloneRead, OpPush, OpAdmin.
| Caller \ Visibility | PUBLIC | UNLISTED | PRIVATE |
|---|---|---|---|
| anonymous / any user | browse, clone | browse, clone (unlisted) | — |
| ACL RO | browse, clone | browse, clone | browse, clone |
| ACL RW | + push | + push | browse, clone, push |
| owner | all | all | all |
PRIVATE + unauthorized ⇒ NotFound (don't leak existence). Suspended users: reads
allowed, push/admin denied. Token grants: dolt.sr.ht/repos:RO for reads, :RW for
push (empty-grant personal tokens pass via HasAll).
grpc.ChainUnaryInterceptor/ChainStreamInterceptor via ServerArgs.Options.
Per RPC: classify method → extract authorization metadata →
authn.ResolveGRPC(ctx, header):
Basic → username + pass; auth.DecodeBearerToken(pass) (offline HMAC) →
require token owner == presented username → parallel LookupUser +
LookupTokenRevocation (like auth.OAuth2) → positive results cached 60 s
in-process keyed by sha512(pass) (a push issues many RPCs; bounds revocation
lag), negatives uncached.Bearer → parse JWS (go-jose, EdDSA): kid header → dolt_key row → verify
signature with stored pubkey; check aud == our host, exp,
sub == "doltClientCredentials/<kid>"; ignore iss. Local-only, no cache needed
(30 s tokens); update last_used. Reuse creds.PubKeyToKIDStr from dolt.GetRepoPath()/RepoId → core.ParseRepoPath (trim /,
optional ~, reject ..) → load repo row → core.Allowed(caller, repo, op) →
handler(authedCtx, req). Stream wrapper overrides Context() and checks each
RecvMsg. Base ctx carries config.Context + database.Context.Get(ctx, path, nbfVer) → require existing
repository row (no push-to-create in v1; creation is explicit via web) →
memoized nbs.NewLocalStore(ctx, nbfVer, absPath, 128MiB, nbs.NewUnlimitedMemQuotaProvider(), false) (same as upstream LocalCSCache);
evict + close on repo delete.remotesrv.NewServer(ServerArgs{HttpHost: "dolt.srht.bigb.es", HttpListenAddr == GrpcListenAddr: "127.0.0.1:5306", FS: filesys.LocalFS, DBCache, Options: ourInterceptors, ConcurrencyControl: PUSH_CONCURRENCY_CONTROL_IGNORE_WORKING_SET, ...}). Sealed-URL data plane needs
no extra auth. Per-process sealer key ⇒ in-flight transfers break on restart
(client retries; acceptable).grpc.NewServer on :5308
registering remotesapi.RegisterCredentialsServiceServer; WhoAmI verifies the
Bearer JWT exactly as above and returns Username/EmailAddress/DisplayName from the
user row. dolt login polls WhoAmI until the key is associated via the web page.doltdb.LoadDoltDB(ctx, types.Format_Default, earl.FileUrlFromPath(absPath), filesys.LocalFS) + ddb.WriteEmptyRepo(ctx, "main", [sr.ht]owner-name, owner-email); remove dir on tx failure. Delete: tx
delete row → os.RemoveAll → DBCache evict.Keypair UX shipped in v1 (user-confirmed): users run
dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys (or set remotes.default_host /
creds.add_url once via dolt config --global); the settings page associates the
posted pubkey with the logged-in SourceHut user; thereafter plain
dolt clone/push works with no --user and no env var — like git SSH keys.
No SQL engine (can't open bare stores). Per request, read-only, closed after:
doltdb.LoadDoltDB(ctx, types.Format_Default, fileURL, filesys.LocalFS) — NBS
manifest-based readers are safe alongside the push writer.
ddb.GetBranches(ctx); default main.ResolveCommitRef + commitwalk.GetTopologicalOrderIterator (from
go/libraries/doltcore/env/actions/commitwalk), paginated by count + start hash.RootValue → GetTableNames / GetTable(...).GetSchema().table.GetRowData → durable.ProllyMapFromIndex →
prolly.Map.IterOrdinalRange(offset, offset+limit), rendered via schema types.Fallback if prolly iteration proves fragile across versions: keep a lightweight
materialized clone per db (real .dolt, refreshed post-push) opened via
dolthub/driver — swap contained inside browse/.
config.example.ini (goes into the single shared instance config.ini; existing
services need only the [dolt.sr.ht] origin= line + restart to show us in nav):
[dolt.sr.ht]
origin=https://dolt.srht.bigb.es
connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
repos=/var/lib/dolt
remotesapi-listen=127.0.0.1:5306
credsapi-listen=127.0.0.1:5308
static-dir=/usr/share/sourcehut/dolt.sr.ht/static
migrate-on-upgrade=yes
Reused in place: [sr.ht] network-key / internal-ipnet / owner-* / site-name /
environment; [webhooks]private-key; [meta.sr.ht]origin. Deployment checklist:
our host must be in meta's internal-ipnet (FetchMetaProfile +
LookupTokenRevocation use internal auth).
nginx (contrib/dolt.sr.ht.conf, sr.ht-nginx style):
server_name dolt.srht.bigb.es; # TLS/http2 via sourcehut.conf + port443.conf
location /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ {
grpc_pass grpc://127.0.0.1:5306;
grpc_set_header X-Forwarded-Proto https;
grpc_read_timeout 600s; grpc_send_timeout 600s; client_max_body_size 0;
}
location /dolt.services.remotesapi.v1alpha1.CredentialsService/ {
grpc_pass grpc://127.0.0.1:5308;
grpc_set_header X-Forwarded-Proto https;
}
location /single_symmetric_key_sealed_request/ { # chunk data plane GET+PUT
proxy_pass http://127.0.0.1:5306;
client_max_body_size 0; proxy_request_buffering off; proxy_read_timeout 600s;
}
location / { proxy_pass http://127.0.0.1:5307; } # web UI + /static
Phase 0 — foundation (serial, commit first): go.mod; pin ALL deps via go get
(+ throwaway smoke-import build to populate go.sum): core-go@mirror-version,
github.com/dolthub/dolt/go@latest-resolved pseudo-version (record it),
go-jose, grpc, chi/v5, lib/pq, brant, logrus. Complete core/ package + unit
tests; schema.sql + migrations/0001; config.example.ini; Makefile; nginx conf;
README. Spike test (storage/spike_test.go, build-tagged): WriteEmptyRepo →
remotesrv on localhost → dolt clone http://127.0.0.1:PORT/x/y with CLI v2.1.10 —
proves format/RPC compat before any parallel work. Gate: go build ./... && go test ./core/....
Phase 1 — parallel wave A (disjoint packages; each implementer builds/tests only
its own package, never ./...):
db/ — repository/access/dolt_key queries.authn/ — cookie + Basic-token + Bearer-JWT resolution; tests forge tokens via
auth.BearerToken.Encode() with synthesized ini (random fernet key + ed25519
seed); revocation stubbed behind an interface; JWT tests sign with
ephemeral ed25519 keys.storage/ — InitStore/DeleteStore/Cache (repo lookup injected as func).browse/ — fixtures built programmatically via doltdb API.Phase 2 — parallel wave B:
remoteapi/ — interceptors, server assembly, credsvc; integration test
(-tags integration) driving real dolt CLI: clone → sql insert → commit → push →
re-clone → verify; ACL denial; anonymous PUBLIC clone; Bearer-JWT clone after key
association; PRIVATE anonymous ⇒ NotFound. (needs 1b, 1c; 1a via interface)web/ — router/handlers/templates/chrome; httptest with stubbed interfaces.
(needs 1a, 1b, 1d)Phase 3 — serial finish: cmd/doltsrht, cmd/doltsrht-migrate, startup smoke
test (/healthz), go mod tidy only now, CSS build against instance scss, README
deployment section.
[dolt.sr.ht] section to shared config.ini, doltsrht-migrate init, deploy
nginx conf + DNS dolt.srht.bigb.es, start doltsrht.demo (PUBLIC) via /create.export DOLT_REMOTE_PASSWORD=<meta PAT>; dolt clone --user bigbes https://dolt.srht.bigb.es/~bigbes/demo; create table, insert, dolt commit,
dolt push origin main.dolt creds new; dolt login --auth-endpoint dolt.srht.bigb.es:443 --login-url https://dolt.srht.bigb.es/settings/keys;
associate key in browser; dolt clone https://dolt.srht.bigb.es/~bigbes/demo
(no --user) + push succeed.dolt/go module compat — de-risked by Phase-0 spike;
worst case pin different module version or require newer CLI.browse/ with
materialized-clone fallback design.