~bigbes/sourcehut-dolt

d6f19a1f — Eugene Blikh 30 days ago 
build: source core-go from the instance fork
b4d6a3e6 — Eugene Blikh 30 days ago 
docs: add design spec
e518fcb9 — Eugene Blikh 30 days ago 
cmd: doltsrht and doltsrht-migrate binaries, module tidy

refs

main
browse  log 

clone

read-only
https://git.srht.bigb.es/~bigbes/sourcehut-dolt
read/write
git@git.srht.bigb.es:~bigbes/sourcehut-dolt

You can also use your local clone with git send-email.

#dolt.sr.ht

A self-hosted Dolt database hosting service for a SourceHut instance — "DoltLab for SourceHut". It hosts Dolt databases the way git.sr.ht hosts git repos: dolt clone/push/pull over HTTPS plus an integrated web UI that shares the SourceHut nav, unified login, and Bootstrap theme.

Pure Go, one module (go.bigb.es/sourcehut-dolt), no upstream SourceHut modification — integration is config-driven: a [dolt.sr.ht] section in the shared instance config.ini puts the service into every other service's nav.

Storage is bare NBS chunk-store directories (no .dolt/, no working set) at <repos>/~<user>/<name> — exactly what remotesrv serves and file:// remotes use. PostgreSQL holds metadata (a mirror of meta's users, plus repositories, ACLs, and dolt keys). Clone URL: dolt clone https://dolt.srht.bigb.es/~user/db.

Two auth flows are supported: username + meta personal access token (--user + DOLT_REMOTE_PASSWORD, HTTP Basic) and dolt's Ed25519 keypair flow (dolt creds / dolt login, Bearer EdDSA JWT — the git-SSH-key-like UX).

#Status

Feature-complete for the v1 scope: the pure core/ domain (name/path validation and the access matrix), the db/ Postgres layer, the authn/ auth stack (unified-login cookie, meta PAT Basic auth, dolt-keypair Bearer JWT), the storage/ bare-store lifecycle and remotesapi DBCache, the remoteapi/ remotesrv assembly plus CredentialsService, the read-only browse/ UI, the web/ router/chrome, and the two binaries (doltsrht, doltsrht-migrate).

#Build prerequisites

  • Go 1.26+

  • A C toolchain — github.com/dolthub/dolt/go uses CGO for gozstd and go-icu-regex.

  • ICU4C development headers — required by go-icu-regex.

    • Debian/Ubuntu: apt install libicu-dev (headers on the default path).

    • macOS (Homebrew): brew install icu4c installs a keg-only, versioned formula. Point CGO at it, e.g. for icu4c@78:

      export CGO_CPPFLAGS="-I/opt/homebrew/opt/icu4c@78/include"
export CGO_LDFLAGS="-L/opt/homebrew/opt/icu4c@78/lib"

  • sassc + minify — only for building CSS (make css); not needed for the default build:

    brew install sassc                                   # or apt install sassc
go install github.com/tdewolff/minify/v2/cmd/minify@latest

    CSS is compiled against the shared sourcehut SCSS: make css ASSETS=/path/to/sourcehut/scss/parent (ASSETS defaults to /usr/share/sourcehut; sassc is invoked with -I $(ASSETS)/scss).

#Pinned dependencies (and why)

  • git.sr.ht/~sircmpwn/core-go, replaced by the instance fork git.srht.bigb.es/~bigbes/core-go (pinned via a replace directive to the fork's master, currently c2c2f38 = upstream + instance patches). The fork is what production actually runs; token validation, config, and crypto must behave identically to the rest of the instance. Never go get -u it and never drop the replace; to bump, pin the fork's new commit pseudo-version in the replace line and re-run the test suite. Fetching the fork needs GOPRIVATE=git.srht.bigb.es (skips the public module proxy/sumdb).
  • github.com/dolthub/dolt/go v0.40.5-0.20260626152440-45335d44ad79 — a pseudo-version pinned to the commit tagged v2.1.10 (45335d44), the dolt CLI version installed on the target host (/opt/homebrew/bin/dolt, v2.1.10). The dolt /go submodule's latest tag is the stale v0.40.4 (2021), which does not interop with a modern CLI; matching the CLI's commit guarantees a common NBS storage format (types.Format_DOLT / __DOLT__) and remotesapi proto. Verified end-to-end by the Phase-0 spike (see below). If the CLI is upgraded, re-pin dolt/go to the new CLI's commit and re-run the spike.
  • gopkg.in/go-jose/go-jose.v2 v2.6.3 — the same JOSE major/version that dolt/go's creds package uses to sign the EdDSA keypair JWTs, so the Bearer verify path stays byte-compatible and no duplicate JOSE lib is pulled in.
  • grpc v1.79.3, logrus v1.8.3, lib/pq v1.10.9, chi/v5, and brant round out the transport, logging, Postgres driver, HTTP router, and migration tooling.

#The spike

storage/spike_test.go (build tag spike) is the Phase-0 de-risk gate. It inits a bare NBS store via doltdb.LoadDoltDB + WriteEmptyRepo, serves it with remotesrv.NewServer on an ephemeral localhost port (single-port http+gRPC multiplex, no auth), then drives the real dolt CLI through a full round-trip: clone → create table + insert → commit → push → fresh re-clone → verify the rows. It skips (does not fail) when the CLI is absent, and uses an isolated $HOME so your real dolt config is untouched.

export CGO_CPPFLAGS="-I/opt/homebrew/opt/icu4c@78/include"
export CGO_LDFLAGS="-L/opt/homebrew/opt/icu4c@78/lib"
go test -tags spike ./storage/ -run TestSpike -v

#Dev commands

go build ./...                 # build every package (needs the CGO env above)
go test ./...                  # full suite (db/ + remoteapi/ skip without Docker)
go test -tags spike ./storage/ -v         # the interop spike (needs dolt CLI)
go test -tags integration ./remoteapi/    # clone/push round-trips (needs Docker)
make                           # build both binaries into ./doltsrht[-migrate]
make css                       # build stylesheets (needs sassc + minify)

#Deployment

#1. Config

dolt.sr.ht reads the single shared instance config.ini (the same file every *.sr.ht service reads). Add a [dolt.sr.ht] section — the full block, copied from config.example.ini:

[dolt.sr.ht]
; External URL. Also the JWT audience checked in the `dolt creds` keypair flow,
; so it must match the host clients pass to `dolt login --auth-endpoint`.
origin=https://dolt.srht.bigb.es
; Postgres connection string for the metadata database.
connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
; Root dir holding the bare NBS chunk-store dirs (<repos>/~<user>/<name>).
repos=/var/lib/dolt
; remotesapi listener (gRPC ChunkStoreService + HTTP chunk data plane, h2c).
remotesapi-listen=127.0.0.1:5306
; CredentialsService.WhoAmI listener (the `dolt login` keypair flow).
credsapi-listen=127.0.0.1:5308
; Directory of built static assets (main.min.<sha>.css, logo.svg).
static-dir=/usr/share/sourcehut/dolt.sr.ht/static
; Run brant migrations automatically on package upgrade.
migrate-on-upgrade=yes

connection-string and origin are required (doltsrht fails fast if either is missing); everything else has the default shown above.

The service also reuses these shared keys owned by other services — do not duplicate their values, they already live in the shared config.ini:

Key Used for
[sr.ht]network-key internal service-to-service auth (crypto.InitCrypto)
[sr.ht]owner-name / owner-email author of each database's initial empty commit
[sr.ht]site-name shared nav brand
[sr.ht]environment a non-production value adds a banner to every page
[webhooks]private-key derives the offline HMAC key that validates meta PATs
[meta.sr.ht]origin login redirects and profile fetches
[meta.sr.ht::api]internal-ipnet subnets allowed to use meta's internal auth

Deployment prerequisite: this host must be inside meta's internal-ipnet. Token validation (FetchMetaProfile + LookupTokenRevocation) and cookie resolution use internal auth against meta; a misconfig here surfaces as a "Temporary error" on the first login or push.

Adding the [dolt.sr.ht] origin= line to a shared config and restarting the other services is all that is needed for dolt.sr.ht to appear in their nav.

#2. Database + migrations

doltsrht-migrate is a single-service brant wrapper. It reads connection-string from [dolt.sr.ht] (override with --dsn) and loads migrations from ./migrations in a checkout or the installed /usr/share/sourcehut/migrations/dolt.sr.ht otherwise.

createdb dolt.sr.ht
doltsrht-migrate init        # apply schema.sql wholesale, stamp to head (fresh install)
doltsrht-migrate up          # apply pending migrations/*.sql (upgrades)
doltsrht-migrate current     # print the current schema version
doltsrht-migrate -a up       # honor migrate-on-upgrade; no-op when disabled

Use init once on a brand-new database; use up for every subsequent upgrade.

#3. nginx + DNS

Install contrib/dolt.sr.ht.conf into the nginx sites directory alongside the other *.sr.ht.conf files (TLS/http2 and the shared proxy snippets come from the included sourcehut.conf / port443.conf). It path-routes four back ends on one server_name dolt.srht.bigb.es:

  • /dolt.services.remotesapi.v1alpha1.ChunkStoreService/ → grpc://127.0.0.1:5306 (clone/pull/push RPCs; sets X-Forwarded-Proto https so the server hands back https:// sealed chunk URLs).
  • /dolt.services.remotesapi.v1alpha1.CredentialsService/ → grpc://127.0.0.1:5308 (the dolt login keypair WhoAmI).
  • /single_symmetric_key_sealed_request/ → http://127.0.0.1:5306 (the AES-GCM sealed chunk data plane; client_max_body_size 0, buffering off).
  • / → http://127.0.0.1:5307 (web UI + /static).

Add a dolt.srht.bigb.es DNS record pointing at the instance.

#4. systemd unit sketch

[Unit]
Description=dolt.sr.ht service
After=network.target postgresql.service

[Service]
User=dolt
ExecStart=/usr/local/bin/doltsrht
Restart=on-failure
# The bare NBS stores live here; the user must own it.
StateDirectory=dolt

[Install]
WantedBy=multi-user.target

doltsrht binds all three listeners (web -b localhost:5307, remotesapi 127.0.0.1:5306, credentials 127.0.0.1:5308) and shuts them down cleanly on SIGINT/SIGTERM. Point [dolt.sr.ht]repos at the StateDirectory (/var/lib/dolt).

#User quickstart

Once the service is up and you are logged into meta:

  1. Create a database. Visit https://dolt.srht.bigb.es, click Create, pick a name and visibility (PUBLIC / UNLISTED / PRIVATE). It appears at ~<you>/<name>.

  2. Clone/push with a meta personal access token (PAT):

    export DOLT_REMOTE_PASSWORD=<your meta PAT>
dolt clone --user <you> https://dolt.srht.bigb.es/~<you>/<name>
cd <name>
dolt sql -q 'CREATE TABLE t (id INT PRIMARY KEY)'
dolt add . && dolt commit -m 'init'
dolt push origin main

    --user selects HTTP Basic auth; the password comes from DOLT_REMOTE_PASSWORD. A PAT with the dolt.sr.ht/repos:RW grant (or an empty-grant personal token) can push; :RO or no grant can only read.

  3. Or clone/push with a dolt keypair (the git-SSH-key-like UX):

    dolt creds new          # generate an Ed25519 keypair
dolt login --auth-endpoint dolt.srht.bigb.es:443 \
           --login-url https://dolt.srht.bigb.es/settings/keys

    dolt login opens the settings page with your public key in the URL fragment; associate it with your account, and dolt login confirms via the CredentialsService. Thereafter plain dolt clone/push (no --user, no env var) works. To make those the defaults so you can drop the flags:

    dolt config --global --add remotes.default_host dolt.srht.bigb.es
dolt config --global --add creds.add_url https://dolt.srht.bigb.es/settings/keys

Anonymous dolt clone works for PUBLIC and UNLISTED databases with no credentials at all; PRIVATE databases return "not found" to unauthorized callers (their existence is not leaked).