~bigbes/sr-ht-dolt

ref: ca9357a0d245b7f8ce3256f9f4524f83925a031e sr-ht-dolt/README.md -rw-r--r-- 5.5 KiB
ca9357a0 — Eugene Blikh foundation: module, core domain, schema, spike-verified dolt interop 30 days ago

#dolt.sr.ht

A self-hosted Dolt database hosting service for a SourceHut instance — "DoltLab for SourceHut". It hosts Dolt databases the way git.sr.ht hosts git repos: dolt clone/push/pull over HTTPS plus an integrated web UI that shares the SourceHut nav, unified login, and Bootstrap theme.

Pure Go, one module (go.bigb.es/sourcehut-dolt), no upstream SourceHut modification — integration is config-driven: a [dolt.sr.ht] section in the shared instance config.ini puts the service into every other service's nav.

Storage is bare NBS chunk-store directories (no .dolt/, no working set) at <repos>/~<user>/<name> — exactly what remotesrv serves and file:// remotes use. PostgreSQL holds metadata (a mirror of meta's users, plus repositories, ACLs, and dolt keys). Clone URL: dolt clone https://dolt.srht.bigb.es/~user/db.

Two auth flows are supported: username + meta personal access token (--user + DOLT_REMOTE_PASSWORD, HTTP Basic) and dolt's Ed25519 keypair flow (dolt creds / dolt login, Bearer EdDSA JWT — the git-SSH-key-like UX).

#Status

Phase 0 (foundation) only: the pure core/ domain (name/path validation and the access matrix), the SQL schema and brant migration, config/nginx/Makefile scaffolding, and the de-risking storage spike. The service binaries, database layer, auth, remotesapi, and web UI land in later phases.

#Build prerequisites

  • Go 1.26+

  • A C toolchain — github.com/dolthub/dolt/go uses CGO for gozstd and go-icu-regex.

  • ICU4C development headers — required by go-icu-regex.

    • Debian/Ubuntu: apt install libicu-dev (headers on the default path).

    • macOS (Homebrew): brew install icu4c installs a keg-only, versioned formula. Point CGO at it, e.g. for icu4c@78:

      export CGO_CPPFLAGS="-I/opt/homebrew/opt/icu4c@78/include"
export CGO_LDFLAGS="-L/opt/homebrew/opt/icu4c@78/lib"

  • sassc + minify — only for building CSS (make css); not needed for the default build:

    brew install sassc                                   # or apt install sassc
go install github.com/tdewolff/minify/v2/cmd/minify@latest

    CSS is compiled against the shared sourcehut SCSS: make css ASSETS=/path/to/sourcehut/scss/parent (ASSETS defaults to /usr/share/sourcehut; sassc is invoked with -I $(ASSETS)/scss).

#Pinned dependencies (and why)

  • git.sr.ht/~sircmpwn/core-go v0.0.0-20260520082310-fdb3662452dc — this exact pseudo-version matches the production instance's core-go submodule commit (fdb3662). Never upgrade it (no bare go get -u, no @latest); token validation, config, and crypto must behave identically to the rest of the instance.
  • github.com/dolthub/dolt/go v0.40.5-0.20260626152440-45335d44ad79 — a pseudo-version pinned to the commit tagged v2.1.10 (45335d44), the dolt CLI version installed on the target host (/opt/homebrew/bin/dolt, v2.1.10). The dolt /go submodule's latest tag is the stale v0.40.4 (2021), which does not interop with a modern CLI; matching the CLI's commit guarantees a common NBS storage format (types.Format_DOLT / __DOLT__) and remotesapi proto. Verified end-to-end by the Phase-0 spike (see below). If the CLI is upgraded, re-pin dolt/go to the new CLI's commit and re-run the spike.
  • gopkg.in/go-jose/go-jose.v2 v2.6.3 — the same JOSE major/version that dolt/go's creds package uses to sign the EdDSA keypair JWTs, so the Bearer verify path stays byte-compatible and no duplicate JOSE lib is pulled in.
  • grpc v1.79.3, logrus v1.8.3, lib/pq v1.10.9, chi/v5, and brant round out the transport, logging, Postgres driver, HTTP router, and migration tooling.

#The spike

storage/spike_test.go (build tag spike) is the Phase-0 de-risk gate. It inits a bare NBS store via doltdb.LoadDoltDB + WriteEmptyRepo, serves it with remotesrv.NewServer on an ephemeral localhost port (single-port http+gRPC multiplex, no auth), then drives the real dolt CLI through a full round-trip: clone → create table + insert → commit → push → fresh re-clone → verify the rows. It skips (does not fail) when the CLI is absent, and uses an isolated $HOME so your real dolt config is untouched.

export CGO_CPPFLAGS="-I/opt/homebrew/opt/icu4c@78/include"
export CGO_LDFLAGS="-L/opt/homebrew/opt/icu4c@78/lib"
go test -tags spike ./storage/ -run TestSpike -v

#Dev commands

go build ./...                 # build every package (needs the CGO env above)
go test ./core/...             # pure-domain unit tests
go test -tags spike ./storage/ -v   # the interop spike
make                           # build the service binaries (no-ops until Phase 3)
make css                       # build stylesheets (needs sassc + minify)

#Deployment sketch

  1. Add a [dolt.sr.ht] section to the shared instance config.ini (see config.example.ini) plus an origin= line so the service appears in every other service's nav. This host must be inside meta's internal-ipnet (token validation uses internal auth against meta).
  2. Create the database and run migrations: doltsrht-migrate applies migrations/*.sql (brant format). schema.sql is the full init DDL.
  3. Install the nginx site config (contrib/dolt.sr.ht.conf) and add a dolt.srht.bigb.es DNS record. nginx path-routes the remotesapi gRPC, the sealed-URL chunk data plane, the CredentialsService, and the web UI to separate listeners.
  4. Start doltsrht.