# dolt.sr.ht

A self-hosted [Dolt](https://www.dolthub.com/) database hosting service for a
SourceHut instance — "DoltLab for SourceHut". It hosts Dolt databases the way
git.sr.ht hosts git repos: `dolt clone`/`push`/`pull` over HTTPS plus an
integrated web UI that shares the SourceHut nav, unified login, and Bootstrap
theme.

Pure Go, one module (`sourcecraft.dev/bigbes/sr-ht-dolt`), no upstream SourceHut
modification — integration is config-driven: a `[dolt.sr.ht]` section in the
shared instance `config.ini` puts the service into every other service's nav.

Storage is bare NBS chunk-store directories (no `.dolt/`, no working set) at
`<repos>/~<user>/<name>` — exactly what `remotesrv` serves and `file://`
remotes use. PostgreSQL holds metadata (a mirror of meta's users, plus
repositories, ACLs, and dolt keys). Clone URL:
`dolt clone https://dolt.srht.bigb.es/~user/db`.

Two auth flows are supported for `dolt clone`/`push`: username + meta personal
access token (`--user` + `DOLT_REMOTE_PASSWORD`, HTTP Basic) and dolt's Ed25519
keypair flow (`dolt creds` / `dolt login`, Bearer EdDSA JWT — the
git-SSH-key-like UX). Agents get a third door, the MCP surface at `/mcp`
described at the end of this file.

## Status

Feature-complete for the v1 scope: the pure `core/` domain (name/path
validation and the access matrix), the `db/` Postgres layer, the `authn/` auth
stack (unified-login cookie, meta PAT Basic auth, dolt-keypair Bearer JWT), the
`storage/` bare-store lifecycle and remotesapi DBCache, the `remoteapi/`
remotesrv assembly plus CredentialsService, the read-only `browse/` UI, the
`web/` router/chrome, and the two binaries (`doltsrht`, `doltsrht-migrate`).

## Build prerequisites

- **Go 1.26+**
- **No C toolchain, no ICU, no zstd headers.** The default build is pure Go
  (`CGO_ENABLED=0`, statically linkable). `github.com/dolthub/dolt/go` normally
  needs CGO for two libraries; both are avoided:
  - **ICU regex** — the `gms_pure_go` build tag selects go-mysql-server's stdlib
    `regexp` fallback instead of `go-icu-regex`. Safe here because this service
    never runs the SQL engine (it serves bare NBS stores and browses read-only),
    so it never evaluates SQL `REGEXP`.
  - **zstd** — a `replace github.com/dolthub/gozstd => ./third_party/gozstd-purego`
    directive backs dolt's zstd dependency with a pure-Go shim over
    [`klauspost/compress/zstd`](https://github.com/klauspost/compress) (see that
    directory's README/tests, incl. libzstd interop). dolt itself is unmodified.

  `make` and `make build` pass `-tags gms_pure_go` and `CGO_ENABLED=0` for you; a
  bare `go build` needs `-tags gms_pure_go`.

  - **Optional cgo variant** (upstream gozstd + ICU): `make CGO_ENABLED=1 GO_TAGS=`.
    It then needs a C toolchain and ICU4C headers — Debian/Ubuntu
    `apt install libicu-dev`; macOS keg-only `brew install icu4c` with
    `CGO_CPPFLAGS="-I/opt/homebrew/opt/icu4c@78/include"` /
    `CGO_LDFLAGS="-L/opt/homebrew/opt/icu4c@78/lib"`.

- **sassc + minify** — only for building CSS (`make css`); not needed for the
  default build:

  ```sh
  brew install sassc                                   # or apt install sassc
  go install github.com/tdewolff/minify/v2/cmd/minify@latest
  ```

  CSS is compiled against the shared sourcehut SCSS: `make css ASSETS=/path/to/sourcehut/scss/parent`
  (`ASSETS` defaults to `/usr/share/sourcehut`; `sassc` is invoked with
  `-I $(ASSETS)/scss`).

## Pinned dependencies (and why)

- **`sourcecraft.dev/bigbes/sr-ht-core`, replaced by the instance fork
  `git.srht.bigb.es/~bigbes/core-go`** (pinned via a `replace` directive to the
  fork's `master`, currently `c2c2f38` = upstream + instance patches). The fork
  is what production actually runs; token validation, config, and crypto must
  behave identically to the rest of the instance. **Never `go get -u` it and
  never drop the replace**; to bump, pin the fork's new commit pseudo-version
  in the `replace` line and re-run the test suite. Fetching the fork needs
  `GOPRIVATE=git.srht.bigb.es` (skips the public module proxy/sumdb).
- **`github.com/dolthub/dolt/go` v0.40.5-0.20260626152440-45335d44ad79** — a
  pseudo-version pinned to the commit tagged **v2.1.10** (`45335d44`), the dolt
  CLI version installed on the target host (`/opt/homebrew/bin/dolt`, v2.1.10).
  The dolt `/go` submodule's latest *tag* is the stale `v0.40.4` (2021), which
  does **not** interop with a modern CLI; matching the CLI's commit guarantees a
  common NBS storage format (`types.Format_DOLT` / `__DOLT__`) and remotesapi
  proto. Verified end-to-end by the Phase-0 spike (see below). If the CLI is
  upgraded, re-pin `dolt/go` to the new CLI's commit and re-run the spike.
- **`gopkg.in/go-jose/go-jose.v2` v2.6.3** — the same JOSE major/version that
  `dolt/go`'s `creds` package uses to sign the EdDSA keypair JWTs, so the Bearer
  verify path stays byte-compatible and no duplicate JOSE lib is pulled in.
- **`github.com/dolthub/gozstd`, replaced by the local `./third_party/gozstd-purego`
  shim** — a pure-Go, drop-in reimplementation of the nine gozstd symbols dolt
  references (Compress/CompressDict/Decompress/DecompressDict/BuildDict, the
  CDict/DDict types and their constructors), backed by
  `github.com/klauspost/compress/zstd` (v1.18.0, already in the graph). This is
  what lets the default build be `CGO_ENABLED=0`. dolt uses gozstd only from its
  NBS archive subsystem; this service only ever hits the *decompress* side at
  runtime, and zstd frames/dictionaries are standard-format, so libzstd-authored
  archives decode correctly (proven by the shim's libzstd-interop tests). Keep
  the `replace`; to drop it, build the cgo variant (see Build prerequisites).
- grpc v1.79.3, logrus v1.8.3, lib/pq v1.10.9, chi/v5, and brant round out the
  transport, logging, Postgres driver, HTTP router, and migration tooling.

## The spike

`storage/spike_test.go` (build tag `spike`) is the Phase-0 de-risk gate. It
inits a bare NBS store via `doltdb.LoadDoltDB` + `WriteEmptyRepo`, serves it
with `remotesrv.NewServer` on an ephemeral localhost port (single-port
http+gRPC multiplex, no auth), then drives the real `dolt` CLI through a full
round-trip: clone → create table + insert → commit → push → fresh re-clone →
verify the rows. It skips (does not fail) when the CLI is absent, and uses an
isolated `$HOME` so your real dolt config is untouched.

```sh
go test -tags 'gms_pure_go spike' ./storage/ -run TestSpike -v
```

## Dev commands

```sh
go build ./...                 # build every package (needs the CGO env above)
go test ./...                  # full suite (db/ + remoteapi/ skip without Docker)
go test -tags spike ./storage/ -v         # the interop spike (needs dolt CLI)
go test -tags integration ./remoteapi/    # clone/push round-trips (needs Docker)
make                           # build both binaries into ./doltsrht[-migrate]
make css                       # build stylesheets (needs sassc + minify)
```

## Deployment

### 1. Config

dolt.sr.ht reads the single shared instance `config.ini` (the same file every
`*.sr.ht` service reads). Add a `[dolt.sr.ht]` section — the full block, copied
from `config.example.ini`:

```ini
[dolt.sr.ht]
; External URL. Also the JWT audience checked in the `dolt creds` keypair flow,
; so it must match the host clients pass to `dolt login --auth-endpoint`.
origin=https://dolt.srht.bigb.es
; Postgres connection string for the metadata database.
connection-string=postgresql://doltsrht@localhost/dolt.sr.ht?sslmode=disable
; Root dir holding the bare NBS chunk-store dirs (<repos>/~<user>/<name>).
repos=/var/lib/dolt
; remotesapi listener (gRPC ChunkStoreService + HTTP chunk data plane, h2c).
remotesapi-listen=127.0.0.1:5306
; CredentialsService.WhoAmI listener (the `dolt login` keypair flow).
credsapi-listen=127.0.0.1:5308
; Directory of built static assets (main.min.<sha>.css, logo.svg).
static-dir=/usr/share/sourcehut/dolt.sr.ht/static
; Run brant migrations automatically on package upgrade.
migrate-on-upgrade=yes
```

`connection-string` and `origin` are **required** (`doltsrht` fails fast if
either is missing); everything else has the default shown above.

The service also reuses these **shared keys owned by other services** — do not
duplicate their values, they already live in the shared `config.ini`:

| Key | Used for |
|---|---|
| `[sr.ht]network-key` | internal service-to-service auth (`crypto.InitCrypto`) |
| `[sr.ht]owner-name` / `owner-email` | author of each database's initial empty commit |
| `[sr.ht]site-name` | shared nav brand |
| `[sr.ht]environment` | a non-`production` value adds a banner to every page |
| `[webhooks]private-key` | derives the offline HMAC key that validates meta PATs |
| `[meta.sr.ht]origin` | login redirects and profile fetches |
| `[meta.sr.ht::api]internal-ipnet` | subnets allowed to use meta's internal auth |

> **Deployment prerequisite:** this host **must** be inside meta's
> `internal-ipnet`. Token validation (`FetchMetaProfile` +
> `LookupTokenRevocation`) and cookie resolution use internal auth against meta;
> a misconfig here surfaces as a "Temporary error" on the first login or push.

Adding the `[dolt.sr.ht] origin=` line to a shared config and restarting the
other services is all that is needed for dolt.sr.ht to appear in their nav.

### 2. Database + migrations

`doltsrht-migrate` is a single-service brant wrapper. It reads
`connection-string` from `[dolt.sr.ht]` (override with `--dsn`) and loads
migrations from `./migrations` in a checkout or the installed
`/usr/share/sourcehut/migrations/dolt.sr.ht` otherwise.

```sh
createdb dolt.sr.ht
doltsrht-migrate init        # apply schema.sql wholesale, stamp to head (fresh install)
doltsrht-migrate up          # apply pending migrations/*.sql (upgrades)
doltsrht-migrate current     # print the current schema version
doltsrht-migrate -a up       # honor migrate-on-upgrade; no-op when disabled
```

Use `init` once on a brand-new database; use `up` for every subsequent upgrade.

### 3. nginx + DNS

Install `contrib/dolt.sr.ht.conf` into the nginx sites directory alongside the
other `*.sr.ht.conf` files (TLS/http2 and the shared proxy snippets come from
the included `sourcehut.conf` / `port443.conf`). It path-routes four back ends
on one `server_name dolt.srht.bigb.es`:

- `/dolt.services.remotesapi.v1alpha1.ChunkStoreService/` → `grpc://127.0.0.1:5306`
  (clone/pull/push RPCs; sets `X-Forwarded-Proto https` so the server hands back
  `https://` sealed chunk URLs).
- `/dolt.services.remotesapi.v1alpha1.CredentialsService/` → `grpc://127.0.0.1:5308`
  (the `dolt login` keypair WhoAmI).
- `/single_symmetric_key_sealed_request/` → `http://127.0.0.1:5306` (the AES-GCM
  sealed chunk data plane; `client_max_body_size 0`, buffering off).
- `/` → `http://127.0.0.1:5307` (web UI + `/static`).

Add a `dolt.srht.bigb.es` DNS record pointing at the instance.

### 4. systemd unit sketch

```ini
[Unit]
Description=dolt.sr.ht service
After=network.target postgresql.service

[Service]
User=dolt
ExecStart=/usr/local/bin/doltsrht
Restart=on-failure
# The bare NBS stores live here; the user must own it.
StateDirectory=dolt

[Install]
WantedBy=multi-user.target
```

`doltsrht` binds all three listeners (web `-b localhost:5307`, remotesapi
`127.0.0.1:5306`, credentials `127.0.0.1:5308`) and shuts them down cleanly on
`SIGINT`/`SIGTERM`. Point `[dolt.sr.ht]repos` at the `StateDirectory`
(`/var/lib/dolt`).

## User quickstart

Once the service is up and you are logged into meta:

1. **Create a database.** Visit `https://dolt.srht.bigb.es`, click **Create**,
   pick a name and visibility (PUBLIC / UNLISTED / PRIVATE). It appears at
   `~<you>/<name>`.

2. **Clone/push with a meta personal access token (PAT):**

   ```sh
   export DOLT_REMOTE_PASSWORD=<your meta PAT>
   dolt clone --user <you> https://dolt.srht.bigb.es/~<you>/<name>
   cd <name>
   dolt sql -q 'CREATE TABLE t (id INT PRIMARY KEY)'
   dolt add . && dolt commit -m 'init'
   dolt push origin main
   ```

   `--user` selects HTTP Basic auth; the password comes from
   `DOLT_REMOTE_PASSWORD`. A PAT with the `dolt.sr.ht/repos:RW` grant (or an
   empty-grant personal token) can push; `:RO` or no grant can only read.

3. **Or clone/push with a dolt keypair (the git-SSH-key-like UX):**

   ```sh
   dolt creds new          # generate an Ed25519 keypair
   dolt login --auth-endpoint dolt.srht.bigb.es:443 \
              --login-url https://dolt.srht.bigb.es/settings/keys
   ```

   `dolt login` opens the settings page with your public key in the URL
   fragment; associate it with your account, and `dolt login` confirms via the
   CredentialsService. Thereafter plain `dolt clone/push` (no `--user`, no env
   var) works. To make those the defaults so you can drop the flags:

   ```sh
   dolt config --global --add remotes.default_host dolt.srht.bigb.es
   dolt config --global --add creds.add_url https://dolt.srht.bigb.es/settings/keys
   ```

4. **Rename a database.** `~<you>/<name>` → **settings** → **Rename**. The
   record and the stored data move together, and no redirect is left behind: the
   old address stops resolving, so an existing checkout needs its remote
   replaced (`dolt remote` has no `set-url`).

   ```sh
   dolt remote remove origin
   dolt remote add origin https://dolt.srht.bigb.es/~<you>/<new-name>
   ```

   A database provisioned as the companion of a git.sr.ht repository is
   re-created under its old name by the next push to that repository — the hook
   provisions by the *git* repo's name, which renaming here does not change.

Anonymous `dolt clone` works for PUBLIC and UNLISTED databases with no
credentials at all; PRIVATE databases return "not found" to unauthorized
callers (their existence is not leaked).

## The MCP surface

An agent reads a hosted database — and the beads tracker inside one — by calling
tools instead of scraping HTML. The endpoint speaks the Model Context Protocol
over streamable HTTP at `/mcp` on the web listener
(`https://dolt.srht.bigb.es/mcp`); there is no second port and no switch that
turns it off, because a surface that is off in production and on in a test is a
surface nobody tests.

It is read-only, and structurally so: there is no `query(sql)` tool and no
mutation of any kind. The tools list databases, branches, tables, rows, commits
and diffs, and — for a database carrying the beads schema — issues, milestones,
memories and the ready set. Writing stays with `bd` in a checkout.

The credential is `Authorization: Bearer <token>`, and nothing else: no cookie
(an MCP client is not a browser) and no HTTP Basic (that is `dolt clone`'s
flow). Two tokens are accepted. A **tokens.sr.ht working token** must carry the
`dolt:read` grant — one grant for the whole surface, because every tool on it is
a read — and it works only on an instance that configures `[tokens.sr.ht]
origin`; without that section there is no daemon to verify it against, so it is
refused with 401 while everything else keeps working. A **meta personal access
token** is accepted with the same `dolt.sr.ht/repos:RO` scoping the clone path
applies. No credential at all is a normal caller: it reads what an anonymous
visitor reads, and a PRIVATE database it may not see is "not found" rather than
"forbidden", exactly as in the web UI.