# CI: what `.build.yml` does and why

The manifest is deliberately short on prose. builds.sr.ht stores a submitted
manifest in a `varchar(16384)`, so a manifest that grows past 16 KiB cannot be
submitted at all — the failure is at submission time, it reads like nothing in
particular, and what it produces is a branch with no CI rather than a red build.
Rationale therefore lives here, and the manifest carries pointers.

The pipeline is one linear job on `alpine/edge`: install the cache helper,
assemble the shared SCSS, stamp a version, restore the Go caches, start a
Postgres, test (with coverage), package with `abuild`, publish the apk to
`repo.bigb.es/alpine/v3.22/bigbes`, save the caches, upload the coverage profile
to cov.sr.ht and this build's own benchmarks to bench.sr.ht.

It is triggered by a push to the **sourcehut** side. A push to sourcecraft
cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on
`git.srht.bigb.es`, and that push is what submits the job. So the mirror is on
the critical path for the package repository, not merely an offsite copy.

## packages

`postgresql` and `postgresql-client` are for the test suite, not for the
package. The apk declares no runtime dependency on Postgres — the daemon talks
to whatever `connection-string` names, which in production is another host.

`sassc` and `minify` are the stylesheet pipeline (`make css`), and they are
build-time only: the compiled CSS is embedded into the binary by `//go:embed`,
so nothing at runtime needs either.

`rclone` is for `publish` and `curl` for the cacher bootstrap.

## secrets

Four, all account-level and shared with the sibling services:

| secret | lands at | used by |
|---|---|---|
| `apk-ci-s3` | `~/.apk-ci.env` | `publish` |
| `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` |
| `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` |
| `c7968415-…` | `~/.srht-token` | `publish_artifacts`, `coverage`, `bench` |

They are file secrets. Listing them is what turns `publish` and the cache tasks
on; a manual submission that asks for no secrets still runs the interesting part
of the pipeline, and `publish` says so and exits 0 rather than failing.

`apk-ci-s3` is referenced by name and the other two by UUID, which is only
because that is how they were written in the donor manifests; both forms work.

`~/.srht-token` holds a tokens.sr.ht **working token**, and it is one secret
shared with every sibling service rather than a per-service one. That is what
centralising issuance buys: the credential is minted once, for a person, and
carries the grants of every service it is meant to reach — so this one must
carry `artifacts:upload` for `publish_artifacts`, `cov:upload` for `coverage`
and `bench:upload` for `bench`. Grants are compared literally, so a token
missing one of the three fails that one task and no other; there is no partial
credit and no fallback.

## environment

`COVER_REPO` and `BENCH_REPO` are both `~bigbes/sr-ht-spec`, and that is not a
guess from the checkout directory: it is the `sources:` line
(`https://git.srht.bigb.es/~bigbes/sr-ht-spec`) read as `~owner/repo`. The
directory this repository is cloned into is `sourcehut-specs` on at least one
machine — plural, and a different word — which is exactly the name that would
have been wrong.

## artifacts

Two, both literal paths relative to `$HOME`: `cover.out` (written by `test`) and
`bench.txt` (written by `bench`). `artifacts:` has no globbing, which is why the
`Makefile` takes `COVERPROFILE` as a variable — CI points it at `$HOME` and a
checkout leaves it in the checkout.

They are **not** a fallback for the two uploads. They are what a build handed no
secrets still leaves behind, so a manual submission that asked for none can
still be read, and a rejected upload can be replayed by hand from the exact
bytes the build produced.

## cacher

`cacher` is an S3-backed cache helper, installed from pages.sr.ht. It is two
tasks and not one because `install.sh` appends its `PATH` export to
`~/.buildenv`, and `~/.buildenv` is sourced by the *next* task's preamble — a
single task would install it and then not find it.

It is installed through `install.sh` rather than by `curl`-ing the raw binary:
the installer verifies the download against the published `checksums.txt`, and
`curl -fsSL` fails on an HTTP error instead of writing the error page to the
destination and marking it executable.

## scss

`sassc` needs the shared sourcehut partials, and no apk ships them: they are
core.sr.ht's own `scss/` plus the Bootstrap 4 tree it pins as a submodule. The
task clones both and drops them in `/usr/share/sourcehut/scss`, cached under a
key made of the two pins so an outage at git.sr.ht or GitHub cannot fail a build
that changed nothing.

**`CORE_VER` must track the deployment's `SRHT_CORE_VER`.** The two being out of
step means this daemon renders against different partials than the rest of the
instance, which shows up as a page that is subtly the wrong shape and as nothing
at all in any log. `BOOTSTRAP_REV` is the submodule commit core.sr.ht pins at
that tag; bump the two together.

The `--exec` form is the cache helper's own miss-and-seed: the block runs only
on a miss and the directory is uploaded afterwards, which is one flow instead of
an `if ! cacher dir download … fi` with a matching upload inside it that has to
be kept in step by hand. It sees exported variables only, which is why the cache
key is spelled inline in the `cacher` invocation rather than computed inside the
block, and why the block is in single quotes.

The cache key here is not the `--prefix` `cacher_init` sets. That one namespaces
this repository's Go caches; the SCSS tree is keyed by the two upstream pins and
nothing about this repository, so it is deliberately not hoisted into a shared
prefix.

## keygen

`abuild-keygen -a -n -i -q` makes a throwaway package signing key: generated per
build, dies with the VM, trusted by nothing. Clients verify against the index
instead, which is rebuilt and signed on phoebe by the garage stack's `apk-mirror`
service — it indexes this repo with `--allow-untrusted` precisely because of
this.

`-i` is not optional. Without it the key is generated but not installed into
`/etc/apk/keys`, and abuild's own final "update the local repository index" step
dies with an UNTRUSTED signature after having built the package perfectly well.

## version

One `git describe` decides the apk's `pkgver`. Alpine's version grammar does not
accept `v1.2.3-4-gabc1234`, so a describe with commits since the tag is
rewritten to `1.2.3_git4`. `_git` sorts **after** the release in Alpine's
comparison, which is what makes an untagged master build look newer than the tag
it follows. With no tags at all it falls back to `0.0.<commit count>`.

Tags are in `allow-refs` for exactly this reason: pushing `v0.9.0` is what
produces the `0.9.0` apk.

### Why it is exported and never `sed`-ed

The version reaches `abuild` as an environment variable — `APKBUILD` reads
`pkgver="${PKGVER:-0.0.0}"` — and the tracked `APKBUILD` is never rewritten.

This is not style. Go records `vcs.modified` in every binary compiled inside a
git repository and reads that flag from `git status --porcelain`. A CI task that
`sed`s a tracked file leaves the tree dirty for the whole `abuild` run, so
`Main.Version` in the packaged binary carries `+dirty` for the life of the apk.
Measured on go1.26.5.

The same measurement is why `.gitignore` matters here more than tidiness would
suggest: `git status --porcelain` counts **untracked** files too. `make css`
writes `web/static/main.min.<sha>.css` *before* `go build` runs, and that name
was not ignored — only the intermediate `main.css` was — so every packaging run
compiled the binaries with an untracked file sitting in the tree. `/src/`,
`/pkg/` and `/tmp/` are the three directories `abuild` works in with
`source=""` and `builddir="$startdir"`, all of them inside the checkout being
packaged, and abuild exports `GOTMPDIR=$startdir/tmp` on top of that.

The `version` task ends with `git status --porcelain` because this is the last
moment the tree is provably clean, and everything that could dirty it happens
after.

## cache_restore

The Go module cache and build cache are keyed by the hash of `go.sum`, so a
build that changed no dependency reuses both. `--optional` turns a miss into a
cold build rather than a failure — `|| true` did the same thing while also
swallowing a real error, such as a bad endpoint or expired credentials.

`abuild` redirects both caches into its throwaway `$tmpdir` (and an upstream
typo slaves `GOMODCACHE` to `GOCACHE`), so exports from this task cannot stick:
the `APKBUILD`'s `build()` re-pins both to the home locations restored here.

### The half-restored module cache

The repair block is not defensive padding. A partially restored module cache is
the *normal* failure mode of this arrangement — an interrupted upload, a
truncated object, a key written while a build was still running — and what it
produces is a compile error deep in a dependency, which reads exactly like a bug
in the code under test.

So: make the tree writable (the module cache is mode 555 and `rm -rf` cannot
remove it otherwise), ask `go mod verify` whether what came back is intact, and
throw the whole thing away if it is not. Then download and verify for real, and
let *that* failure be fatal.

Do not soften the final `go mod verify` to `|| true`. A build that proceeds with
a module cache it could not verify is a build whose result means nothing.

### Why the download is not `go mod download all`

`all` is not a harmless "be thorough" flag. It resolves the entire module
graph — the test dependencies of dependencies included — and **appends their
hashes to the tracked `go.sum`**: 170 lines on this tree, measured with
go1.26.5. A modified tracked file in the checkout at `go build` time is exactly
the dirty stamp the [version](#why-it-is-exported-and-never-sed-ed) section is
about, so the task written to warm the cache would have undone the fix.

Plain `go mod download` leaves `go.sum` byte-identical and `go mod verify` still
passes after it, which is the whole of what this task needs: the build imports
what the main module imports, and those hashes are already in `go.sum` because
they are what `go.sum` is.

This is not theoretical either — it is what the sibling bench.sr.ht build #359
failed on, at its `check-version` gate, with `M go.sum` named as the culprit.

## postgres

A real Postgres in the VM, initialised from scratch each build.

It is what turns this repository's 93 test files, across 16 packages, from files
that compile on the builder into tests that ran there. `db/db_test.go`,
`service/fixture_test.go` and `cmd/specsrht-migrate/main_test.go` all gate their
integration cases on `SPECSRHT_TEST_PG`, and without it **67 tests skip** —
measured on this tree by counting `--- SKIP` with and without the variable. The
persistence layer, the GraphQL/service layer over it and the migration runner
are all in that set, the migration-agreement check included: the one that
catches `schema.sql` and `migrations/` drifting apart, which is a defect a fresh
install and an upgraded instance disagree about and nothing else notices.

`fsync=off`, `full_page_writes=off` and `synchronous_commit=off` are safe here
and only here: the database lives for the length of one build and its durability
guarantees protect nothing.

`createuser -s "$(id -un)"` makes the build user a superuser and the DSN carries
no password, because the tests create a scratch schema per test
(`specsrht_test_<random>`), apply `schema.sql` into it and drop it afterwards.
One database therefore serves the whole suite, and no `CREATE DATABASE` right is
needed.

## test

The guard on an empty `SPECSRHT_TEST_PG` exists because the failure it prevents
is silent. If the `postgres` task did not export the DSN — or if someone
reordered the two tasks — every database suite skips with a friendly message,
`go test` exits 0, and the build is green over untested code. An explicit
refusal is the difference between a broken pipeline and a lying one.

It carries more weight here than in the siblings: `options="!check"` in the
`APKBUILD` means abuild runs no tests at all, so this task is the *only* place
the suites run.

`gofmt` and `go vet` run here too, before the tests, so a formatting regression
fails the build rather than waiting for someone to notice in review. `gofmt`
needs the `test -z "$(gofmt -l .)"` spelling because `gofmt -l` reports the
files it would change and still exits 0.

The suite is invoked as `make cover` and not as a bare `go test ./...`: the
Makefile is where this repository's `-timeout` is named (`TEST_TIMEOUT?=20m`),
and a second copy of that number here is a second copy to forget. The default
`go test` timeout is ten minutes, it is silent about being a default, and what
it produces on a slow builder is a goroutine dump rather than a failure anyone
can read.

It is `make cover` rather than `make test` because the two run the same suites
over the same tree; `cover` only adds `-covermode=atomic -coverprofile=…`. Doing
it in one task means the profile is a by-product of the gate that already had to
pass, and not a second full run of the suites whose result nothing checks.
`atomic` and not the default `set`: the profile carries real hit counts, which
is what a trend across commits is read off, and `set` would flatten every count
to a bit. `COVERPROFILE="$HOME/cover.out"` because that is where `artifacts:`
looks, and because `abuild` packages this checkout in place: a profile written
into the checkout is one more file in the tree the `version` task had just
proved clean.

## build

`REPODEST=$HOME/packages abuild -d` builds and stages the apk.

`-d` disables abuild's dependency check: the makedepends are already installed
by the manifest's `packages:` list, and abuild has no way to know that.

`builddir="$startdir"` in `APKBUILD` means abuild packages *this checkout in
place* rather than unpacking a tarball. There is no tarball to unpack — the
package is built from the commit under test, which has not been released
anywhere yet.

`build()` runs `make css`, then `make check-css`, then `make build`. The order is
forced by `//go:embed`: `web/templates.go` embeds the whole `static` directory,
so a stylesheet built after the compiler ran is a stylesheet no binary contains.
`go build` succeeds perfectly well with an unstyled `static/` — the embed takes
the directory, not the file — so `check-css` is the only thing between that
mistake and an unstyled service in production.

`package()` calls `make install-files`, not `make install`, and this is the one
line in the file that looks like a micro-optimisation and is not. abuild runs
`package()` in a **fresh** abuild process under fakeroot, which re-sources the
`APKBUILD` and never calls `build()`. Nothing `build()` exported reaches it — the
Go cache pins included — and the binary targets are `.PHONY`, so `make install`
relinked both binaries from a cold module cache. The apk therefore shipped a
*second* binary, compiled in a different environment, that no task in this
pipeline had ever tested. `install-files` is the copying half of `install` with
no build in front of it, so what is packaged is the bytes `build()` produced.

Static assets are not installed by either target. They are inside the binary; a
second copy under `$(ASSETS)/spec.sr.ht/static` is dead weight in the apk that
nothing reads.

## publish

The `[ ! -r ~/.apk-ci.env ]` gate is the honest answer to a build that was handed
no secrets, and not a fallback: every earlier task has still run and a signed apk
is sitting in `$HOME/packages`, so the useful part of a manual submission has
happened and only the upload cannot. On a push the secret is there and this
publishes.

`set +x` before sourcing `~/.apk-ci.env` and `set -x` after. Every task runs
under `set -x`, so without this the S3 access key and secret are echoed into a
build log that is world-readable.

The upload is `rclone copyto` per file — copy only, never delete, never sync. A
`sync` would mirror local absence onto the bucket, and the bucket holds every
previously published version of every service on the instance.

`apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it.

## publish_artifacts

The same apk, uploaded a second time — into the `~bigbes/main` channel of
`artifacts.sr.ht`, which indexes and signs it in the same request rather than on
a 15-minute timer. One `PUT` per file to
`/api/v1/pkg/~bigbes/main/apk/v3.22`, with the working token from
`~/.srht-token` as the bearer. No client binary is involved: the upload route
takes the file as the request body, and `curl` is already installed for the
cacher.

It is a task of its own and not two more lines inside `publish` for one reason:
the S3 copy is what phoebe re-indexes today and this one is the road being
opened. A refusal here has to be legible as *this* destination refusing, and it
must not be able to undo an upload that already succeeded.

`set +x` around every `curl`, for the same reason `publish` has it: the token is
on the command line and the log is world-readable.

Three answers are expected and only two of them are good:

- `201` — published, indexed, signed.
- `200` — the same version with the same bytes was already there. A rerun of a
  job that got as far as this task is idempotent, and stays green.
- `409` — the same version with *different* bytes. `abuild` stamps mtimes into
  the archive, so resubmitting one commit produces a byte-different apk under an
  identical `pkgver`; a published version is immutable, so the daemon keeps what
  it has. That is a `WARNING` and not a failure — it means the build was run
  twice, not that anything is wrong. Republishing means removing the version
  through the API first.

Anything else prints the daemon's error body and fails the task, after trying
every remaining file: one bad package must not hide the fate of the others.

The loop is `for file in $(find …)` rather than `find … | while read`, because
under ash the second form runs its body in a subshell and the failure flag set
in it is lost when the pipeline ends — the task would report success it never
had.

Gated on `~/.srht-token` being readable, exactly as `publish` is gated on
`~/.apk-ci.env`: a manual submission without secrets still builds the apk, and
says that nothing was published.

Installing what it publishes, on any Alpine box:

```sh
wget -qO /etc/apk/keys/bigbes@artifacts.srht.bigb.es.rsa.pub \
  https://artifacts.srht.bigb.es/~bigbes/keys/apk.rsa.pub
echo https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22 >> /etc/apk/repositories
apk update && apk add spec.sr.ht
```

The name of the key file is not free: apk looks a signature up by the name
carried in the index's `.SIGN.RSA256.<name>` entry, so it has to be
`bigbes@artifacts.srht.bigb.es.rsa.pub` and nothing else. With the key in place
no `--allow-untrusted` is needed — the daemon signs the index it builds.

## cache_save

After `publish`, deliberately: a cache upload that fails must not strand an apk
that was built and tested successfully but never shipped.

Fatal on purpose, though — an upload failure here means the next build pays for
a cold cache, and that is worth knowing about rather than hiding behind
`|| true`.

There is no `cacher exists "$KEY" ||` guard in front of either upload any more.
Without `--force` an upload skips a key that is already there, so the guard was
a second round-trip that asked the question the upload asks anyway — and one
that goes wrong in the direction that matters, since a key that exists but is
truncated is exactly the half-restored cache the repair block above is about.

## coverage

The profile the `test` task already wrote, POSTed to this instance's own
cov.sr.ht.

It is its own task, and it comes **after** `publish` and `cache_save` for the
reason every upload here does — a rejected report must not cost an apk that was
built, signed and shipped. It comes **before** `bench` because the profile is
already in hand while the benchmark run is minutes long, and a long run has no
business standing between a finished profile and its upload.

Three things in it are not decoration:

- `test -s "$HOME/cover.out"` before anything else. A missing or empty profile
  is a `test` task that did not write one, and the service would answer that
  with a 400 about a body — a message about the request, when the fact is about
  the build.
- The `~/.srht-token` gate. Without the file this build was handed no secrets;
  that is a manual submission, not a failure, and the profile is still this
  build's `cover.out` artifact. **With** the file the upload is fatal on
  purpose: a build that has the credential and cannot publish should say so.
- `set +x` immediately before the `curl`. The task runs under `set -x`, and the
  `Authorization` header would otherwise be printed into a build log that is
  public.

## bench

This service's own benchmarks, uploaded to this instance's own bench.sr.ht,
`&visibility=public` so the repository the first POST creates is readable.

### What is measured, and why those

Serving one revision of a space is two passes over the whole corpus, and a
proposal view is a third over two whole revisions of a document. None of them is
I/O and all of them grow with the corpus rather than with the request, which is
what makes them worth a number: a space that doubles in size doubles what every
page of it costs, and nothing about a single document says so.

- `BenchmarkFromDocuments` / `BenchmarkLinkPass` / `BenchmarkRenderDocument`
  (`doc/bench_test.go`) run over a synthetic space of 208 documents in 8
  sections — headers with ids and parents, aliases, wikilinks that resolve
  inside a section, across sections and through an alias, wikilinks that resolve
  to nothing, relative links, tables and fenced blocks. `FromDocuments` is the
  archive build (headers parsed, the id contest decided over the whole revision,
  the hierarchy linked, the alias and stem indexes filled); `LinkPass` renders
  every document to fill the link graph the backlink, orphan and catalog views
  read; `RenderDocument` is the single page a reader opens. `LinkPass` rebuilds
  the archive per iteration with the timer stopped, because it writes
  `Page.Links` and a reused archive would measure a second pass over an
  already-linked graph.
- `BenchmarkCompare` / `BenchmarkCompareUnchanged` / `BenchmarkSegment` /
  `BenchmarkDiffWords` (`prosediff/bench_test.go`) run the proposal diff over
  two revisions of a 24-chapter specification. The edited revision carries one
  of every change kind the alignment can recognise — a reworded sentence, an
  inserted requirement, a deleted paragraph, an edited code fence and a moved
  chapter — so the alignment is not measured on a wall of insertions.
  `CompareUnchanged` is a document against itself, which is what most files in a
  long proposal are, and `Segment` and `DiffWords` are the two halves alone so a
  regression can be attributed rather than guessed at.

Every one of them asserts its own result inside the loop (the archive holds 208
pages, the diff found all four change kinds, the render resolved a link *and*
left the deliberately unresolvable wikilink unresolved). A pass that silently
produced nothing would otherwise be the fastest entry in the file — and this is
not hypothetical: the `RenderDocument` assertion caught its own fixture picking
a section index, which carries no unresolvable link, on the first run.

None of them needs Postgres, which is why this task can sit at the end of the
pipeline without a DSN guard.

### The two greps

`go test -bench` that matches nothing prints `ok` and exits `0`, and a file with
no benchmark lines in it is *valid* benchfmt. A renamed or deleted benchmark
would therefore upload an empty run and report success. So the names are checked
against the file before the upload:

```sh
grep -q '^BenchmarkCompare' "$HOME/bench.txt"
grep -q '^BenchmarkLinkPass' "$HOME/bench.txt"
```

One per benchmarked package, so losing either package's benchmarks is loud.

`make -s bench > "$HOME/bench.txt"` and then `cat`, and deliberately **not**
`| tee`: a pipeline's exit status is the last command's, so `tee` would let a
failing benchmark run pass. `-s` keeps make from echoing the recipe into a file
the parser will read.

`BENCH_COUNT` is 10. bench.sr.ht marks a point measured under six repetitions
"low n" — a point's confidence interval only becomes finite at six — so a run
uploaded with fewer is a run nobody can read a regression off. A checkout that
only wants to know the benchmarks still run says `make bench BENCH_COUNT=1`.

A builder VM this small measures a **shape**, not a number: the absolute ns/op
is worth nothing next to a laptop's, and the point of uploading it is that it is
measured the same way every time.

## The two requests

Both are a POST with the token as a Bearer header and the file as
`--data-binary`, and both carry the same four query parameters:

| parameter | value | why |
|---|---|---|
| `commit` | `git rev-parse HEAD` | what the numbers are about |
| `ref` | `$GIT_REF` with both prefixes stripped | see below |
| `key` | `$JOB_ID` | the idempotency key: a resubmitted job replaces, not duplicates |
| `job_url` | `$JOB_URL` | the build a report links back to |

`ref="${GIT_REF#refs/heads/}"; ref="${ref#refs/tags/}"` strips **both**
prefixes because this pipeline builds tags too (`allow-refs` carries
`refs/tags/v*`), and a tag build would otherwise report `ref=refs/tags/v0.9.0`.
`GIT_REF` is absent altogether on a manually submitted build, which is fine: the
parameter is optional.

The coverage POST sends **no `Content-Type`** — cov.sr.ht sniffs the body, and a
wrong declared type is worse than none. Both use `curl -sS --fail-with-body`,
which prints the service's JSON error *and* still exits non-zero; plain `--fail`
would swallow the only sentence saying what was wrong.

## What is not here

- **No matrix.** One architecture, one image.