#CI: what .build.yml does and why

The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a varchar(16384), so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time, it reads like nothing in particular, and what it produces is a branch with no CI rather than a red build. Rationale therefore lives here, and the manifest carries pointers.

The pipeline is one linear job on alpine/edge: install the cache helper, assemble the shared SCSS, stamp a version, restore the Go caches, start a Postgres, test (with coverage), package with abuild, publish the apk to repo.bigb.es/alpine/v3.22/bigbes, save the caches, upload the coverage profile to cov.sr.ht and this build's own benchmarks to bench.sr.ht.

It is triggered by a push to the sourcehut side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on git.srht.bigb.es, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy.

#packages

postgresql and postgresql-client are for the test suite, not for the package. The apk declares no runtime dependency on Postgres — the daemon talks to whatever connection-string names, which in production is another host.

sassc and minify are the stylesheet pipeline (make css), and they are build-time only: the compiled CSS is embedded into the binary by //go:embed, so nothing at runtime needs either.

rclone is for publish and curl for the cacher bootstrap.

#secrets

Four, all account-level and shared with the sibling services:

They are file secrets. Listing them is what turns publish and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline, and publish says so and exits 0 rather than failing.

apk-ci-s3 is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work.

~/.srht-token holds a tokens.sr.ht working token, and it is one secret shared with every sibling service rather than a per-service one. That is what centralising issuance buys: the credential is minted once, for a person, and carries the grants of every service it is meant to reach — so this one must carry artifacts:upload for publish_artifacts, cov:upload for coverage and bench:upload for bench. Grants are compared literally, so a token missing one of the three fails that one task and no other; there is no partial credit and no fallback.

#environment

COVER_REPO and BENCH_REPO are both ~bigbes/sr-ht-spec, and that is not a guess from the checkout directory: it is the sources: line (https://git.srht.bigb.es/~bigbes/sr-ht-spec) read as ~owner/repo. The directory this repository is cloned into is sourcehut-specs on at least one machine — plural, and a different word — which is exactly the name that would have been wrong.

#artifacts

Two, both literal paths relative to $HOME: cover.out (written by test) and bench.txt (written by bench). artifacts: has no globbing, which is why the Makefile takes COVERPROFILE as a variable — CI points it at $HOME and a checkout leaves it in the checkout.

They are not a fallback for the two uploads. They are what a build handed no secrets still leaves behind, so a manual submission that asked for none can still be read, and a rejected upload can be replayed by hand from the exact bytes the build produced.

#cacher

cacher is an S3-backed cache helper, installed from pages.sr.ht. It is two tasks and not one because install.sh appends its PATH export to ~/.buildenv, and ~/.buildenv is sourced by the next task's preamble — a single task would install it and then not find it.

It is installed through install.sh rather than by curl-ing the raw binary: the installer verifies the download against the published checksums.txt, and curl -fsSL fails on an HTTP error instead of writing the error page to the destination and marking it executable.

#scss

sassc needs the shared sourcehut partials, and no apk ships them: they are core.sr.ht's own scss/ plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in /usr/share/sourcehut/scss, cached under a key made of the two pins so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing.

CORE_VER must track the deployment's SRHT_CORE_VER. The two being out of step means this daemon renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. BOOTSTRAP_REV is the submodule commit core.sr.ht pins at that tag; bump the two together.

The --exec form is the cache helper's own miss-and-seed: the block runs only on a miss and the directory is uploaded afterwards, which is one flow instead of an if ! cacher dir download … fi with a matching upload inside it that has to be kept in step by hand. It sees exported variables only, which is why the cache key is spelled inline in the cacher invocation rather than computed inside the block, and why the block is in single quotes.

The cache key here is not the --prefix cacher_init sets. That one namespaces this repository's Go caches; the SCSS tree is keyed by the two upstream pins and nothing about this repository, so it is deliberately not hoisted into a shared prefix.

#keygen

abuild-keygen -a -n -i -q makes a throwaway package signing key: generated per build, dies with the VM, trusted by nothing. Clients verify against the index instead, which is rebuilt and signed on phoebe by the garage stack's apk-mirror service — it indexes this repo with --allow-untrusted precisely because of this.

-i is not optional. Without it the key is generated but not installed into /etc/apk/keys, and abuild's own final "update the local repository index" step dies with an UNTRUSTED signature after having built the package perfectly well.

#version

One git describe decides the apk's pkgver. Alpine's version grammar does not accept v1.2.3-4-gabc1234, so a describe with commits since the tag is rewritten to 1.2.3_git4. _git sorts after the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all it falls back to 0.0.<commit count>.

Tags are in allow-refs for exactly this reason: pushing v0.9.0 is what produces the 0.9.0 apk.

#Why it is exported and never sed-ed

The version reaches abuild as an environment variable — APKBUILD reads pkgver="${PKGVER:-0.0.0}" — and the tracked APKBUILD is never rewritten.

This is not style. Go records vcs.modified in every binary compiled inside a git repository and reads that flag from git status --porcelain. A CI task that seds a tracked file leaves the tree dirty for the whole abuild run, so Main.Version in the packaged binary carries +dirty for the life of the apk. Measured on go1.26.5.

The same measurement is why .gitignore matters here more than tidiness would suggest: git status --porcelain counts untracked files too. make css writes web/static/main.min.<sha>.css before go build runs, and that name was not ignored — only the intermediate main.css was — so every packaging run compiled the binaries with an untracked file sitting in the tree. /src/, /pkg/ and /tmp/ are the three directories abuild works in with source="" and builddir="$startdir", all of them inside the checkout being packaged, and abuild exports GOTMPDIR=$startdir/tmp on top of that.

The version task ends with git status --porcelain because this is the last moment the tree is provably clean, and everything that could dirty it happens after.

#cache_restore

The Go module cache and build cache are keyed by the hash of go.sum, so a build that changed no dependency reuses both. --optional turns a miss into a cold build rather than a failure — || true did the same thing while also swallowing a real error, such as a bad endpoint or expired credentials.

abuild redirects both caches into its throwaway $tmpdir (and an upstream typo slaves GOMODCACHE to GOCACHE), so exports from this task cannot stick: the APKBUILD's build() re-pins both to the home locations restored here.

#The half-restored module cache

The repair block is not defensive padding. A partially restored module cache is the normal failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test.

So: make the tree writable (the module cache is mode 555 and rm -rf cannot remove it otherwise), ask go mod verify whether what came back is intact, and throw the whole thing away if it is not. Then download and verify for real, and let that failure be fatal.

Do not soften the final go mod verify to || true. A build that proceeds with a module cache it could not verify is a build whose result means nothing.

#Why the download is not go mod download all

all is not a harmless "be thorough" flag. It resolves the entire module graph — the test dependencies of dependencies included — and appends their hashes to the tracked go.sum: 170 lines on this tree, measured with go1.26.5. A modified tracked file in the checkout at go build time is exactly the dirty stamp the version section is about, so the task written to warm the cache would have undone the fix.

Plain go mod download leaves go.sum byte-identical and go mod verify still passes after it, which is the whole of what this task needs: the build imports what the main module imports, and those hashes are already in go.sum because they are what go.sum is.

This is not theoretical either — it is what the sibling bench.sr.ht build #359 failed on, at its check-version gate, with M go.sum named as the culprit.

#postgres

A real Postgres in the VM, initialised from scratch each build.

It is what turns this repository's 93 test files, across 16 packages, from files that compile on the builder into tests that ran there. db/db_test.go, service/fixture_test.go and cmd/specsrht-migrate/main_test.go all gate their integration cases on SPECSRHT_TEST_PG, and without it 67 tests skip — measured on this tree by counting --- SKIP with and without the variable. The persistence layer, the GraphQL/service layer over it and the migration runner are all in that set, the migration-agreement check included: the one that catches schema.sql and migrations/ drifting apart, which is a defect a fresh install and an upgraded instance disagree about and nothing else notices.

fsync=off, full_page_writes=off and synchronous_commit=off are safe here and only here: the database lives for the length of one build and its durability guarantees protect nothing.

createuser -s "$(id -un)" makes the build user a superuser and the DSN carries no password, because the tests create a scratch schema per test (specsrht_test_<random>), apply schema.sql into it and drop it afterwards. One database therefore serves the whole suite, and no CREATE DATABASE right is needed.

#test

The guard on an empty SPECSRHT_TEST_PG exists because the failure it prevents is silent. If the postgres task did not export the DSN — or if someone reordered the two tasks — every database suite skips with a friendly message, go test exits 0, and the build is green over untested code. An explicit refusal is the difference between a broken pipeline and a lying one.

It carries more weight here than in the siblings: options="!check" in the APKBUILD means abuild runs no tests at all, so this task is the only place the suites run.

gofmt and go vet run here too, before the tests, so a formatting regression fails the build rather than waiting for someone to notice in review. gofmt needs the test -z "$(gofmt -l .)" spelling because gofmt -l reports the files it would change and still exits 0.

The suite is invoked as make cover and not as a bare go test ./...: the Makefile is where this repository's -timeout is named (TEST_TIMEOUT?=20m), and a second copy of that number here is a second copy to forget. The default go test timeout is ten minutes, it is silent about being a default, and what it produces on a slow builder is a goroutine dump rather than a failure anyone can read.

It is make cover rather than make test because the two run the same suites over the same tree; cover only adds -covermode=atomic -coverprofile=…. Doing it in one task means the profile is a by-product of the gate that already had to pass, and not a second full run of the suites whose result nothing checks. atomic and not the default set: the profile carries real hit counts, which is what a trend across commits is read off, and set would flatten every count to a bit. COVERPROFILE="$HOME/cover.out" because that is where artifacts: looks, and because abuild packages this checkout in place: a profile written into the checkout is one more file in the tree the version task had just proved clean.

#build

REPODEST=$HOME/packages abuild -d builds and stages the apk.

-d disables abuild's dependency check: the makedepends are already installed by the manifest's packages: list, and abuild has no way to know that.

builddir="$startdir" in APKBUILD means abuild packages this checkout in place rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet.

build() runs make css, then make check-css, then make build. The order is forced by //go:embed: web/templates.go embeds the whole static directory, so a stylesheet built after the compiler ran is a stylesheet no binary contains. go build succeeds perfectly well with an unstyled static/ — the embed takes the directory, not the file — so check-css is the only thing between that mistake and an unstyled service in production.

package() calls make install-files, not make install, and this is the one line in the file that looks like a micro-optimisation and is not. abuild runs package() in a fresh abuild process under fakeroot, which re-sources the APKBUILD and never calls build(). Nothing build() exported reaches it — the Go cache pins included — and the binary targets are .PHONY, so make install relinked both binaries from a cold module cache. The apk therefore shipped a second binary, compiled in a different environment, that no task in this pipeline had ever tested. install-files is the copying half of install with no build in front of it, so what is packaged is the bytes build() produced.

Static assets are not installed by either target. They are inside the binary; a second copy under $(ASSETS)/spec.sr.ht/static is dead weight in the apk that nothing reads.

#publish

The [ ! -r ~/.apk-ci.env ] gate is the honest answer to a build that was handed no secrets, and not a fallback: every earlier task has still run and a signed apk is sitting in $HOME/packages, so the useful part of a manual submission has happened and only the upload cannot. On a push the secret is there and this publishes.

set +x before sourcing ~/.apk-ci.env and set -x after. Every task runs under set -x, so without this the S3 access key and secret are echoed into a build log that is world-readable.

The upload is rclone copyto per file — copy only, never delete, never sync. A sync would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance.

apk-mirror on phoebe re-indexes within 15 minutes; nothing here waits for it.

#publish_artifacts

The same apk, uploaded a second time — into the ~bigbes/main channel of artifacts.sr.ht, which indexes and signs it in the same request rather than on a 15-minute timer. One PUT per file to /api/v1/pkg/~bigbes/main/apk/v3.22, with the working token from ~/.srht-token as the bearer. No client binary is involved: the upload route takes the file as the request body, and curl is already installed for the cacher.

It is a task of its own and not two more lines inside publish for one reason: the S3 copy is what phoebe re-indexes today and this one is the road being opened. A refusal here has to be legible as this destination refusing, and it must not be able to undo an upload that already succeeded.

set +x around every curl, for the same reason publish has it: the token is on the command line and the log is world-readable.

Three answers are expected and only two of them are good:

• 201 — published, indexed, signed.
• 200 — the same version with the same bytes was already there. A rerun of a job that got as far as this task is idempotent, and stays green.
• 409 — the same version with different bytes. abuild stamps mtimes into the archive, so resubmitting one commit produces a byte-different apk under an identical pkgver; a published version is immutable, so the daemon keeps what it has. That is a WARNING and not a failure — it means the build was run twice, not that anything is wrong. Republishing means removing the version through the API first.

Anything else prints the daemon's error body and fails the task, after trying every remaining file: one bad package must not hide the fate of the others.

The loop is for file in $(find …) rather than find … | while read, because under ash the second form runs its body in a subshell and the failure flag set in it is lost when the pipeline ends — the task would report success it never had.

Gated on ~/.srht-token being readable, exactly as publish is gated on ~/.apk-ci.env: a manual submission without secrets still builds the apk, and says that nothing was published.

Installing what it publishes, on any Alpine box:

wget -qO /etc/apk/keys/bigbes@artifacts.srht.bigb.es.rsa.pub \
  https://artifacts.srht.bigb.es/~bigbes/keys/apk.rsa.pub
echo https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22 >> /etc/apk/repositories
apk update && apk add spec.sr.ht

The name of the key file is not free: apk looks a signature up by the name carried in the index's .SIGN.RSA256.<name> entry, so it has to be bigbes@artifacts.srht.bigb.es.rsa.pub and nothing else. With the key in place no --allow-untrusted is needed — the daemon signs the index it builds.

#cache_save

After publish, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped.

Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind || true.

There is no cacher exists "$KEY" || guard in front of either upload any more. Without --force an upload skips a key that is already there, so the guard was a second round-trip that asked the question the upload asks anyway — and one that goes wrong in the direction that matters, since a key that exists but is truncated is exactly the half-restored cache the repair block above is about.

#coverage

The profile the test task already wrote, POSTed to this instance's own cov.sr.ht.

It is its own task, and it comes after publish and cache_save for the reason every upload here does — a rejected report must not cost an apk that was built, signed and shipped. It comes before bench because the profile is already in hand while the benchmark run is minutes long, and a long run has no business standing between a finished profile and its upload.

Three things in it are not decoration:

• test -s "$HOME/cover.out" before anything else. A missing or empty profile is a test task that did not write one, and the service would answer that with a 400 about a body — a message about the request, when the fact is about the build.
• The ~/.srht-token gate. Without the file this build was handed no secrets; that is a manual submission, not a failure, and the profile is still this build's cover.out artifact. With the file the upload is fatal on purpose: a build that has the credential and cannot publish should say so.
• set +x immediately before the curl. The task runs under set -x, and the Authorization header would otherwise be printed into a build log that is public.

#bench

This service's own benchmarks, uploaded to this instance's own bench.sr.ht, &visibility=public so the repository the first POST creates is readable.

#What is measured, and why those

Serving one revision of a space is two passes over the whole corpus, and a proposal view is a third over two whole revisions of a document. None of them is I/O and all of them grow with the corpus rather than with the request, which is what makes them worth a number: a space that doubles in size doubles what every page of it costs, and nothing about a single document says so.

• BenchmarkFromDocuments / BenchmarkLinkPass / BenchmarkRenderDocument (doc/bench_test.go) run over a synthetic space of 208 documents in 8 sections — headers with ids and parents, aliases, wikilinks that resolve inside a section, across sections and through an alias, wikilinks that resolve to nothing, relative links, tables and fenced blocks. FromDocuments is the archive build (headers parsed, the id contest decided over the whole revision, the hierarchy linked, the alias and stem indexes filled); LinkPass renders every document to fill the link graph the backlink, orphan and catalog views read; RenderDocument is the single page a reader opens. LinkPass rebuilds the archive per iteration with the timer stopped, because it writes Page.Links and a reused archive would measure a second pass over an already-linked graph.
• BenchmarkCompare / BenchmarkCompareUnchanged / BenchmarkSegment / BenchmarkDiffWords (prosediff/bench_test.go) run the proposal diff over two revisions of a 24-chapter specification. The edited revision carries one of every change kind the alignment can recognise — a reworded sentence, an inserted requirement, a deleted paragraph, an edited code fence and a moved chapter — so the alignment is not measured on a wall of insertions. CompareUnchanged is a document against itself, which is what most files in a long proposal are, and Segment and DiffWords are the two halves alone so a regression can be attributed rather than guessed at.

Every one of them asserts its own result inside the loop (the archive holds 208 pages, the diff found all four change kinds, the render resolved a link and left the deliberately unresolvable wikilink unresolved). A pass that silently produced nothing would otherwise be the fastest entry in the file — and this is not hypothetical: the RenderDocument assertion caught its own fixture picking a section index, which carries no unresolvable link, on the first run.

None of them needs Postgres, which is why this task can sit at the end of the pipeline without a DSN guard.

#The two greps

go test -bench that matches nothing prints ok and exits 0, and a file with no benchmark lines in it is valid benchfmt. A renamed or deleted benchmark would therefore upload an empty run and report success. So the names are checked against the file before the upload:

grep -q '^BenchmarkCompare' "$HOME/bench.txt"
grep -q '^BenchmarkLinkPass' "$HOME/bench.txt"

One per benchmarked package, so losing either package's benchmarks is loud.

make -s bench > "$HOME/bench.txt" and then cat, and deliberately not | tee: a pipeline's exit status is the last command's, so tee would let a failing benchmark run pass. -s keeps make from echoing the recipe into a file the parser will read.

BENCH_COUNT is 10. bench.sr.ht marks a point measured under six repetitions "low n" — a point's confidence interval only becomes finite at six — so a run uploaded with fewer is a run nobody can read a regression off. A checkout that only wants to know the benchmarks still run says make bench BENCH_COUNT=1.

A builder VM this small measures a shape, not a number: the absolute ns/op is worth nothing next to a laptop's, and the point of uploading it is that it is measured the same way every time.

#The two requests

Both are a POST with the token as a Bearer header and the file as --data-binary, and both carry the same four query parameters:

ref="${GIT_REF#refs/heads/}"; ref="${ref#refs/tags/}" strips both prefixes because this pipeline builds tags too (allow-refs carries refs/tags/v*), and a tag build would otherwise report ref=refs/tags/v0.9.0. GIT_REF is absent altogether on a manually submitted build, which is fine: the parameter is optional.

The coverage POST sends no Content-Type — cov.sr.ht sniffs the body, and a wrong declared type is worse than none. Both use curl -sS --fail-with-body, which prints the service's JSON error and still exits non-zero; plain --fail would swallow the only sentence saying what was wrong.

#What is not here

• No matrix. One architecture, one image.