#CI: what .build.yml does and why

The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a varchar(16384), so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time and reads like nothing in particular: the branch simply has no CI. Rationale therefore lives here, and the manifest carries pointers.

That headroom is nearly spent: the manifest is ~15.8 KiB of the 16 KiB a submission may carry, which is about eight comment lines. A paragraph added there is a paragraph that has to come out again — write it here instead.

The pipeline is one linear job on alpine/edge: install the cache helper, assemble the shared SCSS, make a throwaway signing key, decide the version, restore the Go caches, test, package with abuild, publish the apk to repo.bigb.es/alpine/v3.22/bigbes, save the caches, and report on itself to cov.sr.ht and bench.sr.ht.

It is triggered by a push to the sourcehut side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on git.srht.bigb.es, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy.

#packages

sassc and minify are for make css, not for the package: the stylesheet is compiled here and embedded into the binary, and the apk has no runtime dependency on either.

curl is for the cacher_install bootstrap and rclone for publish. There is no nodejs/npm: web/static/bundle.<hash>.js is committed, and make bundle is an upgrade step run by hand rather than a build step.

There is no postgresql. This service is stateless — it holds no database at all — and its suites are hermetic: gitx/fixture_test.go builds a bare repository with the local git, and web/web_test.go is httptest plus ecoretest. Nothing in the tree talks to a live git.sr.ht API.

#secrets

Four, all account-level and shared with the sibling services:

They are file secrets. Listing them is what turns publish and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline and stops at cacher_init, which is the right place to notice.

apk-ci-s3 is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work.

~/.srht-token is a tokens.sr.ht working token, minted once for a person and carrying the grants of every service its builds reach: artifacts:upload for publish_artifacts, cov:upload for coverage, bench:upload for bench. It is one secret shared with the sibling pipelines rather than three per-service ones — that is what centralising issuance buys. Grants are compared literally, so a token minted before cover.sr.ht was renamed carries cover:upload and uploads nothing to cov; re-mint it and replace the secret. A token missing one grant fails that upload and no other, because each of the three tasks presents it independently.

#environment

CORE_VER must track the deployment's SRHT_CORE_VER, and BOOTSTRAP_REV is the Bootstrap submodule commit core.sr.ht pins at that tag. The two being out of step means this service renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. They are 0.84.5 / 779ad9f1 here because that is what srht/versions.env pins in the deployment repo; bump all of them together.

REPO is the clone directory sources: produces, and every task that touches the checkout cds into it — task bodies do not inherit a working directory from each other.

#cacher

cacher (go.bigb.es/cacher) is an S3-backed cache helper, installed from its own release on pages.sr.ht.

It is two tasks and not one because install.sh appends its PATH export to ~/.buildenv, and ~/.buildenv is sourced by the next task's preamble — a single task would install the binary and then not find it on PATH.

The installer is fetched with curl -fsSL and verifies what it downloads against checksums.txt. The predecessor of this task fetched the raw binary with a bare curl -sSL; without -f, an HTTP error is not an error to curl, so a 404 page was written to ~/.local/bin/cacher and chmod +x-ed, and the first failure was a syntax error out of a shell reading HTML.

#scss

sassc needs the shared sourcehut partials, and no apk ships them: they are core.sr.ht's own scss/ plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in /usr/share/sourcehut/scss, which is the ASSETS the Makefile and the APKBUILD name.

The whole thing is cached under a key made of the two pins (scss/$CORE_VER-$BOOTSTRAP_REV.tar.zst), so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing. --exec runs only on a cache miss and seeds the cache afterwards; the block sees exported variables only, which is why the cache key is spelled inline in the cacher invocation rather than computed inside the block, and why the block is in single quotes.

It runs before keygen and before anything Go, because the stylesheet has to exist before the compiler does — see css.

#keygen

abuild-keygen -a -n -i -q makes a throwaway package signing key.

-i is not optional: without it the key is generated but not installed into /etc/apk/keys, and abuild's own final "update the local repository index" step then dies with an UNTRUSTED signature after having built the package perfectly well.

The key is per-build and dies with the VM. That is fine because nothing verifies these signatures — the apk index on repo.bigb.es is rebuilt and signed on phoebe by the garage stack's apk-mirror service, which indexes this repo with --allow-untrusted precisely because of this.

#version

One git describe decides the apk's pkgver. Alpine's version grammar does not accept v1.2.3-4-gabc1234, so a describe with commits since the tag is rewritten to 1.2.3_git4; _git sorts after the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all — which is this repository today — it falls back to 0.0.<commit count>, monotonic and unique per commit, which a deployment can pin.

Tags are in allow-refs for exactly this reason: pushing v0.9.0 is what produces the 0.9.0 apk.

$ver is exported through ~/.buildenv as PKGVER, and the APKBUILD reads pkgver="${PKGVER:-0.0.0}". It is not sed-ed into the tracked APKBUILD, and this is the one thing in this file that must not be "tidied" back:

Go records vcs.modified in every binary it builds inside a repository, and it reads that flag from git status --porcelain. Measured on go1.26.5, a sed -i of a tracked file sets vcs.modified=true and Go stamps +dirty into Main.Version for the whole abuild run — for every binary in the package, permanently, for the life of the apk.

The export alone is not enough. Any untracked file present at go build time does the same thing, and abuild works inside the checkout with builddir="$startdir" and exports GOTMPDIR=$startdir/tmp — so ./src, ./pkg and ./tmp all appear inside the tree it is packaging. .gitignore covers those three, and Go does not count ignored files.

The task ends with a bare git status --porcelain because this is the last moment the tree is provably clean, and the output of a clean one is nothing.

#cache_restore

The Go module cache and build cache are keyed by the hash of go.sum, so a build that changed no dependency reuses both.

--optional and not || true. They are not the same promise: --optional returns 0 only for a genuine cache miss, while || true swallows every other outcome too — bad credentials, a dead bucket, a typo in the prefix — and turns a misconfigured cache into a pipeline that is merely slow, silently, forever.

The exports here are only for the repair block below and for cache_save: abuild redirects both caches into its throwaway $tmpdir (and an upstream typo in abuild.in slaves GOMODCACHE to GOCACHE), so environment exports from a task cannot stick. The APKBUILD's build() re-pins both to the same home locations, after abuild's own exports, which is what makes the tarballs saved at the end be the ones the compile used.

#The half-restored module cache

The repair block is not defensive padding. A partially restored module cache is the normal failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test.

So: make the tree writable (the module cache is mode 555 and rm -rf cannot remove it otherwise), ask go mod verify whether what came back is intact, and throw the whole thing away if it is not. Then go mod download and verify for real, and let that failure be fatal.

Do not soften the final go mod verify to || true. A build that proceeds with a module cache it could not verify is a build whose result means nothing.

#Not go mod download all

The warm-up is go mod download. The all must not come back, and it is worth knowing why, because all is exactly the shape a future reader adds as an obvious improvement — warm everything, surely, so nothing is fetched later.

go mod download all resolves the entire module graph, test dependencies of dependencies included, and appends the hashes of everything it resolved to go.sum. Measured on this tree: 145 lines added. In the bench sibling, 228 — and that is not a hypothetical, it is what failed its build #359, which stopped at check-version with a -dirty binary and M go.sum named as the thing in the way.

A modified tracked file in the checkout when go build runs is precisely the stamp this whole pipeline is arranged to avoid — the same failure the version task refuses to cause by not sed-ing the APKBUILD. Warming the cache with a command that edits the tree would have reintroduced it one task later.

Without all, go.sum is untouched, go mod verify still passes afterwards, and what gets fetched is what the main module actually builds. That is all a cache warm-up needs; the packages all would have added are ones no build in this pipeline compiles.

Verified in a fresh clone of this branch — go mod download followed by git status --porcelain leaves the tree clean. compare has the smallest dependency graph of the six services, so this is where all came closest to being harmless, and it still rewrote 145 lines.

#-mod=readonly does not save you

There used to be a sentence here, and another above the git status line in the manifest, saying -mod=readonly is the default so neither command can rewrite go.mod or go.sum. It was false, and it is a large part of why nobody looked at go mod download all for as long as they did: it sat directly above the line that disproved it and invited the reader to skip both.

readonly governs updates to the module requirements. It does not stop writes to go.sum. Measured on this tree rather than inherited, with the flag set explicitly on the command line:

Both still pass go mod verify afterwards, so verification is no guard either.

Set the flag on the command line when reproducing this. go env -w writes a persistent GOFLAGS, and a machine carrying go env -w GOFLAGS=-mod=mod — this one does, and bench's did too — makes a run that omits the flag prove nothing about what the builder would do. Check with go env GOFLAGS first.

So the git status --porcelain at the end of the task is not belt-and-braces, and no flag stands behind it: it is the check. It is the only thing in this pipeline that would notice a go.sum rewrite before the compile, and on the bench sibling it is what did. Do not delete it as redundant and do not replace it with an appeal to a flag.

#test

gofmt -l . piped to a file and then test ! -s, rather than a bare gofmt -l: gofmt -l exits 0 whether or not it printed anything, so the only way to fail on its output is to look at the output. The tee keeps the offending filenames in the build log, where they are the entire diagnosis.

go vet ./... next, then make cover COVERPROFILE="$HOME/cover.out" — the Makefile, not a bare go test ./..., so that the test command lives in one place.

cover is not a second suite. It is test with two flags reached through TESTFLAGS, so CI's run and a developer's make test cannot become two different commands:

test:
	go test $(TESTFLAGS) ./...

cover:
	@$(MAKE) test TESTFLAGS="-covermode=atomic -coverprofile=$(COVERPROFILE)"
	go tool cover -func=$(COVERPROFILE) | tail -1

-covermode=atomic because this is a concurrent HTTP server: the default set mode records that a statement ran and not how often, and cov.sr.ht stores the counts.

The profile is written to $HOME, outside the checkout. A profile written into the working tree would be an untracked file, and Go reads vcs.modified from git status --porcelain, which counts untracked files — the apk built two tasks later would stamp itself +dirty. The COVERPROFILE default (cover.out in the repo root, for a local run) is gitignored for the same reason.

No Postgres, no network and no guard. Every suite in this tree is hermetic: gitx/fixture_test.go builds a bare repository with the local git binary, and web/web_test.go is httptest plus ecoretest. Nothing here is gated on an environment variable, so nothing can skip itself into a green build, and the only thing this task owes is a non-zero exit when the suite fails. That is also what makes the uploaded coverage honest: there is no configuration under which a package silently contributes zero.

That last point is load-bearing for the APKBUILD. options="!check" used to say the tests could not run in the VM — a claim that was never true — and now says they ran before abuild did. Which is a promise about this task: soften it and !check becomes a package built with nothing tested.

#build

REPODEST=$HOME/packages abuild -d builds and stages the apk.

-d disables abuild's dependency check: the makedepends are already installed by the manifest's packages: list, and abuild has no way to know that.

builddir="$startdir" in the APKBUILD means abuild packages this checkout in place rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet.

build() runs make css before the compile and make check-css after it, and package() runs make install-files rather than make install. Both are explained under css and install-files.

#publish

The [ ! -r ~/.apk-ci.env ] gate is the honest answer to a build that was handed no secrets, not a fallback: on a manual submission without the secret, every earlier task has still run and a signed apk is sitting in $HOME/packages, and saying so and exiting 0 is more useful than a red build about a credential nobody asked for. On a push the secret is there and this publishes.

set +x before sourcing ~/.apk-ci.env and set -x after. Every task runs under set -x, so without this the S3 access key and secret are echoed into a build log that is world-readable.

The upload is rclone copyto per file — copy only, never delete, never sync. A sync would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance, which is what lets a pinned deployment be rebuilt. abuild nests its output under $REPODEST/<repo>/<arch>/, so each file is copied to a fixed prefix rather than the tree being mirrored.

apk-mirror on phoebe re-indexes within 15 minutes; nothing here waits for it.

#publish_artifacts

The same apk, uploaded a second time — into the ~bigbes/main channel of artifacts.sr.ht, which indexes and signs it in the same request rather than on a 15-minute timer. One PUT per file to /api/v1/pkg/~bigbes/main/apk/v3.22, with the working token from ~/.srht-token as the bearer. No client binary is involved: the upload route takes the file as the request body, and curl is already installed for the cacher.

It is a task of its own and not two more lines inside publish for one reason: the S3 copy is what phoebe re-indexes today and this one is the road being opened. A refusal here has to be legible as this destination refusing, and it must not be able to undo an upload that already succeeded.

set +x around every curl, for the same reason publish has it: the token is on the command line and the log is world-readable.

Three answers are expected and only two of them are good:

• 201 — published, indexed, signed.
• 200 — the same version with the same bytes was already there. A rerun of a job that got as far as this task is idempotent, and stays green.
• 409 — the same version with different bytes. abuild stamps mtimes into the archive, so resubmitting one commit produces a byte-different apk under an identical pkgver; a published version is immutable, so the daemon keeps what it has. That is a WARNING and not a failure — it means the build was run twice, not that anything is wrong. Republishing means removing the version through the API first.

Anything else prints the daemon's error body and fails the task, after trying every remaining file: one bad package must not hide the fate of the others.

The loop is for file in $(find …) rather than find … | while read, because under ash the second form runs its body in a subshell and the failure flag set in it is lost when the pipeline ends — the task would report success it never had.

Gated on ~/.srht-token being readable, exactly as publish is gated on ~/.apk-ci.env: a manual submission without secrets still builds the apk, and says that nothing was published.

Installing what it publishes, on any Alpine box:

wget -qO /etc/apk/keys/bigbes@artifacts.srht.bigb.es.rsa.pub \
  https://artifacts.srht.bigb.es/~bigbes/keys/apk.rsa.pub
echo https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22 >> /etc/apk/repositories
apk update && apk add diff.sr.ht

The name of the key file is not free: apk looks a signature up by the name carried in the index's .SIGN.RSA256.<name> entry, so it has to be bigbes@artifacts.srht.bigb.es.rsa.pub and nothing else. With the key in place no --allow-untrusted is needed — the daemon signs the index it builds.

#cache_save

After publish, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped.

Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind || true.

There is no cacher exists "$KEY" || guard in front of either upload. It was dead code: without --force, cacher dir upload already skips a key that is present, so the guard asked the same question twice and its only real effect was to make a failure of the exists call look like a decision not to upload.

#artifacts

Two, both literal paths relative to $HOME: cover.out and bench.txt.

artifacts: has no globbing, which is why the apk is still not listed — its name carries the version and changes every commit. These two do not.

They are declared because the two uploads below can fail. An upload that 404s or 401s leaves the build red and the file downloadable, so the report can be POSTed by hand instead of re-running a fifteen-minute pipeline for it. That is also the whole content of the "no token" branch in each task: the work was done, the result is attached, nothing was sent.

#coverage

The profile test produced, POSTed to the instance's cov.sr.ht — this service reporting on itself, to a service it does not otherwise talk to.

POST $COVER_ORIGIN/api/v1/repos/$COVER_REPO/reports
     ?commit=<sha>&ref=<ref>&key=$JOB_ID&job_url=$JOB_URL

COVER_REPO is ~bigbes/sr-ht-compare, taken from the sources: line and not from the service name: the daemon was renamed to diff.sr.ht, the git repository was not, and it is the repository a report is filed under.

Near-last in the pipeline and nothing after it depends on it. A cov.sr.ht outage must not be able to lose an apk that was already built, tested and published.

#bench

make -s bench into $HOME/bench.txt, then the same shape of POST:

POST $BENCH_ORIGIN/api/v1/repos/$BENCH_REPO/runs
     ?commit=<sha>&ref=<ref>&key=$JOB_ID&job_url=$JOB_URL&visibility=public

visibility acts only on the POST that creates $BENCH_REPO; it is ignored afterwards.

-s keeps make from echoing the recipe into the body, which is not decoration: the body is parsed as benchfmt and a go test … line at the top of it is not.

A redirect and a cat, never | tee. A pipeline takes the exit status of its last command, so make bench | tee bench.txt would hand this task tee's status and let a failing benchmark upload its own wreckage as a green run.

#What is measured

gitx, the diffing path, which is what a request to this service is spent in: BenchmarkDiff (both the two-dot and three-dot grammars a compare URL can carry), BenchmarkDiffStat, BenchmarkCommitPatch, and the two pure functions underneath them, BenchmarkMapFilePatches and BenchmarkCutPatch.

They build their own bare repository with the local git — 40 files of 120 lines, a branch that diverged, and a head commit that edits, adds, deletes and renames — because the test fixture's whole history is a dozen lines, over which a benchmark measures opening a repository rather than diffing one.

BENCH_COUNT is 10, the family's number: bench.sr.ht's confidence interval for a point becomes finite at six repetitions and a comparison becomes significant at four, so a lower count uploads points it can only mark "low n".

bench is the one target that names its own -timeout (BENCH_TIMEOUT, 20m). Ten counts of every benchmark, each preceded by building a real git repository, on a VM some multiple slower than a laptop, can outrun go test's 10m default — and what that looks like is panic: test timed out over a goroutine dump, which reads like a hang in the service rather than an unlucky number in a Makefile.

#The greps

grep -q '^BenchmarkDiff/' "$HOME/bench.txt"
grep -q '^BenchmarkCommitPatch' "$HOME/bench.txt"
grep -q '^BenchmarkCutPatch/' "$HOME/bench.txt"

These are not belt-and-braces. go test -bench that matches nothing prints ok and exits 0, and an empty file is valid benchfmt — so a renamed, moved or deleted benchmark would upload nothing at all and the build would go green over it. The three names span the three shapes in the file (a sub-benchmark, a top-level one, and the pure-function pair), so a package that stopped compiling its benchmarks cannot slip through.

Nothing here is gated on an environment variable, so unlike the sibling services there is no DSN guard to write: no benchmark in this tree can skip itself, and a short file therefore means something went wrong rather than something was unavailable.

#The request

Both uploads share the same three rails, and each of them is here because it failed once somewhere in this family:

• set +x immediately before the curl. builds.sr.ht traces task bodies, and -H "Authorization: Bearer $(cat ~/.srht-token)" traced is a working token in a build log anyone can read.
• curl -sS --fail-with-body. --fail alone throws the body away, and the body is the service's JSON explanation; --fail-with-body prints it and still exits non-zero.
• Both ref prefixes stripped, refs/heads/ and refs/tags/. This pipeline builds tags too, and a tag build would otherwise file its report under a ref literally named refs/tags/v0.3.0.

GIT_REF is absent on a manually submitted build; ref is optional for both APIs, so it goes up empty. key=$JOB_ID is the idempotency key: re-running a job replaces its report rather than adding a second one.

The [ ! -r ~/.srht-token ] guard is not a fallback for a failed upload. With the file present the upload is fatal on purpose. It covers exactly one case — a submission that asked for no secrets — and it says so in the log.

#css

The stylesheet is compiled from scss/main.scss against the shared partials and is embedded into the binary by //go:embed static (web/templates.go), so it has to exist before the compiler runs. Hence make css at the top of build() and a fatal make check-css after the compile.

check-css counts the matches of web/static/main.min.*.css and fails on zero and on two. Two is as wrong as none and quieter: web/templates.go resolves cssGlob and takes the first match, so a second file makes the served stylesheet depend on readdir order.

#The committed stylesheet, and the two steps

web/static/main.min.79713f25.css is still tracked in this revision. The end state is the one the siblings have — the file is build output, ignored, and produced by CI — but getting there is two steps and this is the first.

Step one, this change: CI builds the partials, make css runs in build(), check-css gates the result, and the committed file stays in the index. The .gitignore patterns are already in place and do nothing to it, because a tracked file is unaffected by .gitignore.

build() therefore prints two lines right after make css:

git status --porcelain -- web/static || true
sha256sum web/static/main.min.*.css || true

Step two, after the first CI run, is decided by those two lines:

• Empty git status output — make css reproduced the committed bytes exactly, including the content hash in the filename. Nothing about the shipped service changes when the file leaves the index.
• A D line for main.min.79713f25.css plus an untracked main.min.<other>.css — the pipeline produces a different stylesheet than the one committed. That is the interesting case and it wants reading before it is dropped: a hash difference is either drift in CORE_VER/BOOTSTRAP_REV against whatever the committer had materialized locally, or a genuine change in scss/main.scss that was never recompiled. The sha256sum line names the new file so the comparison does not depend on scrolling the log.

Either way step two is git rm --cached web/static/main.min.79713f25.css and nothing else — the ignore patterns are already there. Do it in the first case straight away; in the second, reconcile the versions first, because dropping the committed file makes CI the only producer and whatever it produces is what ships.

Until step two lands, note that make css deletes a tracked file during build(), so the tree is dirty from that point on in the mismatch case. It costs nothing today — this service reads no VCS stamp of its own — but it is the reason step two should not sit around.

#install-files

package() calls make install-files, not make install.

abuild runs package() in a fresh abuild process under fakeroot, which re-sources the APKBUILD and never calls build(). So nothing build() exported reaches it — the GOCACHE/GOMODCACHE pins in particular. build in the Makefile is .PHONY (Go decides staleness itself; a real file target would never rebuild after a source edit), and install depends on build, so make install there recompiles the binary from a cold cache. The apk would then ship a second compilation that nothing in the pipeline had looked at.

install-files is the copying half with no build in front of it: it stages the very bytes build() produced and check-css was asked about. install -Dm755 on a missing comparesrht is a fatal error naming the file, which is the right report for the one way that target can be called too early.

install itself keeps the sub-make (@$(MAKE) install-files) rather than listing it as a third prerequisite: /usr/share/abuild/default.conf exports MAKEFLAGS=-j$(nproc), and under -j the prerequisites of one target run in parallel — the copying would start beside the build it is supposed to follow.

#What is not here

• No Postgres. This service has no database and its suites are hermetic by design; there is nothing for a postgres task to do, and adding one would buy an idle daemon in the VM.
• No apk in artifacts:. Its name carries the version and changes every commit, which artifacts: cannot express — it has no globbing. The apk's two destinations are S3 and artifacts.sr.ht.
• No matrix. One architecture, one image.

#Known wart

make install-files still runs install -Dm644 -t $(DESTDIR)$(STATICDIR) web/static/*, which stages bundle.<hash>.js and main.min.<hash>.css into /usr/share/sourcehut/diff.sr.ht/static. The binary already embeds both (//go:embed static in web/templates.go:25) and serves them from there, so those copies are dead weight in the apk and a second, divergable source of truth for the same bytes. The bench sibling installs no static assets at all for this reason. Left alone deliberately — it is not part of this change, and removing it wants a check that nothing in the deployment's nginx tree serves that directory directly.