#CI: what .build.yml does and why

The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a varchar(16384), so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time and reads like nothing in particular: the branch simply has no CI. Rationale therefore lives here, and the manifest carries pointers.

The pipeline is one linear job on alpine/edge: install the cache helper, assemble the shared SCSS, make a throwaway signing key, decide the version, restore the Go caches, test, package with abuild, publish the apk to repo.bigb.es/alpine/v3.22/bigbes, save the caches.

It is triggered by a push to the sourcehut side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on git.srht.bigb.es, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy.

#packages

sassc and minify are for make css, not for the package: the stylesheet is compiled here and embedded into the binary, and the apk has no runtime dependency on either.

curl is for the cacher_install bootstrap and rclone for publish. There is no nodejs/npm: web/static/bundle.<hash>.js is committed, and make bundle is an upgrade step run by hand rather than a build step.

There is no postgresql. This service is stateless — it holds no database at all — and its suites are hermetic: gitx/fixture_test.go builds a bare repository with the local git, and web/web_test.go is httptest plus ecoretest. Nothing in the tree talks to a live git.sr.ht API.

#secrets

Three, all account-level and shared with the sibling services:

They are file secrets. Listing them is what turns publish and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline and stops at cacher_init, which is the right place to notice.

apk-ci-s3 is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work.

#environment

CORE_VER must track the deployment's SRHT_CORE_VER, and BOOTSTRAP_REV is the Bootstrap submodule commit core.sr.ht pins at that tag. The two being out of step means this service renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. They are 0.84.5 / 779ad9f1 here because that is what srht/versions.env pins in the deployment repo; bump all of them together.

REPO is the clone directory sources: produces, and every task that touches the checkout cds into it — task bodies do not inherit a working directory from each other.

#cacher

cacher (go.bigb.es/cacher) is an S3-backed cache helper, installed from its own release on pages.sr.ht.

It is two tasks and not one because install.sh appends its PATH export to ~/.buildenv, and ~/.buildenv is sourced by the next task's preamble — a single task would install the binary and then not find it on PATH.

The installer is fetched with curl -fsSL and verifies what it downloads against checksums.txt. The predecessor of this task fetched the raw binary with a bare curl -sSL; without -f, an HTTP error is not an error to curl, so a 404 page was written to ~/.local/bin/cacher and chmod +x-ed, and the first failure was a syntax error out of a shell reading HTML.

#scss

sassc needs the shared sourcehut partials, and no apk ships them: they are core.sr.ht's own scss/ plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in /usr/share/sourcehut/scss, which is the ASSETS the Makefile and the APKBUILD name.

The whole thing is cached under a key made of the two pins (scss/$CORE_VER-$BOOTSTRAP_REV.tar.zst), so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing. --exec runs only on a cache miss and seeds the cache afterwards; the block sees exported variables only, which is why the cache key is spelled inline in the cacher invocation rather than computed inside the block, and why the block is in single quotes.

It runs before keygen and before anything Go, because the stylesheet has to exist before the compiler does — see css.

#keygen

abuild-keygen -a -n -i -q makes a throwaway package signing key.

-i is not optional: without it the key is generated but not installed into /etc/apk/keys, and abuild's own final "update the local repository index" step then dies with an UNTRUSTED signature after having built the package perfectly well.

The key is per-build and dies with the VM. That is fine because nothing verifies these signatures — the apk index on repo.bigb.es is rebuilt and signed on phoebe by the garage stack's apk-mirror service, which indexes this repo with --allow-untrusted precisely because of this.

#version

One git describe decides the apk's pkgver. Alpine's version grammar does not accept v1.2.3-4-gabc1234, so a describe with commits since the tag is rewritten to 1.2.3_git4; _git sorts after the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all — which is this repository today — it falls back to 0.0.<commit count>, monotonic and unique per commit, which a deployment can pin.

Tags are in allow-refs for exactly this reason: pushing v0.9.0 is what produces the 0.9.0 apk.

$ver is exported through ~/.buildenv as PKGVER, and the APKBUILD reads pkgver="${PKGVER:-0.0.0}". It is not sed-ed into the tracked APKBUILD, and this is the one thing in this file that must not be "tidied" back:

Go records vcs.modified in every binary it builds inside a repository, and it reads that flag from git status --porcelain. Measured on go1.26.5, a sed -i of a tracked file sets vcs.modified=true and Go stamps +dirty into Main.Version for the whole abuild run — for every binary in the package, permanently, for the life of the apk.

The export alone is not enough. Any untracked file present at go build time does the same thing, and abuild works inside the checkout with builddir="$startdir" and exports GOTMPDIR=$startdir/tmp — so ./src, ./pkg and ./tmp all appear inside the tree it is packaging. .gitignore covers those three, and Go does not count ignored files.

The task ends with a bare git status --porcelain because this is the last moment the tree is provably clean, and the output of a clean one is nothing.

#cache_restore

The Go module cache and build cache are keyed by the hash of go.sum, so a build that changed no dependency reuses both.

--optional and not || true. They are not the same promise: --optional returns 0 only for a genuine cache miss, while || true swallows every other outcome too — bad credentials, a dead bucket, a typo in the prefix — and turns a misconfigured cache into a pipeline that is merely slow, silently, forever.

The exports here are only for the repair block below and for cache_save: abuild redirects both caches into its throwaway $tmpdir (and an upstream typo in abuild.in slaves GOMODCACHE to GOCACHE), so environment exports from a task cannot stick. The APKBUILD's build() re-pins both to the same home locations, after abuild's own exports, which is what makes the tarballs saved at the end be the ones the compile used.

#The half-restored module cache

The repair block is not defensive padding. A partially restored module cache is the normal failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test.

So: make the tree writable (the module cache is mode 555 and rm -rf cannot remove it otherwise), ask go mod verify whether what came back is intact, and throw the whole thing away if it is not. Then go mod download all and verify for real, and let that failure be fatal.

Do not soften the final go mod verify to || true. A build that proceeds with a module cache it could not verify is a build whose result means nothing.

The second git status --porcelain is there because -mod=readonly is the default and neither command can rewrite go.mod or go.sum — but a dirty tree at this point is a +dirty apk, so it is said out loud rather than trusted to a flag.

#test

gofmt -l . piped to a file and then test ! -s, rather than a bare gofmt -l: gofmt -l exits 0 whether or not it printed anything, so the only way to fail on its output is to look at the output. The tee keeps the offending filenames in the build log, where they are the entire diagnosis.

go vet ./... next, then make test — the Makefile, not a bare go test ./..., so that the test command lives in one place.

No Postgres, no network and no guard. Every suite in this tree is hermetic: gitx/fixture_test.go builds a bare repository with the local git binary, and web/web_test.go is httptest plus ecoretest. Nothing here is gated on an environment variable, so nothing can skip itself into a green build, and the only thing this task owes is a non-zero exit when make test fails.

That last point is load-bearing for the APKBUILD. options="!check" used to say the tests could not run in the VM — a claim that was never true — and now says they ran before abuild did. Which is a promise about this task: soften it and !check becomes a package built with nothing tested.

#build

REPODEST=$HOME/packages abuild -d builds and stages the apk.

-d disables abuild's dependency check: the makedepends are already installed by the manifest's packages: list, and abuild has no way to know that.

builddir="$startdir" in the APKBUILD means abuild packages this checkout in place rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet.

build() runs make css before the compile and make check-css after it, and package() runs make install-files rather than make install. Both are explained under css and install-files.

#publish

The [ ! -r ~/.apk-ci.env ] gate is the honest answer to a build that was handed no secrets, not a fallback: on a manual submission without the secret, every earlier task has still run and a signed apk is sitting in $HOME/packages, and saying so and exiting 0 is more useful than a red build about a credential nobody asked for. On a push the secret is there and this publishes.

set +x before sourcing ~/.apk-ci.env and set -x after. Every task runs under set -x, so without this the S3 access key and secret are echoed into a build log that is world-readable.

The upload is rclone copyto per file — copy only, never delete, never sync. A sync would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance, which is what lets a pinned deployment be rebuilt. abuild nests its output under $REPODEST/<repo>/<arch>/, so each file is copied to a fixed prefix rather than the tree being mirrored.

apk-mirror on phoebe re-indexes within 15 minutes; nothing here waits for it.

#cache_save

After publish, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped.

Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind || true.

There is no cacher exists "$KEY" || guard in front of either upload. It was dead code: without --force, cacher dir upload already skips a key that is present, so the guard asked the same question twice and its only real effect was to make a failure of the exists call look like a decision not to upload.

#css

The stylesheet is compiled from scss/main.scss against the shared partials and is embedded into the binary by //go:embed static (web/templates.go), so it has to exist before the compiler runs. Hence make css at the top of build() and a fatal make check-css after the compile.

check-css counts the matches of web/static/main.min.*.css and fails on zero and on two. Two is as wrong as none and quieter: web/templates.go resolves cssGlob and takes the first match, so a second file makes the served stylesheet depend on readdir order.

#The committed stylesheet, and the two steps

web/static/main.min.79713f25.css is still tracked in this revision. The end state is the one the siblings have — the file is build output, ignored, and produced by CI — but getting there is two steps and this is the first.

Step one, this change: CI builds the partials, make css runs in build(), check-css gates the result, and the committed file stays in the index. The .gitignore patterns are already in place and do nothing to it, because a tracked file is unaffected by .gitignore.

build() therefore prints two lines right after make css:

git status --porcelain -- web/static || true
sha256sum web/static/main.min.*.css || true

Step two, after the first CI run, is decided by those two lines:

• Empty git status output — make css reproduced the committed bytes exactly, including the content hash in the filename. Nothing about the shipped service changes when the file leaves the index.
• A D line for main.min.79713f25.css plus an untracked main.min.<other>.css — the pipeline produces a different stylesheet than the one committed. That is the interesting case and it wants reading before it is dropped: a hash difference is either drift in CORE_VER/BOOTSTRAP_REV against whatever the committer had materialized locally, or a genuine change in scss/main.scss that was never recompiled. The sha256sum line names the new file so the comparison does not depend on scrolling the log.

Either way step two is git rm --cached web/static/main.min.79713f25.css and nothing else — the ignore patterns are already there. Do it in the first case straight away; in the second, reconcile the versions first, because dropping the committed file makes CI the only producer and whatever it produces is what ships.

Until step two lands, note that make css deletes a tracked file during build(), so the tree is dirty from that point on in the mismatch case. It costs nothing today — this service reads no VCS stamp of its own — but it is the reason step two should not sit around.

#install-files

package() calls make install-files, not make install.

abuild runs package() in a fresh abuild process under fakeroot, which re-sources the APKBUILD and never calls build(). So nothing build() exported reaches it — the GOCACHE/GOMODCACHE pins in particular. build in the Makefile is .PHONY (Go decides staleness itself; a real file target would never rebuild after a source edit), and install depends on build, so make install there recompiles the binary from a cold cache. The apk would then ship a second compilation that nothing in the pipeline had looked at.

install-files is the copying half with no build in front of it: it stages the very bytes build() produced and check-css was asked about. install -Dm755 on a missing comparesrht is a fatal error naming the file, which is the right report for the one way that target can be called too early.

install itself keeps the sub-make (@$(MAKE) install-files) rather than listing it as a third prerequisite: /usr/share/abuild/default.conf exports MAKEFLAGS=-j$(nproc), and under -j the prerequisites of one target run in parallel — the copying would start beside the build it is supposed to follow.

#What is not here

• No coverage upload. The sibling services POST their profile to cover.sr.ht at the end of the pipeline. Adding it here needs a token secret and a repository on cover.
• No artifacts. There is nothing to download: the apk goes to S3, and its name changes every commit, which artifacts: cannot express (it has no globbing).
• No matrix. One architecture, one image.

#Known wart

make install-files still runs install -Dm644 -t $(DESTDIR)$(STATICDIR) web/static/*, which stages bundle.<hash>.js and main.min.<hash>.css into /usr/share/sourcehut/compare.sr.ht/static. The binary already embeds both (//go:embed static in web/templates.go:25) and serves them from there, so those copies are dead weight in the apk and a second, divergable source of truth for the same bytes. The bench sibling installs no static assets at all for this reason. Left alone deliberately — it is not part of this change, and removing it wants a check that nothing in the deployment's nginx tree serves that directory directly.