# CI: what `.build.yml` does and why The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a `varchar(16384)`, so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time, it reads like nothing in particular, and what it produces is a branch with no CI rather than a red build. Rationale therefore lives here, and the manifest carries pointers. The pipeline is one linear job on `alpine/edge`: install the cache helper, assemble the shared SCSS, stamp a version, restore the Go caches, start a Postgres, test, package with `abuild`, publish the apk to `repo.bigb.es/alpine/v3.22/bigbes`, save the caches. It is triggered by a push to the **sourcehut** side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on `git.srht.bigb.es`, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy. ## packages `postgresql` and `postgresql-client` are for the test suite, not for the package. The apk declares no runtime dependency on Postgres — the daemon talks to whatever `connection-string` names, which in production is another host. `sassc` and `minify` are the stylesheet pipeline (`make css`), and they are build-time only: the compiled CSS is embedded into the binary by `//go:embed`, so nothing at runtime needs either. `rclone` is for `publish` and `curl` for the cacher bootstrap. ## secrets Three, all account-level and shared with the sibling services: | secret | lands at | used by | |---|---|---| | `apk-ci-s3` | `~/.apk-ci.env` | `publish` | | `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` | | `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` | They are file secrets. Listing them is what turns `publish` and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline, and `publish` says so and exits 0 rather than failing. `apk-ci-s3` is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work. ## cacher `cacher` is an S3-backed cache helper, installed from pages.sr.ht. It is two tasks and not one because `install.sh` appends its `PATH` export to `~/.buildenv`, and `~/.buildenv` is sourced by the *next* task's preamble — a single task would install it and then not find it. It is installed through `install.sh` rather than by `curl`-ing the raw binary: the installer verifies the download against the published `checksums.txt`, and `curl -fsSL` fails on an HTTP error instead of writing the error page to the destination and marking it executable. ## scss `sassc` needs the shared sourcehut partials, and no apk ships them: they are core.sr.ht's own `scss/` plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in `/usr/share/sourcehut/scss`, cached under a key made of the two pins so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing. **`CORE_VER` must track the deployment's `SRHT_CORE_VER`.** The two being out of step means this daemon renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. `BOOTSTRAP_REV` is the submodule commit core.sr.ht pins at that tag; bump the two together. The `--exec` form is the cache helper's own miss-and-seed: the block runs only on a miss and the directory is uploaded afterwards, which is one flow instead of an `if ! cacher dir download … fi` with a matching upload inside it that has to be kept in step by hand. It sees exported variables only, which is why the cache key is spelled inline in the `cacher` invocation rather than computed inside the block, and why the block is in single quotes. The cache key here is not the `--prefix` `cacher_init` sets. That one namespaces this repository's Go caches; the SCSS tree is keyed by the two upstream pins and nothing about this repository, so it is deliberately not hoisted into a shared prefix. ## keygen `abuild-keygen -a -n -i -q` makes a throwaway package signing key: generated per build, dies with the VM, trusted by nothing. Clients verify against the index instead, which is rebuilt and signed on phoebe by the garage stack's `apk-mirror` service — it indexes this repo with `--allow-untrusted` precisely because of this. `-i` is not optional. Without it the key is generated but not installed into `/etc/apk/keys`, and abuild's own final "update the local repository index" step dies with an UNTRUSTED signature after having built the package perfectly well. ## version One `git describe` decides the apk's `pkgver`. Alpine's version grammar does not accept `v1.2.3-4-gabc1234`, so a describe with commits since the tag is rewritten to `1.2.3_git4`. `_git` sorts **after** the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all it falls back to `0.0.`. Tags are in `allow-refs` for exactly this reason: pushing `v0.9.0` is what produces the `0.9.0` apk. ### Why it is exported and never `sed`-ed The version reaches `abuild` as an environment variable — `APKBUILD` reads `pkgver="${PKGVER:-0.0.0}"` — and the tracked `APKBUILD` is never rewritten. This is not style. Go records `vcs.modified` in every binary compiled inside a git repository and reads that flag from `git status --porcelain`. A CI task that `sed`s a tracked file leaves the tree dirty for the whole `abuild` run, so `Main.Version` in the packaged binary carries `+dirty` for the life of the apk. Measured on go1.26.5. The same measurement is why `.gitignore` matters here more than tidiness would suggest: `git status --porcelain` counts **untracked** files too. `make css` writes `web/static/main.min..css` *before* `go build` runs, and that name was not ignored — only the intermediate `main.css` was — so every packaging run compiled the binaries with an untracked file sitting in the tree. `/src/`, `/pkg/` and `/tmp/` are the three directories `abuild` works in with `source=""` and `builddir="$startdir"`, all of them inside the checkout being packaged, and abuild exports `GOTMPDIR=$startdir/tmp` on top of that. The `version` task ends with `git status --porcelain` because this is the last moment the tree is provably clean, and everything that could dirty it happens after. ## cache_restore The Go module cache and build cache are keyed by the hash of `go.sum`, so a build that changed no dependency reuses both. `--optional` turns a miss into a cold build rather than a failure — `|| true` did the same thing while also swallowing a real error, such as a bad endpoint or expired credentials. `abuild` redirects both caches into its throwaway `$tmpdir` (and an upstream typo slaves `GOMODCACHE` to `GOCACHE`), so exports from this task cannot stick: the `APKBUILD`'s `build()` re-pins both to the home locations restored here. ### The half-restored module cache The repair block is not defensive padding. A partially restored module cache is the *normal* failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test. So: make the tree writable (the module cache is mode 555 and `rm -rf` cannot remove it otherwise), ask `go mod verify` whether what came back is intact, and throw the whole thing away if it is not. Then download and verify for real, and let *that* failure be fatal. Do not soften the final `go mod verify` to `|| true`. A build that proceeds with a module cache it could not verify is a build whose result means nothing. ### Why the download is not `go mod download all` `all` is not a harmless "be thorough" flag. It resolves the entire module graph — the test dependencies of dependencies included — and **appends their hashes to the tracked `go.sum`**: 170 lines on this tree, measured with go1.26.5. A modified tracked file in the checkout at `go build` time is exactly the dirty stamp the [version](#why-it-is-exported-and-never-sed-ed) section is about, so the task written to warm the cache would have undone the fix. Plain `go mod download` leaves `go.sum` byte-identical and `go mod verify` still passes after it, which is the whole of what this task needs: the build imports what the main module imports, and those hashes are already in `go.sum` because they are what `go.sum` is. This is not theoretical either — it is what the sibling bench.sr.ht build #359 failed on, at its `check-version` gate, with `M go.sum` named as the culprit. ## postgres A real Postgres in the VM, initialised from scratch each build. It is what turns this repository's 93 test files, across 16 packages, from files that compile on the builder into tests that ran there. `db/db_test.go`, `service/fixture_test.go` and `cmd/specsrht-migrate/main_test.go` all gate their integration cases on `SPECSRHT_TEST_PG`, and without it **67 tests skip** — measured on this tree by counting `--- SKIP` with and without the variable. The persistence layer, the GraphQL/service layer over it and the migration runner are all in that set, the migration-agreement check included: the one that catches `schema.sql` and `migrations/` drifting apart, which is a defect a fresh install and an upgraded instance disagree about and nothing else notices. `fsync=off`, `full_page_writes=off` and `synchronous_commit=off` are safe here and only here: the database lives for the length of one build and its durability guarantees protect nothing. `createuser -s "$(id -un)"` makes the build user a superuser and the DSN carries no password, because the tests create a scratch schema per test (`specsrht_test_`), apply `schema.sql` into it and drop it afterwards. One database therefore serves the whole suite, and no `CREATE DATABASE` right is needed. ## test The guard on an empty `SPECSRHT_TEST_PG` exists because the failure it prevents is silent. If the `postgres` task did not export the DSN — or if someone reordered the two tasks — every database suite skips with a friendly message, `go test` exits 0, and the build is green over untested code. An explicit refusal is the difference between a broken pipeline and a lying one. It carries more weight here than in the siblings: `options="!check"` in the `APKBUILD` means abuild runs no tests at all, so this task is the *only* place the suites run. `gofmt` and `go vet` run here too, before the tests, so a formatting regression fails the build rather than waiting for someone to notice in review. `gofmt` needs the `test -z "$(gofmt -l .)"` spelling because `gofmt -l` reports the files it would change and still exits 0. The suite is invoked as `make test` and not as a bare `go test ./...`: the Makefile is where this repository's `-timeout` is named (`TEST_TIMEOUT?=20m`), and a second copy of that number here is a second copy to forget. The default `go test` timeout is ten minutes, it is silent about being a default, and what it produces on a slow builder is a goroutine dump rather than a failure anyone can read. ## build `REPODEST=$HOME/packages abuild -d` builds and stages the apk. `-d` disables abuild's dependency check: the makedepends are already installed by the manifest's `packages:` list, and abuild has no way to know that. `builddir="$startdir"` in `APKBUILD` means abuild packages *this checkout in place* rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet. `build()` runs `make css`, then `make check-css`, then `make build`. The order is forced by `//go:embed`: `web/templates.go` embeds the whole `static` directory, so a stylesheet built after the compiler ran is a stylesheet no binary contains. `go build` succeeds perfectly well with an unstyled `static/` — the embed takes the directory, not the file — so `check-css` is the only thing between that mistake and an unstyled service in production. `package()` calls `make install-files`, not `make install`, and this is the one line in the file that looks like a micro-optimisation and is not. abuild runs `package()` in a **fresh** abuild process under fakeroot, which re-sources the `APKBUILD` and never calls `build()`. Nothing `build()` exported reaches it — the Go cache pins included — and the binary targets are `.PHONY`, so `make install` relinked both binaries from a cold module cache. The apk therefore shipped a *second* binary, compiled in a different environment, that no task in this pipeline had ever tested. `install-files` is the copying half of `install` with no build in front of it, so what is packaged is the bytes `build()` produced. Static assets are not installed by either target. They are inside the binary; a second copy under `$(ASSETS)/spec.sr.ht/static` is dead weight in the apk that nothing reads. ## publish The `[ ! -r ~/.apk-ci.env ]` gate is the honest answer to a build that was handed no secrets, and not a fallback: every earlier task has still run and a signed apk is sitting in `$HOME/packages`, so the useful part of a manual submission has happened and only the upload cannot. On a push the secret is there and this publishes. `set +x` before sourcing `~/.apk-ci.env` and `set -x` after. Every task runs under `set -x`, so without this the S3 access key and secret are echoed into a build log that is world-readable. The upload is `rclone copyto` per file — copy only, never delete, never sync. A `sync` would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance. `apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it. ## cache_save After `publish`, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped. Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind `|| true`. There is no `cacher exists "$KEY" ||` guard in front of either upload any more. Without `--force` an upload skips a key that is already there, so the guard was a second round-trip that asked the question the upload asks anyway — and one that goes wrong in the direction that matters, since a key that exists but is truncated is exactly the half-restored cache the repair block above is about. ## What is not here - **No coverage upload.** The sibling services POST their profile to cover.sr.ht at the end of the pipeline. Adding it here needs a token secret and a repository on cover. - **No artifacts.** There is nothing to download: the apk goes to S3, and its name changes every commit, which `artifacts:` cannot express (it has no globbing). - **No matrix.** One architecture, one image.