# CI: what `.build.yml` does and why The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a `varchar(16384)`, so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time and reads like nothing in particular: the branch simply has no CI. Rationale therefore lives here, and the manifest carries pointers. The pipeline is one linear job on `alpine/edge`: install the cache helper, assemble the shared SCSS, make a throwaway signing key, decide the version, restore the Go caches, test, package with `abuild`, publish the apk to `repo.bigb.es/alpine/v3.22/bigbes`, save the caches. It is triggered by a push to the **sourcehut** side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on `git.srht.bigb.es`, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy. ## packages `sassc` and `minify` are for `make css`, not for the package: the stylesheet is compiled here and embedded into the binary, and the apk has no runtime dependency on either. `curl` is for the `cacher_install` bootstrap and `rclone` for `publish`. There is no `nodejs`/`npm`: `web/static/bundle..js` is committed, and `make bundle` is an upgrade step run by hand rather than a build step. There is no `postgresql`. This service is stateless — it holds no database at all — and its suites are hermetic: `gitx/fixture_test.go` builds a bare repository with the local `git`, and `web/web_test.go` is `httptest` plus `ecoretest`. Nothing in the tree talks to a live git.sr.ht API. ## secrets Three, all account-level and shared with the sibling services: | secret | lands at | used by | |---|---|---| | `apk-ci-s3` | `~/.apk-ci.env` | `publish` | | `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` | | `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` | They are file secrets. Listing them is what turns `publish` and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline and stops at `cacher_init`, which is the right place to notice. `apk-ci-s3` is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work. ## environment `CORE_VER` **must track the deployment's `SRHT_CORE_VER`**, and `BOOTSTRAP_REV` is the Bootstrap submodule commit `core.sr.ht` pins at that tag. The two being out of step means this service renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. They are `0.84.5` / `779ad9f1` here because that is what `srht/versions.env` pins in the deployment repo; bump all of them together. `REPO` is the clone directory `sources:` produces, and every task that touches the checkout `cd`s into it — task bodies do not inherit a working directory from each other. ## cacher `cacher` (`go.bigb.es/cacher`) is an S3-backed cache helper, installed from its own release on pages.sr.ht. It is **two tasks and not one** because `install.sh` appends its `PATH` export to `~/.buildenv`, and `~/.buildenv` is sourced by the *next* task's preamble — a single task would install the binary and then not find it on `PATH`. The installer is fetched with `curl -fsSL` and verifies what it downloads against `checksums.txt`. The predecessor of this task fetched the raw binary with a bare `curl -sSL`; without `-f`, an HTTP error is not an error to `curl`, so a 404 page was written to `~/.local/bin/cacher` and `chmod +x`-ed, and the first failure was a syntax error out of a shell reading HTML. ## scss `sassc` needs the shared sourcehut partials, and **no apk ships them**: they are core.sr.ht's own `scss/` plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in `/usr/share/sourcehut/scss`, which is the `ASSETS` the Makefile and the APKBUILD name. The whole thing is cached under a key made of the two pins (`scss/$CORE_VER-$BOOTSTRAP_REV.tar.zst`), so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing. `--exec` runs only on a cache miss and seeds the cache afterwards; the block sees exported variables only, which is why the cache key is spelled inline in the `cacher` invocation rather than computed inside the block, and why the block is in single quotes. It runs before `keygen` and before anything Go, because the stylesheet has to exist before the compiler does — see [css](#css). ## keygen `abuild-keygen -a -n -i -q` makes a throwaway package signing key. `-i` is not optional: without it the key is generated but not installed into `/etc/apk/keys`, and `abuild`'s own final "update the local repository index" step then dies with an UNTRUSTED signature after having built the package perfectly well. The key is per-build and dies with the VM. That is fine because nothing verifies these signatures — the apk index on `repo.bigb.es` is rebuilt and signed on phoebe by the garage stack's `apk-mirror` service, which indexes this repo with `--allow-untrusted` precisely because of this. ## version One `git describe` decides the apk's `pkgver`. Alpine's version grammar does not accept `v1.2.3-4-gabc1234`, so a describe with commits since the tag is rewritten to `1.2.3_git4`; `_git` sorts **after** the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all — which is this repository today — it falls back to `0.0.`, monotonic and unique per commit, which a deployment can pin. Tags are in `allow-refs` for exactly this reason: pushing `v0.9.0` is what produces the `0.9.0` apk. `$ver` is **exported** through `~/.buildenv` as `PKGVER`, and the APKBUILD reads `pkgver="${PKGVER:-0.0.0}"`. It is not `sed`-ed into the tracked APKBUILD, and this is the one thing in this file that must not be "tidied" back: > Go records `vcs.modified` in every binary it builds inside a repository, and > it reads that flag from `git status --porcelain`. Measured on go1.26.5, a > `sed -i` of a tracked file sets `vcs.modified=true` and Go stamps `+dirty` > into `Main.Version` for the whole `abuild` run — for every binary in the > package, permanently, for the life of the apk. The export alone is not enough. Any **untracked** file present at `go build` time does the same thing, and `abuild` works inside the checkout with `builddir="$startdir"` and exports `GOTMPDIR=$startdir/tmp` — so `./src`, `./pkg` and `./tmp` all appear inside the tree it is packaging. `.gitignore` covers those three, and Go does not count ignored files. The task ends with a bare `git status --porcelain` because this is the last moment the tree is provably clean, and the output of a clean one is nothing. ## cache_restore The Go module cache and build cache are keyed by the hash of `go.sum`, so a build that changed no dependency reuses both. `--optional` and not `|| true`. They are not the same promise: `--optional` returns 0 only for a genuine cache miss, while `|| true` swallows every other outcome too — bad credentials, a dead bucket, a typo in the prefix — and turns a misconfigured cache into a pipeline that is merely slow, silently, forever. The exports here are only for the repair block below and for `cache_save`: `abuild` redirects both caches into its throwaway `$tmpdir` (and an upstream typo in `abuild.in` slaves `GOMODCACHE` to `GOCACHE`), so environment exports from a task cannot stick. The APKBUILD's `build()` re-pins both to the same home locations, after abuild's own exports, which is what makes the tarballs saved at the end be the ones the compile used. ### The half-restored module cache The repair block is not defensive padding. A partially restored module cache is the *normal* failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test. So: make the tree writable (the module cache is mode 555 and `rm -rf` cannot remove it otherwise), ask `go mod verify` whether what came back is intact, and throw the whole thing away if it is not. Then `go mod download all` and verify for real, and let *that* failure be fatal. Do not soften the final `go mod verify` to `|| true`. A build that proceeds with a module cache it could not verify is a build whose result means nothing. The second `git status --porcelain` is there because `-mod=readonly` is the default and neither command *can* rewrite `go.mod` or `go.sum` — but a dirty tree at this point is a `+dirty` apk, so it is said out loud rather than trusted to a flag. ## test `gofmt -l .` piped to a file and then `test ! -s`, rather than a bare `gofmt -l`: `gofmt -l` exits 0 whether or not it printed anything, so the only way to fail on its output is to look at the output. The `tee` keeps the offending filenames in the build log, where they are the entire diagnosis. `go vet ./...` next, then `make test` — the Makefile, not a bare `go test ./...`, so that the test command lives in one place. No Postgres, no network and **no guard**. Every suite in this tree is hermetic: `gitx/fixture_test.go` builds a bare repository with the local `git` binary, and `web/web_test.go` is `httptest` plus `ecoretest`. Nothing here is gated on an environment variable, so nothing can skip itself into a green build, and the only thing this task owes is a non-zero exit when `make test` fails. That last point is load-bearing for the APKBUILD. `options="!check"` used to say the tests could not run in the VM — a claim that was never true — and now says they ran *before* `abuild` did. Which is a promise about **this** task: soften it and `!check` becomes a package built with nothing tested. ## build `REPODEST=$HOME/packages abuild -d` builds and stages the apk. `-d` disables abuild's dependency check: the makedepends are already installed by the manifest's `packages:` list, and abuild has no way to know that. `builddir="$startdir"` in the APKBUILD means abuild packages *this checkout in place* rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet. `build()` runs `make css` before the compile and `make check-css` after it, and `package()` runs `make install-files` rather than `make install`. Both are explained under [css](#css) and [install-files](#install-files). ## publish The `[ ! -r ~/.apk-ci.env ]` gate is the honest answer to a build that was handed no secrets, not a fallback: on a manual submission without the secret, every earlier task has still run and a signed apk is sitting in `$HOME/packages`, and saying so and exiting 0 is more useful than a red build about a credential nobody asked for. On a push the secret is there and this publishes. `set +x` before sourcing `~/.apk-ci.env` and `set -x` after. Every task runs under `set -x`, so without this the S3 access key and secret are echoed into a build log that is world-readable. The upload is `rclone copyto` per file — copy only, never delete, never sync. A `sync` would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance, which is what lets a pinned deployment be rebuilt. abuild nests its output under `$REPODEST///`, so each file is copied to a fixed prefix rather than the tree being mirrored. `apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it. ## cache_save **After `publish`**, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped. Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind `|| true`. There is no `cacher exists "$KEY" ||` guard in front of either upload. It was dead code: without `--force`, `cacher dir upload` already skips a key that is present, so the guard asked the same question twice and its only real effect was to make a failure of the `exists` call look like a decision not to upload. ## css The stylesheet is compiled from `scss/main.scss` against the shared partials and is embedded into the binary by `//go:embed static` (`web/templates.go`), so it has to exist **before** the compiler runs. Hence `make css` at the top of `build()` and a fatal `make check-css` after the compile. `check-css` counts the matches of `web/static/main.min.*.css` and fails on zero *and* on two. Two is as wrong as none and quieter: `web/templates.go` resolves `cssGlob` and takes the first match, so a second file makes the served stylesheet depend on readdir order. ### The committed stylesheet, and the two steps `web/static/main.min.79713f25.css` is **still tracked** in this revision. The end state is the one the siblings have — the file is build output, ignored, and produced by CI — but getting there is two steps and this is the first. Step one, this change: CI builds the partials, `make css` runs in `build()`, `check-css` gates the result, and the committed file stays in the index. The `.gitignore` patterns are already in place and do nothing to it, because a tracked file is unaffected by `.gitignore`. `build()` therefore prints two lines right after `make css`: ``` git status --porcelain -- web/static || true sha256sum web/static/main.min.*.css || true ``` Step two, after the first CI run, is decided by those two lines: - **Empty `git status` output** — `make css` reproduced the committed bytes exactly, including the content hash in the filename. Nothing about the shipped service changes when the file leaves the index. - **A `D` line for `main.min.79713f25.css` plus an untracked `main.min..css`** — the pipeline produces a different stylesheet than the one committed. That is the interesting case and it wants reading before it is dropped: a hash difference is either drift in `CORE_VER`/`BOOTSTRAP_REV` against whatever the committer had materialized locally, or a genuine change in `scss/main.scss` that was never recompiled. The `sha256sum` line names the new file so the comparison does not depend on scrolling the log. Either way step two is `git rm --cached web/static/main.min.79713f25.css` and nothing else — the ignore patterns are already there. Do it in the first case straight away; in the second, reconcile the versions first, because dropping the committed file makes CI the only producer and whatever it produces is what ships. Until step two lands, note that `make css` deletes a tracked file during `build()`, so the tree is dirty from that point on in the mismatch case. It costs nothing today — this service reads no VCS stamp of its own — but it is the reason step two should not sit around. ## install-files `package()` calls `make install-files`, not `make install`. `abuild` runs `package()` in a **fresh abuild process** under fakeroot, which re-sources the APKBUILD and never calls `build()`. So nothing `build()` exported reaches it — the `GOCACHE`/`GOMODCACHE` pins in particular. `build` in the Makefile is `.PHONY` (Go decides staleness itself; a real file target would never rebuild after a source edit), and `install` depends on `build`, so `make install` there recompiles the binary from a cold cache. The apk would then ship a second compilation that nothing in the pipeline had looked at. `install-files` is the copying half with no build in front of it: it stages the very bytes `build()` produced and `check-css` was asked about. `install -Dm755` on a missing `comparesrht` is a fatal error naming the file, which is the right report for the one way that target can be called too early. `install` itself keeps the sub-make (`@$(MAKE) install-files`) rather than listing it as a third prerequisite: `/usr/share/abuild/default.conf` exports `MAKEFLAGS=-j$(nproc)`, and under `-j` the prerequisites of one target run in parallel — the copying would start beside the build it is supposed to follow. ## What is not here - **No coverage upload.** The sibling services POST their profile to cover.sr.ht at the end of the pipeline. Adding it here needs a token secret and a repository on cover. - **No artifacts.** There is nothing to download: the apk goes to S3, and its name changes every commit, which `artifacts:` cannot express (it has no globbing). - **No matrix.** One architecture, one image. ## Known wart `make install-files` still runs `install -Dm644 -t $(DESTDIR)$(STATICDIR) web/static/*`, which stages `bundle..js` and `main.min..css` into `/usr/share/sourcehut/compare.sr.ht/static`. The binary already **embeds** both (`//go:embed static` in `web/templates.go:25`) and serves them from there, so those copies are dead weight in the apk and a second, divergable source of truth for the same bytes. The bench sibling installs no static assets at all for this reason. Left alone deliberately — it is not part of this change, and removing it wants a check that nothing in the deployment's nginx tree serves that directory directly.