# CI: what `.build.yml` does and why The manifest is deliberately short on prose. builds.sr.ht stores a submitted manifest in a `varchar(16384)`, so a manifest that grows past 16 KiB cannot be submitted at all — the failure is at submission time and reads like nothing in particular: the branch simply has no CI. Rationale therefore lives here, and the manifest carries pointers. The pipeline is one linear job on `alpine/edge`: install the cache helper, assemble the shared SCSS, make a throwaway signing key, decide the version, restore the Go caches, test, package with `abuild`, publish the apk to `repo.bigb.es/alpine/v3.22/bigbes`, save the caches. It is triggered by a push to the **sourcehut** side. A push to sourcecraft cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on `git.srht.bigb.es`, and that push is what submits the job. So the mirror is on the critical path for the package repository, not merely an offsite copy. ## packages `sassc` and `minify` are for `make css`, not for the package: the stylesheet is compiled here and embedded into the binary, and the apk has no runtime dependency on either. `curl` is for the `cacher_install` bootstrap and `rclone` for `publish`. There is no `nodejs`/`npm`: `web/static/bundle..js` is committed, and `make bundle` is an upgrade step run by hand rather than a build step. There is no `postgresql`. This service is stateless — it holds no database at all — and its suites are hermetic: `gitx/fixture_test.go` builds a bare repository with the local `git`, and `web/web_test.go` is `httptest` plus `ecoretest`. Nothing in the tree talks to a live git.sr.ht API. ## secrets Four, all account-level and shared with the sibling services: | secret | lands at | used by | |---|---|---| | `apk-ci-s3` | `~/.apk-ci.env` | `publish` | | `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` | | `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` | | `c7968415-…` | `~/.srht-token` | `publish_artifacts` | They are file secrets. Listing them is what turns `publish` and the cache tasks on; a manual submission that asks for no secrets still runs the interesting part of the pipeline and stops at `cacher_init`, which is the right place to notice. `apk-ci-s3` is referenced by name and the other two by UUID, which is only because that is how they were written in the donor manifests; both forms work. ## environment `CORE_VER` **must track the deployment's `SRHT_CORE_VER`**, and `BOOTSTRAP_REV` is the Bootstrap submodule commit `core.sr.ht` pins at that tag. The two being out of step means this service renders against different partials than the rest of the instance, which shows up as a page that is subtly the wrong shape and as nothing at all in any log. They are `0.84.5` / `779ad9f1` here because that is what `srht/versions.env` pins in the deployment repo; bump all of them together. `REPO` is the clone directory `sources:` produces, and every task that touches the checkout `cd`s into it — task bodies do not inherit a working directory from each other. ## cacher `cacher` (`go.bigb.es/cacher`) is an S3-backed cache helper, installed from its own release on pages.sr.ht. It is **two tasks and not one** because `install.sh` appends its `PATH` export to `~/.buildenv`, and `~/.buildenv` is sourced by the *next* task's preamble — a single task would install the binary and then not find it on `PATH`. The installer is fetched with `curl -fsSL` and verifies what it downloads against `checksums.txt`. The predecessor of this task fetched the raw binary with a bare `curl -sSL`; without `-f`, an HTTP error is not an error to `curl`, so a 404 page was written to `~/.local/bin/cacher` and `chmod +x`-ed, and the first failure was a syntax error out of a shell reading HTML. ## scss `sassc` needs the shared sourcehut partials, and **no apk ships them**: they are core.sr.ht's own `scss/` plus the Bootstrap 4 tree it pins as a submodule. The task clones both and drops them in `/usr/share/sourcehut/scss`, which is the `ASSETS` the Makefile and the APKBUILD name. The whole thing is cached under a key made of the two pins (`scss/$CORE_VER-$BOOTSTRAP_REV.tar.zst`), so an outage at git.sr.ht or GitHub cannot fail a build that changed nothing. `--exec` runs only on a cache miss and seeds the cache afterwards; the block sees exported variables only, which is why the cache key is spelled inline in the `cacher` invocation rather than computed inside the block, and why the block is in single quotes. It runs before `keygen` and before anything Go, because the stylesheet has to exist before the compiler does — see [css](#css). ## keygen `abuild-keygen -a -n -i -q` makes a throwaway package signing key. `-i` is not optional: without it the key is generated but not installed into `/etc/apk/keys`, and `abuild`'s own final "update the local repository index" step then dies with an UNTRUSTED signature after having built the package perfectly well. The key is per-build and dies with the VM. That is fine because nothing verifies these signatures — the apk index on `repo.bigb.es` is rebuilt and signed on phoebe by the garage stack's `apk-mirror` service, which indexes this repo with `--allow-untrusted` precisely because of this. ## version One `git describe` decides the apk's `pkgver`. Alpine's version grammar does not accept `v1.2.3-4-gabc1234`, so a describe with commits since the tag is rewritten to `1.2.3_git4`; `_git` sorts **after** the release in Alpine's comparison, which is what makes an untagged master build look newer than the tag it follows. With no tags at all — which is this repository today — it falls back to `0.0.`, monotonic and unique per commit, which a deployment can pin. Tags are in `allow-refs` for exactly this reason: pushing `v0.9.0` is what produces the `0.9.0` apk. `$ver` is **exported** through `~/.buildenv` as `PKGVER`, and the APKBUILD reads `pkgver="${PKGVER:-0.0.0}"`. It is not `sed`-ed into the tracked APKBUILD, and this is the one thing in this file that must not be "tidied" back: > Go records `vcs.modified` in every binary it builds inside a repository, and > it reads that flag from `git status --porcelain`. Measured on go1.26.5, a > `sed -i` of a tracked file sets `vcs.modified=true` and Go stamps `+dirty` > into `Main.Version` for the whole `abuild` run — for every binary in the > package, permanently, for the life of the apk. The export alone is not enough. Any **untracked** file present at `go build` time does the same thing, and `abuild` works inside the checkout with `builddir="$startdir"` and exports `GOTMPDIR=$startdir/tmp` — so `./src`, `./pkg` and `./tmp` all appear inside the tree it is packaging. `.gitignore` covers those three, and Go does not count ignored files. The task ends with a bare `git status --porcelain` because this is the last moment the tree is provably clean, and the output of a clean one is nothing. ## cache_restore The Go module cache and build cache are keyed by the hash of `go.sum`, so a build that changed no dependency reuses both. `--optional` and not `|| true`. They are not the same promise: `--optional` returns 0 only for a genuine cache miss, while `|| true` swallows every other outcome too — bad credentials, a dead bucket, a typo in the prefix — and turns a misconfigured cache into a pipeline that is merely slow, silently, forever. The exports here are only for the repair block below and for `cache_save`: `abuild` redirects both caches into its throwaway `$tmpdir` (and an upstream typo in `abuild.in` slaves `GOMODCACHE` to `GOCACHE`), so environment exports from a task cannot stick. The APKBUILD's `build()` re-pins both to the same home locations, after abuild's own exports, which is what makes the tarballs saved at the end be the ones the compile used. ### The half-restored module cache The repair block is not defensive padding. A partially restored module cache is the *normal* failure mode of this arrangement — an interrupted upload, a truncated object, a key written while a build was still running — and what it produces is a compile error deep in a dependency, which reads exactly like a bug in the code under test. So: make the tree writable (the module cache is mode 555 and `rm -rf` cannot remove it otherwise), ask `go mod verify` whether what came back is intact, and throw the whole thing away if it is not. Then `go mod download` and verify for real, and let *that* failure be fatal. Do not soften the final `go mod verify` to `|| true`. A build that proceeds with a module cache it could not verify is a build whose result means nothing. ### Not `go mod download all` The warm-up is `go mod download`. **The `all` must not come back**, and it is worth knowing why, because `all` is exactly the shape a future reader adds as an obvious improvement — warm *everything*, surely, so nothing is fetched later. `go mod download all` resolves the entire module graph, test dependencies of dependencies included, and **appends the hashes of everything it resolved to `go.sum`**. Measured on this tree: 145 lines added. In the bench sibling, 228 — and that is not a hypothetical, it is what failed its build #359, which stopped at `check-version` with a `-dirty` binary and `M go.sum` named as the thing in the way. A modified tracked file in the checkout when `go build` runs is precisely the stamp this whole pipeline is arranged to avoid — the same failure the `version` task refuses to cause by not `sed`-ing the APKBUILD. Warming the cache with a command that edits the tree would have reintroduced it one task later. Without `all`, `go.sum` is untouched, `go mod verify` still passes afterwards, and what gets fetched is what the main module actually builds. That is all a cache warm-up needs; the packages `all` would have added are ones no build in this pipeline compiles. Verified in a fresh clone of this branch — `go mod download` followed by `git status --porcelain` leaves the tree clean. compare has the smallest dependency graph of the six services, so this is where `all` came closest to being harmless, and it still rewrote 145 lines. ### `-mod=readonly` does not save you There used to be a sentence here, and another above the `git status` line in the manifest, saying `-mod=readonly` is the default so neither command *can* rewrite `go.mod` or `go.sum`. **It was false**, and it is a large part of why nobody looked at `go mod download all` for as long as they did: it sat directly above the line that disproved it and invited the reader to skip both. `readonly` governs updates to the module **requirements**. It does not stop writes to `go.sum`. Measured on this tree rather than inherited, with the flag set explicitly on the command line: | command | `go.sum` | exit | |---|---|---| | `GOFLAGS=-mod=readonly go mod download all` | **+145 lines** | 0 | | `GOFLAGS=-mod=readonly go mod download` | untouched | 0 | Both still pass `go mod verify` afterwards, so verification is no guard either. Set the flag **on the command line** when reproducing this. `go env -w` writes a persistent `GOFLAGS`, and a machine carrying `go env -w GOFLAGS=-mod=mod` — this one does, and bench's did too — makes a run that omits the flag prove nothing about what the builder would do. Check with `go env GOFLAGS` first. So the `git status --porcelain` at the end of the task is not belt-and-braces, and no flag stands behind it: **it is the check**. It is the only thing in this pipeline that would notice a `go.sum` rewrite before the compile, and on the bench sibling it is what did. Do not delete it as redundant and do not replace it with an appeal to a flag. ## test `gofmt -l .` piped to a file and then `test ! -s`, rather than a bare `gofmt -l`: `gofmt -l` exits 0 whether or not it printed anything, so the only way to fail on its output is to look at the output. The `tee` keeps the offending filenames in the build log, where they are the entire diagnosis. `go vet ./...` next, then `make test` — the Makefile, not a bare `go test ./...`, so that the test command lives in one place. No Postgres, no network and **no guard**. Every suite in this tree is hermetic: `gitx/fixture_test.go` builds a bare repository with the local `git` binary, and `web/web_test.go` is `httptest` plus `ecoretest`. Nothing here is gated on an environment variable, so nothing can skip itself into a green build, and the only thing this task owes is a non-zero exit when `make test` fails. That last point is load-bearing for the APKBUILD. `options="!check"` used to say the tests could not run in the VM — a claim that was never true — and now says they ran *before* `abuild` did. Which is a promise about **this** task: soften it and `!check` becomes a package built with nothing tested. ## build `REPODEST=$HOME/packages abuild -d` builds and stages the apk. `-d` disables abuild's dependency check: the makedepends are already installed by the manifest's `packages:` list, and abuild has no way to know that. `builddir="$startdir"` in the APKBUILD means abuild packages *this checkout in place* rather than unpacking a tarball. There is no tarball to unpack — the package is built from the commit under test, which has not been released anywhere yet. `build()` runs `make css` before the compile and `make check-css` after it, and `package()` runs `make install-files` rather than `make install`. Both are explained under [css](#css) and [install-files](#install-files). ## publish The `[ ! -r ~/.apk-ci.env ]` gate is the honest answer to a build that was handed no secrets, not a fallback: on a manual submission without the secret, every earlier task has still run and a signed apk is sitting in `$HOME/packages`, and saying so and exiting 0 is more useful than a red build about a credential nobody asked for. On a push the secret is there and this publishes. `set +x` before sourcing `~/.apk-ci.env` and `set -x` after. Every task runs under `set -x`, so without this the S3 access key and secret are echoed into a build log that is world-readable. The upload is `rclone copyto` per file — copy only, never delete, never sync. A `sync` would mirror local absence onto the bucket, and the bucket holds every previously published version of every service on the instance, which is what lets a pinned deployment be rebuilt. abuild nests its output under `$REPODEST///`, so each file is copied to a fixed prefix rather than the tree being mirrored. `apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it. ## publish_artifacts The same apk, uploaded a second time — into the `~bigbes/main` channel of `artifacts.sr.ht`, which indexes and signs it in the same request rather than on a 15-minute timer. One `PUT` per file to `/api/v1/pkg/~bigbes/main/apk/v3.22`, with the working token from `~/.srht-token` as the bearer. No client binary is involved: the upload route takes the file as the request body, and `curl` is already installed for the cacher. It is a task of its own and not two more lines inside `publish` for one reason: the S3 copy is what phoebe re-indexes today and this one is the road being opened. A refusal here has to be legible as *this* destination refusing, and it must not be able to undo an upload that already succeeded. `set +x` around every `curl`, for the same reason `publish` has it: the token is on the command line and the log is world-readable. Three answers are expected and only two of them are good: - `201` — published, indexed, signed. - `200` — the same version with the same bytes was already there. A rerun of a job that got as far as this task is idempotent, and stays green. - `409` — the same version with *different* bytes. `abuild` stamps mtimes into the archive, so resubmitting one commit produces a byte-different apk under an identical `pkgver`; a published version is immutable, so the daemon keeps what it has. That is a `WARNING` and not a failure — it means the build was run twice, not that anything is wrong. Republishing means removing the version through the API first. Anything else prints the daemon's error body and fails the task, after trying every remaining file: one bad package must not hide the fate of the others. The loop is `for file in $(find …)` rather than `find … | while read`, because under ash the second form runs its body in a subshell and the failure flag set in it is lost when the pipeline ends — the task would report success it never had. Gated on `~/.srht-token` being readable, exactly as `publish` is gated on `~/.apk-ci.env`: a manual submission without secrets still builds the apk, and says that nothing was published. Installing what it publishes, on any Alpine box: ```sh wget -qO /etc/apk/keys/bigbes@artifacts.srht.bigb.es.rsa.pub \ https://artifacts.srht.bigb.es/~bigbes/keys/apk.rsa.pub echo https://artifacts.srht.bigb.es/~bigbes/main/apk/v3.22 >> /etc/apk/repositories apk update && apk add diff.sr.ht ``` The name of the key file is not free: apk looks a signature up by the name carried in the index's `.SIGN.RSA256.` entry, so it has to be `bigbes@artifacts.srht.bigb.es.rsa.pub` and nothing else. With the key in place no `--allow-untrusted` is needed — the daemon signs the index it builds. ## cache_save **After `publish`**, deliberately: a cache upload that fails must not strand an apk that was built and tested successfully but never shipped. Fatal on purpose, though — an upload failure here means the next build pays for a cold cache, and that is worth knowing about rather than hiding behind `|| true`. There is no `cacher exists "$KEY" ||` guard in front of either upload. It was dead code: without `--force`, `cacher dir upload` already skips a key that is present, so the guard asked the same question twice and its only real effect was to make a failure of the `exists` call look like a decision not to upload. ## css The stylesheet is compiled from `scss/main.scss` against the shared partials and is embedded into the binary by `//go:embed static` (`web/templates.go`), so it has to exist **before** the compiler runs. Hence `make css` at the top of `build()` and a fatal `make check-css` after the compile. `check-css` counts the matches of `web/static/main.min.*.css` and fails on zero *and* on two. Two is as wrong as none and quieter: `web/templates.go` resolves `cssGlob` and takes the first match, so a second file makes the served stylesheet depend on readdir order. ### The committed stylesheet, and the two steps `web/static/main.min.79713f25.css` is **still tracked** in this revision. The end state is the one the siblings have — the file is build output, ignored, and produced by CI — but getting there is two steps and this is the first. Step one, this change: CI builds the partials, `make css` runs in `build()`, `check-css` gates the result, and the committed file stays in the index. The `.gitignore` patterns are already in place and do nothing to it, because a tracked file is unaffected by `.gitignore`. `build()` therefore prints two lines right after `make css`: ``` git status --porcelain -- web/static || true sha256sum web/static/main.min.*.css || true ``` Step two, after the first CI run, is decided by those two lines: - **Empty `git status` output** — `make css` reproduced the committed bytes exactly, including the content hash in the filename. Nothing about the shipped service changes when the file leaves the index. - **A `D` line for `main.min.79713f25.css` plus an untracked `main.min..css`** — the pipeline produces a different stylesheet than the one committed. That is the interesting case and it wants reading before it is dropped: a hash difference is either drift in `CORE_VER`/`BOOTSTRAP_REV` against whatever the committer had materialized locally, or a genuine change in `scss/main.scss` that was never recompiled. The `sha256sum` line names the new file so the comparison does not depend on scrolling the log. Either way step two is `git rm --cached web/static/main.min.79713f25.css` and nothing else — the ignore patterns are already there. Do it in the first case straight away; in the second, reconcile the versions first, because dropping the committed file makes CI the only producer and whatever it produces is what ships. Until step two lands, note that `make css` deletes a tracked file during `build()`, so the tree is dirty from that point on in the mismatch case. It costs nothing today — this service reads no VCS stamp of its own — but it is the reason step two should not sit around. ## install-files `package()` calls `make install-files`, not `make install`. `abuild` runs `package()` in a **fresh abuild process** under fakeroot, which re-sources the APKBUILD and never calls `build()`. So nothing `build()` exported reaches it — the `GOCACHE`/`GOMODCACHE` pins in particular. `build` in the Makefile is `.PHONY` (Go decides staleness itself; a real file target would never rebuild after a source edit), and `install` depends on `build`, so `make install` there recompiles the binary from a cold cache. The apk would then ship a second compilation that nothing in the pipeline had looked at. `install-files` is the copying half with no build in front of it: it stages the very bytes `build()` produced and `check-css` was asked about. `install -Dm755` on a missing `comparesrht` is a fatal error naming the file, which is the right report for the one way that target can be called too early. `install` itself keeps the sub-make (`@$(MAKE) install-files`) rather than listing it as a third prerequisite: `/usr/share/abuild/default.conf` exports `MAKEFLAGS=-j$(nproc)`, and under `-j` the prerequisites of one target run in parallel — the copying would start beside the build it is supposed to follow. ## What is not here - **No coverage upload.** The sibling services POST their profile to cover.sr.ht at the end of the pipeline. Adding it here needs a token secret and a repository on cover. - **No artifacts.** There is nothing to download: the apk goes to S3, and its name changes every commit, which `artifacts:` cannot express (it has no globbing). - **No matrix.** One architecture, one image. ## Known wart `make install-files` still runs `install -Dm644 -t $(DESTDIR)$(STATICDIR) web/static/*`, which stages `bundle..js` and `main.min..css` into `/usr/share/sourcehut/diff.sr.ht/static`. The binary already **embeds** both (`//go:embed static` in `web/templates.go:25`) and serves them from there, so those copies are dead weight in the apk and a second, divergable source of truth for the same bytes. The bench sibling installs no static assets at all for this reason. Left alone deliberately — it is not part of this change, and removing it wants a check that nothing in the deployment's nginx tree serves that directory directly.