# CI: what `.build.yml` does and why

The manifest is deliberately short on prose. builds.sr.ht stores a submitted
manifest in a `varchar(16384)`, so a manifest that grows past 16 KiB cannot be
submitted at all — the failure is at submission time and reads like nothing in
particular: the branch simply has no CI. Rationale therefore lives here, and the
manifest carries pointers.

The pipeline is one linear job on `alpine/edge`: install the cache helper,
assemble the shared SCSS, make a throwaway signing key, decide the version,
restore the Go caches, test, package with `abuild`, publish the apk to
`repo.bigb.es/alpine/v3.22/bigbes`, save the caches.

It is triggered by a push to the **sourcehut** side. A push to sourcecraft
cannot reach builds.sr.ht; the gitsync mirror is what puts the commit on
`git.srht.bigb.es`, and that push is what submits the job. So the mirror is on
the critical path for the package repository, not merely an offsite copy.

## packages

`sassc` and `minify` are for `make css`, not for the package: the stylesheet is
compiled here and embedded into the binary, and the apk has no runtime
dependency on either.

`curl` is for the `cacher_install` bootstrap and `rclone` for `publish`. There is
no `nodejs`/`npm`: `web/static/bundle.<hash>.js` is committed, and `make bundle`
is an upgrade step run by hand rather than a build step.

There is no `postgresql`. This service is stateless — it holds no database at
all — and its suites are hermetic: `gitx/fixture_test.go` builds a bare
repository with the local `git`, and `web/web_test.go` is `httptest` plus
`ecoretest`. Nothing in the tree talks to a live git.sr.ht API.

## secrets

Three, all account-level and shared with the sibling services:

| secret | lands at | used by |
|---|---|---|
| `apk-ci-s3` | `~/.apk-ci.env` | `publish` |
| `7dde4219-…` | `~/.s3-cache-key-id` | `cacher_init` |
| `0e5b3530-…` | `~/.s3-cache-key-secret` | `cacher_init` |

They are file secrets. Listing them is what turns `publish` and the cache tasks
on; a manual submission that asks for no secrets still runs the interesting part
of the pipeline and stops at `cacher_init`, which is the right place to notice.

`apk-ci-s3` is referenced by name and the other two by UUID, which is only
because that is how they were written in the donor manifests; both forms work.

## environment

`CORE_VER` **must track the deployment's `SRHT_CORE_VER`**, and `BOOTSTRAP_REV`
is the Bootstrap submodule commit `core.sr.ht` pins at that tag. The two being
out of step means this service renders against different partials than the rest
of the instance, which shows up as a page that is subtly the wrong shape and as
nothing at all in any log. They are `0.84.5` / `779ad9f1` here because that is
what `srht/versions.env` pins in the deployment repo; bump all of them together.

`REPO` is the clone directory `sources:` produces, and every task that touches
the checkout `cd`s into it — task bodies do not inherit a working directory from
each other.

## cacher

`cacher` (`go.bigb.es/cacher`) is an S3-backed cache helper, installed from its
own release on pages.sr.ht.

It is **two tasks and not one** because `install.sh` appends its `PATH` export
to `~/.buildenv`, and `~/.buildenv` is sourced by the *next* task's preamble — a
single task would install the binary and then not find it on `PATH`.

The installer is fetched with `curl -fsSL` and verifies what it downloads
against `checksums.txt`. The predecessor of this task fetched the raw binary
with a bare `curl -sSL`; without `-f`, an HTTP error is not an error to `curl`,
so a 404 page was written to `~/.local/bin/cacher` and `chmod +x`-ed, and the
first failure was a syntax error out of a shell reading HTML.

## scss

`sassc` needs the shared sourcehut partials, and **no apk ships them**: they are
core.sr.ht's own `scss/` plus the Bootstrap 4 tree it pins as a submodule. The
task clones both and drops them in `/usr/share/sourcehut/scss`, which is the
`ASSETS` the Makefile and the APKBUILD name.

The whole thing is cached under a key made of the two pins
(`scss/$CORE_VER-$BOOTSTRAP_REV.tar.zst`), so an outage at git.sr.ht or GitHub
cannot fail a build that changed nothing. `--exec` runs only on a cache miss and
seeds the cache afterwards; the block sees exported variables only, which is why
the cache key is spelled inline in the `cacher` invocation rather than computed
inside the block, and why the block is in single quotes.

It runs before `keygen` and before anything Go, because the stylesheet has to
exist before the compiler does — see [css](#css).

## keygen

`abuild-keygen -a -n -i -q` makes a throwaway package signing key.

`-i` is not optional: without it the key is generated but not installed into
`/etc/apk/keys`, and `abuild`'s own final "update the local repository index"
step then dies with an UNTRUSTED signature after having built the package
perfectly well.

The key is per-build and dies with the VM. That is fine because nothing verifies
these signatures — the apk index on `repo.bigb.es` is rebuilt and signed on
phoebe by the garage stack's `apk-mirror` service, which indexes this repo with
`--allow-untrusted` precisely because of this.

## version

One `git describe` decides the apk's `pkgver`. Alpine's version grammar does not
accept `v1.2.3-4-gabc1234`, so a describe with commits since the tag is
rewritten to `1.2.3_git4`; `_git` sorts **after** the release in Alpine's
comparison, which is what makes an untagged master build look newer than the tag
it follows. With no tags at all — which is this repository today — it falls back
to `0.0.<commit count>`, monotonic and unique per commit, which a deployment can
pin.

Tags are in `allow-refs` for exactly this reason: pushing `v0.9.0` is what
produces the `0.9.0` apk.

`$ver` is **exported** through `~/.buildenv` as `PKGVER`, and the APKBUILD reads
`pkgver="${PKGVER:-0.0.0}"`. It is not `sed`-ed into the tracked APKBUILD, and
this is the one thing in this file that must not be "tidied" back:

> Go records `vcs.modified` in every binary it builds inside a repository, and
> it reads that flag from `git status --porcelain`. Measured on go1.26.5, a
> `sed -i` of a tracked file sets `vcs.modified=true` and Go stamps `+dirty`
> into `Main.Version` for the whole `abuild` run — for every binary in the
> package, permanently, for the life of the apk.

The export alone is not enough. Any **untracked** file present at `go build` time
does the same thing, and `abuild` works inside the checkout with
`builddir="$startdir"` and exports `GOTMPDIR=$startdir/tmp` — so `./src`,
`./pkg` and `./tmp` all appear inside the tree it is packaging. `.gitignore`
covers those three, and Go does not count ignored files.

The task ends with a bare `git status --porcelain` because this is the last
moment the tree is provably clean, and the output of a clean one is nothing.

## cache_restore

The Go module cache and build cache are keyed by the hash of `go.sum`, so a
build that changed no dependency reuses both.

`--optional` and not `|| true`. They are not the same promise: `--optional`
returns 0 only for a genuine cache miss, while `|| true` swallows every other
outcome too — bad credentials, a dead bucket, a typo in the prefix — and turns a
misconfigured cache into a pipeline that is merely slow, silently, forever.

The exports here are only for the repair block below and for `cache_save`:
`abuild` redirects both caches into its throwaway `$tmpdir` (and an upstream typo
in `abuild.in` slaves `GOMODCACHE` to `GOCACHE`), so environment exports from a
task cannot stick. The APKBUILD's `build()` re-pins both to the same home
locations, after abuild's own exports, which is what makes the tarballs saved at
the end be the ones the compile used.

### The half-restored module cache

The repair block is not defensive padding. A partially restored module cache is
the *normal* failure mode of this arrangement — an interrupted upload, a
truncated object, a key written while a build was still running — and what it
produces is a compile error deep in a dependency, which reads exactly like a bug
in the code under test.

So: make the tree writable (the module cache is mode 555 and `rm -rf` cannot
remove it otherwise), ask `go mod verify` whether what came back is intact, and
throw the whole thing away if it is not. Then `go mod download` and verify for
real, and let *that* failure be fatal.

Do not soften the final `go mod verify` to `|| true`. A build that proceeds with
a module cache it could not verify is a build whose result means nothing.

### Not `go mod download all`

The warm-up is `go mod download`. **The `all` must not come back**, and it is
worth knowing why, because `all` is exactly the shape a future reader adds as an
obvious improvement — warm *everything*, surely, so nothing is fetched later.

`go mod download all` resolves the entire module graph, test dependencies of
dependencies included, and **appends the hashes of everything it resolved to
`go.sum`**. Measured on this tree: 145 lines added. In the bench sibling, 228 —
and that is not a hypothetical, it is what failed its build #359, which stopped
at `check-version` with a `-dirty` binary and `M go.sum` named as the thing in
the way.

A modified tracked file in the checkout when `go build` runs is precisely the
stamp this whole pipeline is arranged to avoid — the same failure the `version`
task refuses to cause by not `sed`-ing the APKBUILD. Warming the cache with a
command that edits the tree would have reintroduced it one task later.

Without `all`, `go.sum` is untouched, `go mod verify` still passes afterwards,
and what gets fetched is what the main module actually builds. That is all a
cache warm-up needs; the packages `all` would have added are ones no build in
this pipeline compiles.

Verified in a fresh clone of this branch — `go mod download` followed by
`git status --porcelain` leaves the tree clean. compare has the smallest
dependency graph of the six services, so this is where `all` came closest to
being harmless, and it still rewrote 145 lines.

### `-mod=readonly` does not save you

There used to be a sentence here, and another above the `git status` line in the
manifest, saying `-mod=readonly` is the default so neither command *can* rewrite
`go.mod` or `go.sum`. **It was false**, and it is a large part of why nobody
looked at `go mod download all` for as long as they did: it sat directly above
the line that disproved it and invited the reader to skip both.

`readonly` governs updates to the module **requirements**. It does not stop
writes to `go.sum`. Measured on this tree rather than inherited, with the flag
set explicitly on the command line:

| command | `go.sum` | exit |
|---|---|---|
| `GOFLAGS=-mod=readonly go mod download all` | **+145 lines** | 0 |
| `GOFLAGS=-mod=readonly go mod download` | untouched | 0 |

Both still pass `go mod verify` afterwards, so verification is no guard either.

Set the flag **on the command line** when reproducing this. `go env -w` writes a
persistent `GOFLAGS`, and a machine carrying `go env -w GOFLAGS=-mod=mod` — this
one does, and bench's did too — makes a run that omits the flag prove nothing
about what the builder would do. Check with `go env GOFLAGS` first.

So the `git status --porcelain` at the end of the task is not belt-and-braces,
and no flag stands behind it: **it is the check**. It is the only thing in this
pipeline that would notice a `go.sum` rewrite before the compile, and on the
bench sibling it is what did. Do not delete it as redundant and do not replace
it with an appeal to a flag.

## test

`gofmt -l .` piped to a file and then `test ! -s`, rather than a bare `gofmt -l`:
`gofmt -l` exits 0 whether or not it printed anything, so the only way to fail on
its output is to look at the output. The `tee` keeps the offending filenames in
the build log, where they are the entire diagnosis.

`go vet ./...` next, then `make test` — the Makefile, not a bare `go test ./...`,
so that the test command lives in one place.

No Postgres, no network and **no guard**. Every suite in this tree is hermetic:
`gitx/fixture_test.go` builds a bare repository with the local `git` binary, and
`web/web_test.go` is `httptest` plus `ecoretest`. Nothing here is gated on an
environment variable, so nothing can skip itself into a green build, and the
only thing this task owes is a non-zero exit when `make test` fails.

That last point is load-bearing for the APKBUILD. `options="!check"` used to say
the tests could not run in the VM — a claim that was never true — and now says
they ran *before* `abuild` did. Which is a promise about **this** task: soften it
and `!check` becomes a package built with nothing tested.

## build

`REPODEST=$HOME/packages abuild -d` builds and stages the apk.

`-d` disables abuild's dependency check: the makedepends are already installed
by the manifest's `packages:` list, and abuild has no way to know that.

`builddir="$startdir"` in the APKBUILD means abuild packages *this checkout in
place* rather than unpacking a tarball. There is no tarball to unpack — the
package is built from the commit under test, which has not been released
anywhere yet.

`build()` runs `make css` before the compile and `make check-css` after it, and
`package()` runs `make install-files` rather than `make install`. Both are
explained under [css](#css) and [install-files](#install-files).

## publish

The `[ ! -r ~/.apk-ci.env ]` gate is the honest answer to a build that was handed
no secrets, not a fallback: on a manual submission without the secret, every
earlier task has still run and a signed apk is sitting in `$HOME/packages`, and
saying so and exiting 0 is more useful than a red build about a credential
nobody asked for. On a push the secret is there and this publishes.

`set +x` before sourcing `~/.apk-ci.env` and `set -x` after. Every task runs
under `set -x`, so without this the S3 access key and secret are echoed into a
build log that is world-readable.

The upload is `rclone copyto` per file — copy only, never delete, never sync. A
`sync` would mirror local absence onto the bucket, and the bucket holds every
previously published version of every service on the instance, which is what
lets a pinned deployment be rebuilt. abuild nests its output under
`$REPODEST/<repo>/<arch>/`, so each file is copied to a fixed prefix rather than
the tree being mirrored.

`apk-mirror` on phoebe re-indexes within 15 minutes; nothing here waits for it.

## cache_save

**After `publish`**, deliberately: a cache upload that fails must not strand an
apk that was built and tested successfully but never shipped.

Fatal on purpose, though — an upload failure here means the next build pays for
a cold cache, and that is worth knowing about rather than hiding behind
`|| true`.

There is no `cacher exists "$KEY" ||` guard in front of either upload. It was
dead code: without `--force`, `cacher dir upload` already skips a key that is
present, so the guard asked the same question twice and its only real effect was
to make a failure of the `exists` call look like a decision not to upload.

## css

The stylesheet is compiled from `scss/main.scss` against the shared partials and
is embedded into the binary by `//go:embed static` (`web/templates.go`), so it
has to exist **before** the compiler runs. Hence `make css` at the top of
`build()` and a fatal `make check-css` after the compile.

`check-css` counts the matches of `web/static/main.min.*.css` and fails on zero
*and* on two. Two is as wrong as none and quieter: `web/templates.go` resolves
`cssGlob` and takes the first match, so a second file makes the served
stylesheet depend on readdir order.

### The committed stylesheet, and the two steps

`web/static/main.min.79713f25.css` is **still tracked** in this revision. The end
state is the one the siblings have — the file is build output, ignored, and
produced by CI — but getting there is two steps and this is the first.

Step one, this change: CI builds the partials, `make css` runs in `build()`,
`check-css` gates the result, and the committed file stays in the index. The
`.gitignore` patterns are already in place and do nothing to it, because a
tracked file is unaffected by `.gitignore`.

`build()` therefore prints two lines right after `make css`:

```
git status --porcelain -- web/static || true
sha256sum web/static/main.min.*.css || true
```

Step two, after the first CI run, is decided by those two lines:

- **Empty `git status` output** — `make css` reproduced the committed bytes
  exactly, including the content hash in the filename. Nothing about the shipped
  service changes when the file leaves the index.
- **A `D` line for `main.min.79713f25.css` plus an untracked
  `main.min.<other>.css`** — the pipeline produces a different stylesheet than
  the one committed. That is the interesting case and it wants reading before it
  is dropped: a hash difference is either drift in `CORE_VER`/`BOOTSTRAP_REV`
  against whatever the committer had materialized locally, or a genuine change in
  `scss/main.scss` that was never recompiled. The `sha256sum` line names the new
  file so the comparison does not depend on scrolling the log.

Either way step two is `git rm --cached web/static/main.min.79713f25.css` and
nothing else — the ignore patterns are already there. Do it in the first case
straight away; in the second, reconcile the versions first, because dropping the
committed file makes CI the only producer and whatever it produces is what
ships.

Until step two lands, note that `make css` deletes a tracked file during
`build()`, so the tree is dirty from that point on in the mismatch case. It
costs nothing today — this service reads no VCS stamp of its own — but it is the
reason step two should not sit around.

## install-files

`package()` calls `make install-files`, not `make install`.

`abuild` runs `package()` in a **fresh abuild process** under fakeroot, which
re-sources the APKBUILD and never calls `build()`. So nothing `build()` exported
reaches it — the `GOCACHE`/`GOMODCACHE` pins in particular. `build` in the
Makefile is `.PHONY` (Go decides staleness itself; a real file target would never
rebuild after a source edit), and `install` depends on `build`, so `make install`
there recompiles the binary from a cold cache. The apk would then ship a second
compilation that nothing in the pipeline had looked at.

`install-files` is the copying half with no build in front of it: it stages the
very bytes `build()` produced and `check-css` was asked about. `install -Dm755`
on a missing `comparesrht` is a fatal error naming the file, which is the right
report for the one way that target can be called too early.

`install` itself keeps the sub-make (`@$(MAKE) install-files`) rather than
listing it as a third prerequisite: `/usr/share/abuild/default.conf` exports
`MAKEFLAGS=-j$(nproc)`, and under `-j` the prerequisites of one target run in
parallel — the copying would start beside the build it is supposed to follow.

## What is not here

- **No coverage upload.** The sibling services POST their profile to cover.sr.ht
  at the end of the pipeline. Adding it here needs a token secret and a
  repository on cover.
- **No artifacts.** There is nothing to download: the apk goes to S3, and its
  name changes every commit, which `artifacts:` cannot express (it has no
  globbing).
- **No matrix.** One architecture, one image.

## Known wart

`make install-files` still runs
`install -Dm644 -t $(DESTDIR)$(STATICDIR) web/static/*`, which stages
`bundle.<hash>.js` and `main.min.<hash>.css` into
`/usr/share/sourcehut/diff.sr.ht/static`. The binary already **embeds** both
(`//go:embed static` in `web/templates.go:25`) and serves them from there, so
those copies are dead weight in the apk and a second, divergable source of truth
for the same bytes. The bench sibling installs no static assets at all for this
reason. Left alone deliberately — it is not part of this change, and removing it
wants a check that nothing in the deployment's nginx tree serves that directory
directly.
