# builds.sr.ht manifest for sr-ht-ecore. One linear pipeline: restore the Go # caches, refuse an unformatted or vet-dirty tree, test with a coverage profile, # run the benchmarks, then upload both to the instance's own cov.sr.ht and # bench.sr.ht. # # This repository is a LIBRARY — no cmd/, no daemon, no stylesheet, no apk — and # seven services pin it by pseudo-version, which is exactly why it needs a # pipeline of its own: a break here is discovered in whichever service next runs # `go get -u`, at the moment that service is being released. Everything a # sibling manifest carries for its package (scss, keygen, abuild, publish, # publish_artifacts) has no meaning here and is absent rather than stubbed, and # there is no postgres task because nothing in this module imports database/sql: # no test opens a database, no test skips on a missing DSN. # # The reasoning lives in comments here rather than in a docs/ci.md, because this # repository has no docs/ tree. Keep it under 16 KiB: builds.sr.ht stores the # submitted manifest in a varchar(16384), and a manifest over that cannot be # submitted at all — the failure is a branch with no CI, not a red build. image: alpine/edge packages: - go - git # Every task below drives the Makefile. The sibling manifests never list this # and their `make test` works, so the image evidently ships one — but abuild's # 14 dependencies do not include it (checked on pkgs.alpinelinux.org), so what # they rely on is the image's furniture rather than a package they asked for. # A build that needs make asks for make. - make # For the cacher's install.sh and for the two uploads, and for nothing else. - curl secrets: # S3 credentials for the cacher CI cache (Garage `docker-cache` bucket). # Account-level secrets shared with the sibling services, so this repository # holds nothing of its own. Absent, cacher_install turns the cache off and the # build compiles cold instead of failing. - 7dde4219-0783-4581-a67d-c94749de3600 # ~/.s3-cache-key-id - 0e5b3530-6f19-4f30-9b73-9339dd382e46 # ~/.s3-cache-key-secret # ~/.srht-token holds a tokens.sr.ht WORKING TOKEN, and it is one secret shared # with the cov.sr.ht and bench.sr.ht pipelines rather than a per-repository # one: the credential is minted once, for a person, and carries the grants of # every service it is meant to reach. This build needs cov:upload and # bench:upload from it; a token missing one fails that upload and no other. # The grant was cover:upload before the service was renamed to cov.sr.ht, and # grants are compared literally, so a token minted before the rename uploads # no coverage here — re-mint it on tokens.sr.ht and replace this secret. - c7968415-1a6d-4ca0-a188-150fb7f57b65 # ~/.srht-token sources: - https://git.srht.bigb.es/~bigbes/sr-ht-ecore environment: REPO: sr-ht-ecore COVER_ORIGIN: https://cov.srht.bigb.es COVER_REPO: "~bigbes/sr-ht-ecore" BENCH_ORIGIN: https://bench.srht.bigb.es BENCH_REPO: "~bigbes/sr-ht-ecore" # Literal paths relative to $HOME — which is why the two tasks below write there # and not into the checkout. They are not a fallback for the uploads: a build # submitted without secrets still leaves both files downloadable, and a POST # that failed leaves the body that was meant to be sent. artifacts: - cover.out - bench.txt submitter: git.sr.ht: allow-refs: - refs/heads/master # A library is consumed by tag as well as by pseudo-version, so a pushed # tag has to be tested too. Without this line it starts no build at all. - "refs/tags/v*" tasks: # S3-backed CI cache helper. install.sh's PATH export goes to ~/.buildenv, # which only the NEXT task sources — hence two tasks and not one. # # The guard is not in the donors' copies of this task, and it is here because # this repository is a library: a contributor's manually submitted build asks # for no secrets, and cacher init would then die on an unreadable key file # before a single test had run. Without the cache the build is slower and # exactly as truthful. - cacher_install: | if [ ! -r ~/.s3-cache-key-id ] || [ ! -r ~/.s3-cache-key-secret ]; then echo "no S3 cache credentials: this build compiles from cold" echo "export NO_CACHE=1" >> ~/.buildenv exit 0 fi curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh - cacher_init: | [ -z "$NO_CACHE" ] || { echo "cache disabled: nothing to init"; exit 0; } cacher init \ --endpoint https://s3.bigb.es \ --region garage \ --bucket docker-cache \ --prefix sr-ht-ecore/deps \ --key-file ~/.s3-cache-key-id \ --secret-file ~/.s3-cache-key-secret - cache_restore: | cd "$REPO" if [ -z "$NO_CACHE" ]; then # Module and build caches keyed by go.sum; --optional makes a miss a # cold build, not an error. KEY_MOD=$(cacher key "gomod/{hash}.tar.zst" --hash-from go.sum) KEY_GOC=$(cacher key "gocache/{hash}.tar.zst" --hash-from go.sum) echo "export KEY_MOD=$KEY_MOD KEY_GOC=$KEY_GOC" >> ~/.buildenv cacher dir download "$KEY_MOD" ~/go/pkg/mod --optional cacher dir download "$KEY_GOC" ~/.cache/go-build --optional # Repair block for the HALF-restored module cache: it is the normal # failure of a restore, not a freak one, and it surfaces later as # compile errors inside a dependency that read like a code bug # (cover.sr.ht's build #284). Do not soften this to `|| true`. chmod -R u+w ~/go/pkg/mod 2>/dev/null || true if ! go mod verify >/dev/null 2>&1; then echo "restored module cache did not verify — discarding it" rm -rf ~/go/pkg/mod fi fi # `go mod download`, NEVER `go mod download all`. The `all` pattern # resolves the whole module graph including dependencies' test-only # requirements and APPENDS their hashes to the TRACKED go.sum — silently, # exit 0 — which leaves a modified file in the checkout. -mod=readonly # does NOT prevent it: readonly governs the module requirements, not # writes to go.sum. Without `all`, go.sum is untouched and what the cache # holds is what this module builds and tests anyway. go mod download go mod verify # And the proof that it did not: a print here would be a flag we trust, # so the clean tree is asserted instead. The `all` spelling fails this # line with a go.sum of a few hundred added hashes. test -z "$(git status --porcelain)" || { git status --porcelain; exit 1; } - lint: | cd "$REPO" # check-fmt and not fmt: `gofmt -l` prints the offending files and STILL # EXITS 0, so a task that ran plain gofmt could not fail, and `make fmt` # rewrites — a gate that edits the tree it is judging is not a gate. make check-fmt make vet - test: | cd "$REPO" # -covermode=atomic (real hit counts, which is what cov.sr.ht reads) and # $HOME, because that is where artifacts: looks. The Makefile owns the # command; this task owns the destination. `make cover` also refuses an # empty profile, which would otherwise upload a report covering nothing # and call it a success. make cover COVERPROFILE="$HOME/cover.out" - bench: | cd "$REPO" # -s so make does not echo the recipe into the body; a redirect and a cat # and NOT `| tee`, which would hand this task tee's exit status and let a # failing benchmark pass. # # What lands in the file is benchfmt and nothing else because the Makefile # filters it (see BENCH_FILTER there): a benchmark that provokes logging # in the code under test writes those lines into this same stream, and one # of the benchmarks in this tree currently produces hundreds of megabytes # of them. The command that ran is echoed to stderr, so it is in this # task's log and not in the body being uploaded. make -s bench > "$HOME/bench.txt" cat "$HOME/bench.txt" # `go test -bench` that matches nothing prints `ok` and exits 0, and a # file with no benchmark lines is still valid benchfmt — so a renamed or # deleted benchmark would upload an empty run and report success. One # name per benchmark file, so that losing any one file is caught: grep -q '^BenchmarkValidate' "$HOME/bench.txt" # bearer grep -q '^BenchmarkRequestLogger' "$HOME/bench.txt" # chimw grep -q '^BenchmarkRequire' "$HOME/bench.txt" # csrf grep -q '^BenchmarkParse' "$HOME/bench.txt" # grants grep -q '^BenchmarkChain' "$HOME/bench.txt" # middleware - cache_save: | [ -z "$NO_CACHE" ] || { echo "cache disabled: nothing to save"; exit 0; } # After the runs that warm it, and fatal on purpose. Without --force an # upload skips a key that is already there, so no `cacher exists` guard is # needed. cacher dir upload "$KEY_MOD" ~/go/pkg/mod cacher dir upload "$KEY_GOC" ~/.cache/go-build - coverage: | cd "$REPO" # The gate is the honest answer to a build that was handed no secrets: the # profile is made, it is this build's cover.out artifact, and it can be # POSTed by hand. With the file present the upload is fatal on purpose. if [ ! -r ~/.srht-token ]; then echo "no ~/.srht-token: this build has no cov.sr.ht credentials" echo "the profile is still available as this build's cover.out artifact" exit 0 fi # GIT_REF is absent on a manually submitted build and ref is optional for # the API; key is the idempotency key, so a resubmitted job replaces its # own report instead of adding a second one. Both prefixes are stripped # because this pipeline builds tags too, and a tag build would otherwise # report ref=refs/tags/v0.1.0. ref="${GIT_REF#refs/heads/}" ref="${ref#refs/tags/}" url="$COVER_ORIGIN/api/v1/repos/$COVER_REPO/reports" url="$url?commit=$(git rev-parse HEAD)&ref=$ref&key=$JOB_ID&job_url=$JOB_URL" echo "uploading cover.out to $url" # Tracing off to the end of the task: the Authorization header must not # reach the log. No Content-Type — the service sniffs the format, and a # wrong one is a 400. --fail-with-body prints the JSON error AND still # exits non-zero, which plain --fail does not. set +x curl -sS --fail-with-body -X POST \ -H "Authorization: Bearer $(cat ~/.srht-token)" \ --data-binary "@$HOME/cover.out" \ "$url" echo - bench_upload: | cd "$REPO" if [ ! -r ~/.srht-token ]; then echo "no ~/.srht-token: this build has no bench.sr.ht credentials" echo "the benchmarks ran and are in the bench task's log" echo "the file is this build's bench.txt artifact, and can be POSTed by hand" exit 0 fi # visibility acts only on the POST that creates $BENCH_REPO; on every # later run it is ignored. ref="${GIT_REF#refs/heads/}" ref="${ref#refs/tags/}" url="$BENCH_ORIGIN/api/v1/repos/$BENCH_REPO/runs" url="$url?commit=$(git rev-parse HEAD)&ref=$ref&key=$JOB_ID&job_url=$JOB_URL" url="$url&visibility=public" echo "uploading bench.txt to $url" set +x curl -sS --fail-with-body -X POST \ -H "Authorization: Bearer $(cat ~/.srht-token)" \ --data-binary "@$HOME/bench.txt" \ "$url" echo