~bigbes/ci-cacher

46be89f3 — Eugene Blikh 11 days ago v0.2.1
install.sh: verify checksums without coreutils extensions; bump to 0.2.1

sha256sum -c --ignore-missing is a coreutils extension. Alpine ships
busybox sha256sum, which prints its usage and exits non-zero, so the
bootstrap died with a 'checksum mismatch' that was nothing of the sort —
on the exact distro the sr-ht-cover manifest builds on. Pull the expected
hash out of checksums.txt with awk and compare strings instead; busybox,
coreutils and macOS shasum all agree on 'sha256sum FILE'.

Also stage the binary inside the target directory and rename it into
place, so an install over a running cacher can't hit ETXTBSY.

Verified against the published release on alpine:edge (busybox) and
ubuntu:noble (coreutils), and across all four published assets by faking
uname: Linux/x86_64, Linux/aarch64, Darwin/x86_64, Darwin/arm64 each
fetch the matching binary and verify.
8851cb58 — Eugene Blikh 11 days ago
test.yml: install via install.sh, collapse cache_gomod to one dir download

The module cache task was the last if/fi block in this manifest: download
a tarball, untar it, or else go mod download, tar it up, upload. With
v0.2.0's `dir download --exec` it is a single invocation that restores
the tree or builds it and seeds the cache — and the manifest that ships
with the tool now demonstrates the shape it is selling.

Key moves from gomod/<sha>.tar.gz (a tar.gz of ~/go containing pkg/mod)
to gomod/<sha>.tar.zst (cacher's own dir archive of ~/go/pkg/mod), so
the old objects are simply left behind rather than misread.

The bootstrap task loses its mkdir/curl/chmod trio to install.sh, which
additionally verifies the download against checksums.txt.
f26ddc0e — Eugene Blikh 11 days ago v0.2.0
Bump VERSION to 0.2.0 for tag
a574cffb — Eugene Blikh 11 days ago
README, landing page, CHANGELOG: document --exec, --optional, install.sh

Rewrites the directory-caching and docker sections around --exec instead
of the shell branch, replaces the bootstrap task in the full builds.sr.ht
example with the install.sh one-liner, and records that the --exec
script's exit status is propagated verbatim in the exit-code table.
a42721d6 — Eugene Blikh 11 days ago
Add install.sh one-line bootstrap, published next to the binaries

Every manifest repeated the same four lines to get the binary: mkdir,
curl, chmod, and a PATH export into ~/.buildenv. install.sh does all of
it, picks the asset for the host platform, and verifies it against the
published checksums.txt — which the hand-rolled version never did.

On builds.sr.ht ~/.buildenv is sourced before each task, so the PATH
export lands from the next task onwards; keep install and `cacher init`
in separate tasks.

The checksums.txt glob stays cacher-*: install.sh must not appear in the
manifest it verifies against.
aff7cc68 — Eugene Blikh 11 days ago
cmd: add --exec fallback and --optional to the download commands

--exec generalises the existing --url (file) and --pull (docker)
fallbacks to anything expressible as a shell command: on a cache miss
the script runs through sh -c and whatever it produced seeds the cache.
That collapses the restore-or-build if/fi block CI manifests repeat
around every cache into a single invocation.

The destination directory is created before the script runs, so the
script needs no mkdir -p of its own. The seed upload stays best-effort
(the content is already on disk), but a failing script is fatal and
propagates its own exit status rather than a generic 2. For docker the
tag is verified with docker image inspect afterwards — with a
best-effort upload, a script exiting 0 without building the image would
otherwise pass silently and fail a later task.

--optional turns a cache miss into exit 0 for download and dir
download, so set -e manifests drop the trailing || echo "cache miss".
bca3f572 — Eugene Blikh 2 months ago
README: refresh for v0.1.1+v0.1.2

* Lead-in mentions 'cacher docker download --pull' alongside the
  existing 'cacher download --url' as the canonical one-liners.
* Docker section now shows the --pull cache-or-pull form first; the
  exists/upload/download dance kept for locally-built images.
* New 'Full builds.sr.ht example' section — bootstrap-cacher,
  init, install_go, cache_gomod, cache_postgres, test. Mirrors the
  shape of this repo's .builds/test.yml.
* List subcommand: --recursive and --root flags documented.
* Install: name all four published platforms; link checksums.txt.
* Releases/changelog link to bigbes.pages.srht.bigb.es/ci-cacher.
* Drop the 'directory caching is brand new' line — shipped in v0.1.0.
9dc739e5 — Eugene Blikh 2 months ago v0.1.2
Bump VERSION to 0.1.2; widen landing page to 55rem
c22d27ee — Eugene Blikh 2 months ago
test.yml: collapse cache_garage_image to single 'cacher docker download --pull'

Now that v0.1.1 (with --pull) is the published binary on pages, the
if/else with exists/download/upload can fold into the one-line form.
SilenceUsage in v0.1.1 also removes the cobra-dump-on-miss noise that
required the stderr swallow on the previous `exists` check.
519ce411 — Eugene Blikh 2 months ago v0.1.1
Bump VERSION to 0.1.1 for tag
4c3e4701 — Eugene Blikh 2 months ago
test.yml: use published cacher binary instead of building from source

Walking back e1b71b2. Building cacher from source dropped the
go-tarball caching (chicken-and-egg). Bootstrapping from the published
v0.1.0 binary instead lets all three caches (go tarball, gomod, garage
image) use cacher.

Cost: cache_garage_image stays on the if/else cacher docker
{exists,download,upload} pattern instead of the single `cacher docker
download --pull` introduced on master, because v0.1.0 doesn't have
--pull yet. Comment notes the switch as a follow-up after the next
release.
e1b71b2f — Eugene Blikh 2 months ago
test.yml: build cacher from source, cache go.mod, use 'docker download --pull'

* install_cacher now goes via 'go build' from the cloned source
  instead of curl-from-pages. The whole point of test.yml is to
  exercise the binary this branch produces; bootstrapping from the
  previously-published release defeats that. Loses go-tarball caching
  as a side effect (chicken-and-egg: need go before cacher).

* cache_gomod task between cacher_init and test: tar ~/go/pkg/mod
  keyed by sha256(go.sum). Skips proxy.golang.org on subsequent runs;
  go.sum change invalidates automatically. Biggest cache win — the
  docker SDK transitives pulled in by testcontainers are heavy.

* cache_garage_image collapsed to a single 'cacher docker download
  --pull' call. Key prefix changed garage/ → docker/ (cache format
  is now zstd-compressed via cacher docker, incompatible with the
  raw-tar entry seeded by job 153).

* Dropped the redundant smoke 'build' task — install_cacher already
  exercises 'go build' from the same sources.
d0418115 — Eugene Blikh 2 months ago
cacher: add 'docker download --pull' and silence cobra usage on errors

* docker download --pull: on cache miss, falls back to docker pull
  + docker save + S3 seed, mirroring the --url fallback on file
  download. Collapses the if/else cache-or-pull bash dance in CI
  manifests to a single command.

* SilenceUsage on rootCmd: cache-miss / key-not-found exits aren't
  bad-usage errors, so the cobra Usage block dump on every error was
  noise. Short error line remains.
8a662e22 — Eugene Blikh 2 months ago
merge .builds/{unit,e2e}.yml into test.yml; cache go + garage image

Single job runs unit tests → smoke build → e2e (against a real Garage
container via testcontainers-go). Unit failure short-circuits before
paying the ~150 MB Garage image pull.

Caches via dogfooded cacher:
* Go tarball — `cacher download --url` fallback, same pattern as
  publish.yml.
* Garage docker image — `cacher download` of the tar; on miss does
  docker pull + save + upload to seed.

Also silences testcontainers' default logger in the garage testutil
(pull progress, container IDs, port mappings); test output now only
shows what the tests print themselves. `-v` dropped from the e2e
`go test` since per-test progress is no longer drowned out and not
worth printing on green runs.

`TESTCONTAINERS_RYUK_DISABLED=true` — VM is torn down at job end,
reaper container is just startup noise.

Verified end-to-end on job #153 (all 8 tasks ✔, cache MISS seeded).
b659c698 — Eugene Blikh 2 months ago
publish.yml: restore artifacts block now that storage works

Job 148 reached the upload stage where 30347e5 didn't — admin has
configured the artifact storage backend, so the four cacher binaries
+ checksums.txt are useful again as a 90-day-TTL view on the job page
alongside the permanent pages.sr.ht copy.
30347e53 — Eugene Blikh 2 months ago
publish.yml: dogfood cacher for deps, drop failing artifacts block

Two changes:

* Drop `artifacts:` block. The srht.bigb.es self-hosted instance
  doesn't have artifact storage configured (the upload step fails
  silently right after "Running task stage_artifacts"). Pages.sr.ht
  already serves the binaries with stable URLs, so artifacts were
  just redundant. Removes the stage_artifacts task too.

* Cache the Go and goreleaser tarballs through cacher itself.
  Bootstrap: curl the previously-published cacher binary from pages;
  cacher init against s3.bigb.es using the existing srht-registered
  secrets (same UUIDs as tarantool-protobuf .builds); then cacher
  download "goreleaser/<ver>.tar.gz" --url "https://github…" handles
  cache-or-fetch + back-fill. First run after this commit pays the
  full curl cost and seeds s3; every later run is an S3 hit.

  hut isn't cached — no upstream binary release, building from
  source is small + fast.
202baab1 — Eugene Blikh 2 months ago
publish.yml: override hut pages origin to public URL

The srht.bigb.es OAuth worker writes the internal docker-compose
hostname (http://pages:5112) into ~/.config/hut/config, which is
unreachable from outside the cluster. sed the config to use the
public origin (https://pages.srht.bigb.es) before publish.

This is a srht admin-side gotcha; only `builds` and `meta` get
public origins in the worker config, the other services
(pages/git/hub/todo/paste) all keep internal hostnames.
d44d75de — Eugene Blikh 2 months ago
publish.yml: goreleaser --skip=validate

When a CI manifest patch lands after tagging, the version tag won't
point at HEAD. --skip=validate lets goreleaser proceed with the
already-tagged version anyway. Safe because the binary built from
HEAD vs the tag is byte-identical (only publish.yml differs).
462e6166 — Eugene Blikh 2 months ago
publish.yml: mkdir ~/.local/bin before extracting goreleaser

Caught by job 143 install_goreleaser step — install_go creates
~/.local but not ~/.local/bin, so the tar -C target didn't exist.
1c1f3858 — Eugene Blikh 2 months ago
publish.yml: hut version (not --version)

Caught by job 141 install_hut step. `hut` doesn't expose a --version
flag; the subcommand is `hut version`. `go install` had succeeded
silently and the failure was just the version probe.
Next