~bigbes/ci-cacher

ci-cacher/README.md -rw-r--r-- 9.7 KiB
bca3f572 — Eugene Blikh README: refresh for v0.1.1+v0.1.2 a day ago

#cacher

S3-backed CI cache helper. A single static Go binary that downloads, uploads, lists, and invalidates cached build artifacts in any S3-compatible bucket. Built for builds.sr.ht but works anywhere you can run a binary and reach an S3 endpoint.

Replaces the typical CI cache shell loop:

# before — install awscli, write ~/.aws/config, then in every task:
if aws s3api head-object --bucket "$B" --key "$K" >/dev/null 2>&1; then
  aws s3 cp "s3://$B/$K" "$out"
else
  curl -sSL "$url" -o "$out"
  aws s3 cp "$out" "s3://$B/$K"
fi

# after — one binary, one config, one command:
cacher download "$key" "$out" --url "$url"

Same collapse for docker images:

cacher docker download "$key" "$image:tag" --pull

Releases and changelog: bigbes.pages.srht.bigb.es/ci-cacher.

#Install

# Pre-built binary (latest tag). Substitute your platform:
#   cacher-linux-amd64, cacher-linux-arm64,
#   cacher-darwin-amd64, cacher-darwin-arm64
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
chmod +x ~/.local/bin/cacher

# Pin to a known sha256 (checksums.txt lives next to the binaries):
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
echo "<expected-sha256>  ~/.local/bin/cacher" | sha256sum -c

# From source:
go install go.bigb.es/cacher@latest

Verify with cacher version.

#Setup

cacher init writes ~/.config/cacher/config.toml and runs a smoke test to fail fast on bad credentials. Run it once per CI job after secrets are mounted:

cacher init \
  --endpoint    https://s3.example.com \
  --region      us-east-1 \
  --bucket      ci-cache \
  --prefix      my-project \
  --key-file    ~/.s3-cache-key-id \
  --secret-file ~/.s3-cache-key-secret

Credentials resolve as CACHER_S3_KEY_ID / CACHER_S3_SECRET env vars > --key-file / --secret-file > files recorded in config. Use the env vars for local-dev runs; use the files in CI where secrets are mounted.

cacher doctor repeats the smoke test (HEAD bucket + 1-byte write/read/delete canary) and prints credential-length diagnostics without leaking the secret.

#Usage

#Single file — fetch-or-download

cacher download go-1.26.3.tar.gz /tmp/go.tar.gz \
  --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz \
  --sha256 abc123…

Cache HIT: pulls from S3. Cache MISS: GETs the URL, verifies the optional sha256, writes to the destination, and uploads to S3 so the next run hits.

cacher upload my-key /path/to/artifact            # skip if present
cacher upload my-key /path/to/artifact --force    # overwrite
cacher exists my-key                              # exit 0 hit, 1 miss
cacher list   my-prefix                           # /-delimited (aws s3 ls)
cacher list   my-prefix --recursive               # flat
cacher list   --root                              # ignore configured prefix
cacher delete my-key                              # invalidate

#Docker images — streamed save/load

For an image pulled from a registry, the cache-or-pull pattern is a single command:

# Cache HIT → docker load from S3. MISS → docker pull + seed S3 + tag stays local.
cacher docker download "docker/dxflrs-garage-v2.3.0.tar.zst" \
  dxflrs/garage:v2.3.0 --pull

For images you build locally, drive the cache by hand — same primitives, key by Dockerfile content:

KEY=$(cacher key "images/{hash}.tar.zst" --hash-from Dockerfile)

if ! cacher docker exists "$KEY"; then
  docker build -t myimage:latest .
  cacher docker upload "$KEY" myimage:latest
else
  cacher docker download "$KEY" myimage:latest
fi

The save/load pipeline is fully streamed — docker save | zstd | s3 and the inverse, no on-disk tempfile. The zstd codec is pure Go (klauspost/compress), so no external zstd binary is needed.

#Directory caching — the real CI speedup

Cache resolved trees keyed by a lockfile hash. Skip resolution entirely when nothing changed:

# Go module cache, keyed by go.sum:
KEY=$(cacher key "go-mod/{hash}.tar.zst" --hash-from go.sum)
cacher dir download "$KEY" ~/go/pkg/mod 2>/dev/null || {
  go mod download
  cacher dir upload "$KEY" ~/go/pkg/mod
}

# Lua rocks tree, keyed by rockspec:
KEY=$(cacher key "rocks/{hash}.tar.zst" --hash-from project-scm-1.rockspec)
cacher dir download "$KEY" .rocks 2>/dev/null || {
  tt rocks install ...
  cacher dir upload "$KEY" .rocks
}

#Key derivation

--hash-from <path> is repeatable; files are hashed by content, directories are hashed by recursively walking entries in sorted relative-path order. The resulting hex digest is truncated to --hash-length characters (default 16, matching the sha256sum file | cut -c1-16 convention).

For a single file path, the digest equals sha256sum file | head -c 16 exactly — so you can migrate existing keys without recomputing them.

Substitution into the key template:

Template --hash-from Result
img/{hash}.tar.zst Dockerfile img/abcd1234….tar.zst
img/build.tar.zst Dockerfile img/build-abcd1234….tar.zst
bin/cacher + --arch-suffix (none) bin/cacher-linux-amd64

cacher key <template> [--hash-from ...] resolves and prints the key without doing anything else — handy when the same key feeds multiple subsequent commands.

#Full builds.sr.ht example

Realistic CI flow: bootstrap the cacher binary itself from the previous release, then dogfood it to cache the Go toolchain, module cache, and a docker image before running tests. See this repo's .builds/test.yml for the live version.

image: ubuntu/noble
packages: [curl, ca-certificates, docker.io]
secrets:
  - <s3-key-id-secret-uuid>
  - <s3-secret-key-secret-uuid>
environment:
  PATH: /home/build/.local/go/bin:/home/build/.local/bin:/usr/local/bin:/usr/bin:/bin
sources:
  - https://git.example.com/myproject
tasks:
  - install_cacher: |
      mkdir -p ~/.local/bin
      curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \
        -o ~/.local/bin/cacher
      chmod +x ~/.local/bin/cacher
  - cacher_init: |
      cacher init \
        --endpoint https://s3.example.com --region garage --bucket cache \
        --prefix myproject \
        --key-file ~/.s3-cache-key-id --secret-file ~/.s3-cache-key-secret
  - install_go: |
      cacher download "golang/go1.26.3.linux-amd64.tar.gz" /tmp/go.tar.gz \
        --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz
      mkdir -p ~/.local && tar -xzf /tmp/go.tar.gz -C ~/.local
  - cache_gomod: |
      KEY="gomod/$(sha256sum myproject/go.sum | cut -c1-16).tar.gz"
      if cacher download "$KEY" /tmp/gomod.tar.gz; then
        mkdir -p ~/go && tar -xzf /tmp/gomod.tar.gz -C ~/go
      else
        cd myproject && go mod download && cd ..
        tar -czf /tmp/gomod.tar.gz -C ~/go pkg/mod
        cacher upload "$KEY" /tmp/gomod.tar.gz
      fi
  - cache_postgres: |
      cacher docker download "docker/postgres-16.tar.zst" \
        postgres:16 --pull
  - test: |
      cd myproject && go test ./...

#Configuration

~/.config/cacher/config.toml:

endpoint    = "https://s3.example.com"
region      = "us-east-1"
bucket      = "ci-cache"
prefix      = "my-project"
arch_suffix = false
key_file    = "~/.s3-cache-key-id"
secret_file = "~/.s3-cache-key-secret"

Every field is overridable, with precedence flag > CACHER_<UPPER> env var > config file > built-in default.

prefix is joined to every key, so callers refer to keys relative to their project namespace.

arch_suffix = true appends -<goos>-<goarch> to every key — useful if you build the same project on multiple architectures. Off by default because turning it on invalidates existing keys.

#Garage / MinIO compatibility

cacher is configured for Garage by default and works identically on MinIO and AWS S3:

  • Path-style addressing (https://endpoint/bucket/key, not https://bucket.endpoint/key).
  • Signature v4.
  • RequestChecksumCalculation / ResponseChecksumValidation set to when_required (Garage doesn't implement boto3's trailing CRC32 checksums introduced in 1.36+).
  • Multipart upload for objects above 5 MiB (handled by manager.Uploader).

#Exit codes

Code Meaning
0 Success
1 exists returned false / key missing
2 Operational error (credentials, network, permission, …)
3 download cache miss with no --url fallback

So shell can branch on exists cleanly:

if cacher exists "$key"; then
  echo "cached"
else
  echo "missing — build it"
fi

#Why not just call aws s3 from shell?

The shell version this replaced repeated five things in every CI task:

  1. Install AWS CLI v2 (~50 MB download per build).
  2. Write ~/.aws/config with the Garage tweaks.
  3. Compute the cache key from file content with sha256sum | cut.
  4. Branch HIT/MISS by hand.
  5. For docker, pipe docker save | zstd | aws s3 cp - (and the inverse).

cacher does (1) by being a single ~14 MB static binary fetched with one wget, and (2)–(5) as built-in commands. Directories aren't handled by the shell version at all — cacher dir closes that gap.

--pull on cacher docker download collapses the docker HIT/MISS branch into one call; the equivalent for files exists too via cacher download --url.

#License

BSD-2-Clause. See LICENSE.