#cacher

S3-backed CI cache helper. A single static Go binary that downloads, uploads, lists, and invalidates cached build artifacts in any S3-compatible bucket. Built for builds.sr.ht but works anywhere you can run a binary and reach an S3 endpoint.

Replaces the typical CI cache shell loop:

# before — install awscli, write ~/.aws/config, then in every task:
if aws s3api head-object --bucket "$B" --key "$K" >/dev/null 2>&1; then
  aws s3 cp "s3://$B/$K" "$out"
else
  curl -sSL "$url" -o "$out"
  aws s3 cp "$out" "s3://$B/$K"
fi

# after — one binary, one config, one command:
cacher download "$key" "$out" --url "$url"

Same collapse for docker images:

cacher docker download "$key" "$image:tag" --pull

…and for a directory you build yourself, where the fallback is your own build script:

cacher dir download "$key" ~/scss --exec 'git clone … && cp -r …'

Releases and changelog: bigbes.pages.srht.bigb.es/ci-cacher.

#Install

# One-liner: detects the platform, verifies against checksums.txt,
# installs into ~/.local/bin (override with CACHER_BINDIR). On
# builds.sr.ht it also appends the PATH export to ~/.buildenv, so the
# whole bootstrap task is this single line.
curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh

# Pre-built binary (latest tag). Substitute your platform:
#   cacher-linux-amd64, cacher-linux-arm64,
#   cacher-darwin-amd64, cacher-darwin-arm64
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
chmod +x ~/.local/bin/cacher

# Pin to a known sha256 (checksums.txt lives next to the binaries):
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
echo "<expected-sha256>  ~/.local/bin/cacher" | sha256sum -c

# From source:
go install go.bigb.es/cacher@latest

Verify with cacher version.

#Setup

cacher init writes ~/.config/cacher/config.toml and runs a smoke test to fail fast on bad credentials. Run it once per CI job after secrets are mounted:

cacher init \
  --endpoint    https://s3.example.com \
  --region      us-east-1 \
  --bucket      ci-cache \
  --prefix      my-project \
  --key-file    ~/.s3-cache-key-id \
  --secret-file ~/.s3-cache-key-secret

Credentials resolve as CACHER_S3_KEY_ID / CACHER_S3_SECRET env vars > --key-file / --secret-file > files recorded in config. Use the env vars for local-dev runs; use the files in CI where secrets are mounted.

cacher doctor repeats the smoke test (HEAD bucket + 1-byte write/read/delete canary) and prints credential-length diagnostics without leaking the secret.

#Usage

#Single file — fetch-or-download

cacher download go-1.26.3.tar.gz /tmp/go.tar.gz \
  --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz \
  --sha256 abc123…

Cache HIT: pulls from S3. Cache MISS: GETs the URL, verifies the optional sha256, writes to the destination, and uploads to S3 so the next run hits.

When the artifact isn't a URL away but a command away, --exec is the same shape — the script runs only on a miss and must leave the destination behind, which is then cached:

cacher download "bin/tool-$VER" ~/.local/bin/tool \
  --exec 'go build -o ~/.local/bin/tool ./cmd/tool'

--optional turns a miss into exit 0 instead of an error, for caches whose absence just means a cold build:

cacher dir download "$KEY_GOC" ~/.cache/go-build --optional

cacher upload my-key /path/to/artifact            # skip if present
cacher upload my-key /path/to/artifact --force    # overwrite
cacher exists my-key                              # exit 0 hit, 1 miss
cacher list   my-prefix                           # /-delimited (aws s3 ls)
cacher list   my-prefix --recursive               # flat
cacher list   --root                              # ignore configured prefix
cacher delete my-key                              # invalidate

#Docker images — streamed save/load

For an image pulled from a registry, the cache-or-pull pattern is a single command:

# Cache HIT → docker load from S3. MISS → docker pull + seed S3 + tag stays local.
cacher docker download "docker/dxflrs-garage-v2.3.0.tar.zst" \
  dxflrs/garage:v2.3.0 --pull

For images you build locally, --exec replaces --pull — key by Dockerfile content, and the build only runs on a miss:

cacher docker download "images/{hash}.tar.zst" myimage:latest \
  --hash-from Dockerfile \
  --exec 'docker build -t myimage:latest .'

The primitives are still there if you want the branch by hand:

KEY=$(cacher key "images/{hash}.tar.zst" --hash-from Dockerfile)

if ! cacher docker exists "$KEY"; then
  docker build -t myimage:latest .
  cacher docker upload "$KEY" myimage:latest
else
  cacher docker download "$KEY" myimage:latest
fi

The save/load pipeline is fully streamed — docker save | zstd | s3 and the inverse, no on-disk tempfile. The zstd codec is pure Go (klauspost/compress), so no external zstd binary is needed.

#Directory caching — the real CI speedup

Cache resolved trees keyed by a lockfile hash. Skip resolution entirely when nothing changed:

# Go module cache, keyed by go.sum:
cacher dir download "go-mod/{hash}.tar.zst" ~/go/pkg/mod \
  --hash-from go.sum --exec 'go mod download'

# Lua rocks tree, keyed by rockspec:
cacher dir download "rocks/{hash}.tar.zst" .rocks \
  --hash-from project-scm-1.rockspec --exec 'tt rocks install ...'

--exec runs only on a miss, with sh -c, stdout and stderr wired straight to the build log. The destination directory is created before the script starts, so the script can write into it without its own mkdir -p. On success the tree is packed and uploaded; if the upload fails the build still continues, because the content the script produced is already on disk. If the script itself fails, cacher exits with the script's own status and nothing is cached.

When restore and save can't be one command — because the tree is only final several tasks later, as with a Go build cache — use the pair:

KEY=$(cacher key "gocache/{hash}.tar.zst" --hash-from go.sum)
cacher dir download "$KEY" ~/.cache/go-build --optional  # miss is fine
... build, test, package ...
cacher dir upload "$KEY" ~/.cache/go-build               # no-op if present

#Key derivation

--hash-from <path> is repeatable; files are hashed by content, directories are hashed by recursively walking entries in sorted relative-path order. The resulting hex digest is truncated to --hash-length characters (default 16, matching the sha256sum file | cut -c1-16 convention).

For a single file path, the digest equals sha256sum file | head -c 16 exactly — so you can migrate existing keys without recomputing them.

Substitution into the key template:

cacher key <template> [--hash-from ...] resolves and prints the key without doing anything else — handy when the same key feeds multiple subsequent commands.

#Full builds.sr.ht example

Realistic CI flow: bootstrap the cacher binary itself from the previous release, then dogfood it to cache the Go toolchain, module cache, and a docker image before running tests. See this repo's .builds/test.yml for the live version.

image: ubuntu/noble
packages: [curl, ca-certificates, docker.io]
secrets:
  - <s3-key-id-secret-uuid>
  - <s3-secret-key-secret-uuid>
environment:
  PATH: /home/build/.local/go/bin:/home/build/.local/bin:/usr/local/bin:/usr/bin:/bin
sources:
  - https://git.example.com/myproject
tasks:
  - install_cacher: |
      curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
  - cacher_init: |
      cacher init \
        --endpoint https://s3.example.com --region garage --bucket cache \
        --prefix myproject \
        --key-file ~/.s3-cache-key-id --secret-file ~/.s3-cache-key-secret
  - install_go: |
      cacher download "golang/go1.26.3.linux-amd64.tar.gz" /tmp/go.tar.gz \
        --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz
      mkdir -p ~/.local && tar -xzf /tmp/go.tar.gz -C ~/.local
  - cache_gomod: |
      cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
        --hash-from myproject/go.sum \
        --exec 'cd myproject && go mod download'
  - cache_postgres: |
      cacher docker download "docker/postgres-16.tar.zst" \
        postgres:16 --pull
  - test: |
      cd myproject && go test ./...

#Configuration

~/.config/cacher/config.toml:

endpoint    = "https://s3.example.com"
region      = "us-east-1"
bucket      = "ci-cache"
prefix      = "my-project"
arch_suffix = false
key_file    = "~/.s3-cache-key-id"
secret_file = "~/.s3-cache-key-secret"

Every field is overridable, with precedence flag > CACHER_<UPPER> env var > config file > built-in default.

prefix is joined to every key, so callers refer to keys relative to their project namespace.

arch_suffix = true appends -<goos>-<goarch> to every key — useful if you build the same project on multiple architectures. Off by default because turning it on invalidates existing keys.

#Garage / MinIO compatibility

cacher is configured for Garage by default and works identically on MinIO and AWS S3:

• Path-style addressing (https://endpoint/bucket/key, not https://bucket.endpoint/key).
• Signature v4.
• RequestChecksumCalculation / ResponseChecksumValidation set to when_required (Garage doesn't implement boto3's trailing CRC32 checksums introduced in 1.36+).
• Multipart upload for objects above 5 MiB (handled by manager.Uploader).

#Exit codes

So shell can branch on exists cleanly:

if cacher exists "$key"; then
  echo "cached"
else
  echo "missing — build it"
fi

#Why not just call aws s3 from shell?

The shell version this replaced repeated five things in every CI task:

1. Install AWS CLI v2 (~50 MB download per build).
2. Write ~/.aws/config with the Garage tweaks.
3. Compute the cache key from file content with sha256sum | cut.
4. Branch HIT/MISS by hand.
5. For docker, pipe docker save | zstd | aws s3 cp - (and the inverse).

cacher does (1) by being a single ~14 MB static binary fetched with one wget, and (2)–(5) as built-in commands. Directories aren't handled by the shell version at all — cacher dir closes that gap.

--pull on cacher docker download collapses the docker HIT/MISS branch into one call; the equivalent for files exists too via cacher download --url, and --exec generalises both to any fallback you can express as a shell command — which is what removes the last if/fi block from a CI manifest.

#License

BSD-2-Clause. See LICENSE.