# cacher

S3-backed CI cache helper. A single static Go binary that downloads,
uploads, lists, and invalidates cached build artifacts in any S3-compatible
bucket. Built for [builds.sr.ht](https://builds.sr.ht) but works anywhere
you can run a binary and reach an S3 endpoint.

Replaces the typical CI cache shell loop:

```sh
# before — install awscli, write ~/.aws/config, then in every task:
if aws s3api head-object --bucket "$B" --key "$K" >/dev/null 2>&1; then
  aws s3 cp "s3://$B/$K" "$out"
else
  curl -sSL "$url" -o "$out"
  aws s3 cp "$out" "s3://$B/$K"
fi
```

```sh
# after — one binary, one config, one command:
cacher download "$key" "$out" --url "$url"
```

Same collapse for docker images:

```sh
cacher docker download "$key" "$image:tag" --pull
```

…and for a directory you build yourself, where the fallback is your own
build script:

```sh
cacher dir download "$key" ~/scss --exec 'git clone … && cp -r …'
```

Releases and changelog: [bigbes.pages.srht.bigb.es/ci-cacher](https://bigbes.pages.srht.bigb.es/ci-cacher/).

## Install

```sh
# One-liner: detects the platform, verifies against checksums.txt,
# installs into ~/.local/bin (override with CACHER_BINDIR). On
# builds.sr.ht it also appends the PATH export to ~/.buildenv, so the
# whole bootstrap task is this single line.
curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
```

```sh
# Pre-built binary (latest tag). Substitute your platform:
#   cacher-linux-amd64, cacher-linux-arm64,
#   cacher-darwin-amd64, cacher-darwin-arm64
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
chmod +x ~/.local/bin/cacher

# Pin to a known sha256 (checksums.txt lives next to the binaries):
wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 -O ~/.local/bin/cacher
echo "<expected-sha256>  ~/.local/bin/cacher" | sha256sum -c

# From source:
go install go.bigb.es/cacher@latest
```

Verify with `cacher version`.

## Setup

`cacher init` writes `~/.config/cacher/config.toml` and runs a smoke test
to fail fast on bad credentials. Run it once per CI job after secrets are
mounted:

```sh
cacher init \
  --endpoint    https://s3.example.com \
  --region      us-east-1 \
  --bucket      ci-cache \
  --prefix      my-project \
  --key-file    ~/.s3-cache-key-id \
  --secret-file ~/.s3-cache-key-secret
```

Credentials resolve as **`CACHER_S3_KEY_ID` / `CACHER_S3_SECRET` env
vars > `--key-file` / `--secret-file` > files recorded in config**. Use the
env vars for local-dev runs; use the files in CI where secrets are mounted.

`cacher doctor` repeats the smoke test (HEAD bucket + 1-byte
write/read/delete canary) and prints credential-length diagnostics
without leaking the secret.

## Usage

### Single file — fetch-or-download

```sh
cacher download go-1.26.3.tar.gz /tmp/go.tar.gz \
  --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz \
  --sha256 abc123…
```

Cache HIT: pulls from S3. Cache MISS: GETs the URL, verifies the optional
sha256, writes to the destination, **and uploads to S3** so the next run
hits.

When the artifact isn't a URL away but a command away, `--exec` is the
same shape — the script runs only on a miss and must leave the
destination behind, which is then cached:

```sh
cacher download "bin/tool-$VER" ~/.local/bin/tool \
  --exec 'go build -o ~/.local/bin/tool ./cmd/tool'
```

`--optional` turns a miss into exit 0 instead of an error, for caches
whose absence just means a cold build:

```sh
cacher dir download "$KEY_GOC" ~/.cache/go-build --optional
```

```sh
cacher upload my-key /path/to/artifact            # skip if present
cacher upload my-key /path/to/artifact --force    # overwrite
cacher exists my-key                              # exit 0 hit, 1 miss
cacher list   my-prefix                           # /-delimited (aws s3 ls)
cacher list   my-prefix --recursive               # flat
cacher list   --root                              # ignore configured prefix
cacher delete my-key                              # invalidate
```

### Docker images — streamed save/load

For an image pulled from a registry, the cache-or-pull pattern is a
single command:

```sh
# Cache HIT → docker load from S3. MISS → docker pull + seed S3 + tag stays local.
cacher docker download "docker/dxflrs-garage-v2.3.0.tar.zst" \
  dxflrs/garage:v2.3.0 --pull
```

For images you build locally, `--exec` replaces `--pull` — key by
Dockerfile content, and the build only runs on a miss:

```sh
cacher docker download "images/{hash}.tar.zst" myimage:latest \
  --hash-from Dockerfile \
  --exec 'docker build -t myimage:latest .'
```

The primitives are still there if you want the branch by hand:

```sh
KEY=$(cacher key "images/{hash}.tar.zst" --hash-from Dockerfile)

if ! cacher docker exists "$KEY"; then
  docker build -t myimage:latest .
  cacher docker upload "$KEY" myimage:latest
else
  cacher docker download "$KEY" myimage:latest
fi
```

The save/load pipeline is fully streamed — `docker save | zstd | s3` and
the inverse, no on-disk tempfile. The zstd codec is pure Go
([klauspost/compress](https://github.com/klauspost/compress)), so no
external `zstd` binary is needed.

### Directory caching — the real CI speedup

Cache resolved trees keyed by a lockfile hash. Skip resolution entirely
when nothing changed:

```sh
# Go module cache, keyed by go.sum:
cacher dir download "go-mod/{hash}.tar.zst" ~/go/pkg/mod \
  --hash-from go.sum --exec 'go mod download'

# Lua rocks tree, keyed by rockspec:
cacher dir download "rocks/{hash}.tar.zst" .rocks \
  --hash-from project-scm-1.rockspec --exec 'tt rocks install ...'
```

`--exec` runs only on a miss, with `sh -c`, stdout and stderr wired
straight to the build log. The destination directory is created before
the script starts, so the script can write into it without its own
`mkdir -p`. On success the tree is packed and uploaded; if the upload
fails the build still continues, because the content the script produced
is already on disk. If the script itself fails, `cacher` exits with the
script's own status and nothing is cached.

When restore and save can't be one command — because the tree is only
final several tasks later, as with a Go build cache — use the pair:

```sh
KEY=$(cacher key "gocache/{hash}.tar.zst" --hash-from go.sum)
cacher dir download "$KEY" ~/.cache/go-build --optional  # miss is fine
... build, test, package ...
cacher dir upload "$KEY" ~/.cache/go-build               # no-op if present
```

### Key derivation

`--hash-from <path>` is repeatable; files are hashed by content, directories
are hashed by recursively walking entries in sorted relative-path order.
The resulting hex digest is truncated to `--hash-length` characters
(default 16, matching the `sha256sum file | cut -c1-16` convention).

For a single file path, the digest equals `sha256sum file | head -c 16`
exactly — so you can migrate existing keys without recomputing them.

Substitution into the key template:

| Template                       | --hash-from   | Result                              |
|--------------------------------|---------------|-------------------------------------|
| `img/{hash}.tar.zst`           | `Dockerfile`  | `img/abcd1234….tar.zst`             |
| `img/build.tar.zst`            | `Dockerfile`  | `img/build-abcd1234….tar.zst`       |
| `bin/cacher` + `--arch-suffix` | (none)        | `bin/cacher-linux-amd64`            |

`cacher key <template> [--hash-from ...]` resolves and prints the key
without doing anything else — handy when the same key feeds multiple
subsequent commands.

## Full builds.sr.ht example

Realistic CI flow: bootstrap the cacher binary itself from the previous
release, then dogfood it to cache the Go toolchain, module cache, and a
docker image before running tests. See this repo's
[`.builds/test.yml`](.builds/test.yml) for the live version.

```yaml
image: ubuntu/noble
packages: [curl, ca-certificates, docker.io]
secrets:
  - <s3-key-id-secret-uuid>
  - <s3-secret-key-secret-uuid>
environment:
  PATH: /home/build/.local/go/bin:/home/build/.local/bin:/usr/local/bin:/usr/bin:/bin
sources:
  - https://git.example.com/myproject
tasks:
  - install_cacher: |
      curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
  - cacher_init: |
      cacher init \
        --endpoint https://s3.example.com --region garage --bucket cache \
        --prefix myproject \
        --key-file ~/.s3-cache-key-id --secret-file ~/.s3-cache-key-secret
  - install_go: |
      cacher download "golang/go1.26.3.linux-amd64.tar.gz" /tmp/go.tar.gz \
        --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz
      mkdir -p ~/.local && tar -xzf /tmp/go.tar.gz -C ~/.local
  - cache_gomod: |
      cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
        --hash-from myproject/go.sum \
        --exec 'cd myproject && go mod download'
  - cache_postgres: |
      cacher docker download "docker/postgres-16.tar.zst" \
        postgres:16 --pull
  - test: |
      cd myproject && go test ./...
```

## Configuration

`~/.config/cacher/config.toml`:

```toml
endpoint    = "https://s3.example.com"
region      = "us-east-1"
bucket      = "ci-cache"
prefix      = "my-project"
arch_suffix = false
key_file    = "~/.s3-cache-key-id"
secret_file = "~/.s3-cache-key-secret"
```

Every field is overridable, with precedence **flag > `CACHER_<UPPER>`
env var > config file > built-in default**.

`prefix` is joined to every key, so callers refer to keys relative to
their project namespace.

`arch_suffix = true` appends `-<goos>-<goarch>` to every key — useful if
you build the same project on multiple architectures. Off by default
because turning it on invalidates existing keys.

## Garage / MinIO compatibility

`cacher` is configured for [Garage](https://garagehq.deuxfleurs.fr/) by
default and works identically on MinIO and AWS S3:

- Path-style addressing (`https://endpoint/bucket/key`, not
  `https://bucket.endpoint/key`).
- Signature v4.
- `RequestChecksumCalculation` / `ResponseChecksumValidation` set to
  `when_required` (Garage doesn't implement boto3's trailing CRC32
  checksums introduced in 1.36+).
- Multipart upload for objects above 5 MiB (handled by
  `manager.Uploader`).

## Exit codes

| Code | Meaning                                                    |
|------|------------------------------------------------------------|
| `0`  | Success (including a miss under `--optional`)              |
| `1`  | `exists` returned false / key missing                      |
| `2`  | Operational error (credentials, network, permission, …)   |
| `3`  | `download` cache miss with no `--url`/`--exec` fallback     |
| *n*  | The `--exec` script's own exit status, propagated verbatim |

So shell can branch on `exists` cleanly:

```sh
if cacher exists "$key"; then
  echo "cached"
else
  echo "missing — build it"
fi
```

## Why not just call `aws s3` from shell?

The shell version this replaced repeated five things in every CI task:

1. Install AWS CLI v2 (~50 MB download per build).
2. Write `~/.aws/config` with the Garage tweaks.
3. Compute the cache key from file content with `sha256sum | cut`.
4. Branch HIT/MISS by hand.
5. For docker, pipe `docker save | zstd | aws s3 cp -` (and the inverse).

`cacher` does (1) by being a single ~14 MB static binary fetched with
one `wget`, and (2)–(5) as built-in commands. Directories aren't
handled by the shell version at all — `cacher dir` closes that gap.

`--pull` on `cacher docker download` collapses the docker HIT/MISS
branch into one call; the equivalent for files exists too via
`cacher download --url`, and `--exec` generalises both to any fallback
you can express as a shell command — which is what removes the last
`if`/`fi` block from a CI manifest.

## License

BSD-2-Clause. See `LICENSE`.