~bigbes/ci-cacher

a574cffb871d0b4324494880ee589fb41e76ba58 — Eugene Blikh 11 days ago a42721d
README, landing page, CHANGELOG: document --exec, --optional, install.sh

Rewrites the directory-caching and docker sections around --exec instead
of the shell branch, replaces the bootstrap task in the full builds.sr.ht
example with the install.sh one-liner, and records that the --exec
script's exit status is propagated verbatim in the exit-code table.
3 files changed, 112 insertions(+), 32 deletions(-)

M CHANGELOG.md
M README.md
M docs/index.html
M CHANGELOG.md => CHANGELOG.md +24 -0
@@ 6,6 6,30 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- `--exec <script>` on `download`, `dir download` and `docker download` —
  on a cache miss the script runs through `sh -c` and its result seeds
  the cache. Generalises the `--url` and `--pull` fallbacks to anything
  expressible as a shell command, which collapses the
  restore-or-build `if`/`fi` block CI manifests repeat per cache:

  ```sh
  cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
    --hash-from go.sum --exec 'go mod download'
  ```

  The destination directory is created before the script runs; the seed
  upload is best-effort; a failing script is fatal and propagates its own
  exit status.
- `--optional` on `download` and `dir download` — a cache miss exits 0
  instead of 1/3, so `set -e` manifests no longer need a trailing
  `|| echo "cache miss"`.
- `install.sh`, published next to the binaries: detects the platform,
  verifies against `checksums.txt`, installs into `~/.local/bin` and, on
  builds.sr.ht, appends the `PATH` export to `~/.buildenv`. Reduces the
  bootstrap task to
  `curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh`.

## [0.1.2] — 2026-05-26

### Changed

M README.md => README.md +77 -31
@@ 28,11 28,26 @@ Same collapse for docker images:
cacher docker download "$key" "$image:tag" --pull
```

…and for a directory you build yourself, where the fallback is your own
build script:

```sh
cacher dir download "$key" ~/scss --exec 'git clone … && cp -r …'
```

Releases and changelog: [bigbes.pages.srht.bigb.es/ci-cacher](https://bigbes.pages.srht.bigb.es/ci-cacher/).

## Install

```sh
# One-liner: detects the platform, verifies against checksums.txt,
# installs into ~/.local/bin (override with CACHER_BINDIR). On
# builds.sr.ht it also appends the PATH export to ~/.buildenv, so the
# whole bootstrap task is this single line.
curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
```

```sh
# Pre-built binary (latest tag). Substitute your platform:
#   cacher-linux-amd64, cacher-linux-arm64,
#   cacher-darwin-amd64, cacher-darwin-arm64


@@ 87,6 102,22 @@ Cache HIT: pulls from S3. Cache MISS: GETs the URL, verifies the optional
sha256, writes to the destination, **and uploads to S3** so the next run
hits.

When the artifact isn't a URL away but a command away, `--exec` is the
same shape — the script runs only on a miss and must leave the
destination behind, which is then cached:

```sh
cacher download "bin/tool-$VER" ~/.local/bin/tool \
  --exec 'go build -o ~/.local/bin/tool ./cmd/tool'
```

`--optional` turns a miss into exit 0 instead of an error, for caches
whose absence just means a cold build:

```sh
cacher dir download "$KEY_GOC" ~/.cache/go-build --optional
```

```sh
cacher upload my-key /path/to/artifact            # skip if present
cacher upload my-key /path/to/artifact --force    # overwrite


@@ 108,8 139,16 @@ cacher docker download "docker/dxflrs-garage-v2.3.0.tar.zst" \
  dxflrs/garage:v2.3.0 --pull
```

For images you build locally, drive the cache by hand — same primitives,
key by Dockerfile content:
For images you build locally, `--exec` replaces `--pull` — key by
Dockerfile content, and the build only runs on a miss:

```sh
cacher docker download "images/{hash}.tar.zst" myimage:latest \
  --hash-from Dockerfile \
  --exec 'docker build -t myimage:latest .'
```

The primitives are still there if you want the branch by hand:

```sh
KEY=$(cacher key "images/{hash}.tar.zst" --hash-from Dockerfile)


@@ 134,18 173,30 @@ when nothing changed:

```sh
# Go module cache, keyed by go.sum:
KEY=$(cacher key "go-mod/{hash}.tar.zst" --hash-from go.sum)
cacher dir download "$KEY" ~/go/pkg/mod 2>/dev/null || {
  go mod download
  cacher dir upload "$KEY" ~/go/pkg/mod
}
cacher dir download "go-mod/{hash}.tar.zst" ~/go/pkg/mod \
  --hash-from go.sum --exec 'go mod download'

# Lua rocks tree, keyed by rockspec:
KEY=$(cacher key "rocks/{hash}.tar.zst" --hash-from project-scm-1.rockspec)
cacher dir download "$KEY" .rocks 2>/dev/null || {
  tt rocks install ...
  cacher dir upload "$KEY" .rocks
}
cacher dir download "rocks/{hash}.tar.zst" .rocks \
  --hash-from project-scm-1.rockspec --exec 'tt rocks install ...'
```

`--exec` runs only on a miss, with `sh -c`, stdout and stderr wired
straight to the build log. The destination directory is created before
the script starts, so the script can write into it without its own
`mkdir -p`. On success the tree is packed and uploaded; if the upload
fails the build still continues, because the content the script produced
is already on disk. If the script itself fails, `cacher` exits with the
script's own status and nothing is cached.

When restore and save can't be one command — because the tree is only
final several tasks later, as with a Go build cache — use the pair:

```sh
KEY=$(cacher key "gocache/{hash}.tar.zst" --hash-from go.sum)
cacher dir download "$KEY" ~/.cache/go-build --optional  # miss is fine
... build, test, package ...
cacher dir upload "$KEY" ~/.cache/go-build               # no-op if present
```

### Key derivation


@@ 189,10 240,7 @@ sources:
  - https://git.example.com/myproject
tasks:
  - install_cacher: |
      mkdir -p ~/.local/bin
      curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \
        -o ~/.local/bin/cacher
      chmod +x ~/.local/bin/cacher
      curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
  - cacher_init: |
      cacher init \
        --endpoint https://s3.example.com --region garage --bucket cache \


@@ 203,14 251,9 @@ tasks:
        --url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz
      mkdir -p ~/.local && tar -xzf /tmp/go.tar.gz -C ~/.local
  - cache_gomod: |
      KEY="gomod/$(sha256sum myproject/go.sum | cut -c1-16).tar.gz"
      if cacher download "$KEY" /tmp/gomod.tar.gz; then
        mkdir -p ~/go && tar -xzf /tmp/gomod.tar.gz -C ~/go
      else
        cd myproject && go mod download && cd ..
        tar -czf /tmp/gomod.tar.gz -C ~/go pkg/mod
        cacher upload "$KEY" /tmp/gomod.tar.gz
      fi
      cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
        --hash-from myproject/go.sum \
        --exec 'cd myproject && go mod download'
  - cache_postgres: |
      cacher docker download "docker/postgres-16.tar.zst" \
        postgres:16 --pull


@@ 258,12 301,13 @@ default and works identically on MinIO and AWS S3:

## Exit codes

| Code | Meaning                                                   |
|------|-----------------------------------------------------------|
| `0`  | Success                                                   |
| `1`  | `exists` returned false / key missing                     |
| `2`  | Operational error (credentials, network, permission, …)  |
| `3`  | `download` cache miss with no `--url` fallback            |
| Code | Meaning                                                    |
|------|------------------------------------------------------------|
| `0`  | Success (including a miss under `--optional`)              |
| `1`  | `exists` returned false / key missing                      |
| `2`  | Operational error (credentials, network, permission, …)   |
| `3`  | `download` cache miss with no `--url`/`--exec` fallback     |
| *n*  | The `--exec` script's own exit status, propagated verbatim |

So shell can branch on `exists` cleanly:



@@ 291,7 335,9 @@ handled by the shell version at all — `cacher dir` closes that gap.

`--pull` on `cacher docker download` collapses the docker HIT/MISS
branch into one call; the equivalent for files exists too via
`cacher download --url`.
`cacher download --url`, and `--exec` generalises both to any fallback
you can express as a shell command — which is what removes the last
`if`/`fi` block from a CI manifest.

## License


M docs/index.html => docs/index.html +11 -1
@@ 107,7 107,15 @@
  </tbody>
</table>
<p>
  All four hashes plus filenames are also available in a single
  The <a href="install.sh">install.sh</a> bootstrap picks the right asset for
  the host, verifies it against <a href="checksums.txt">checksums.txt</a>, and
  installs into <code>~/.local/bin</code> — on builds.sr.ht it also appends the
  <code>PATH</code> export to <code>~/.buildenv</code>, so a CI manifest needs
  exactly one line:
</p>
<pre><code>curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh</code></pre>
<p>
  By hand, all four hashes plus filenames are also available in a single
  <a href="checksums.txt">checksums.txt</a> for piping into <code>sha256sum -c</code>:
</p>
<pre><code>wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \


@@ 135,6 143,8 @@ cacher download "$key" "$out" --url "$url"</code></pre>
  <li><code>exists</code> / <code>list</code> / <code>delete</code> / <code>key</code> — management + shell helpers</li>
  <li><code>docker {exists,download,upload}</code> — streamed save/load via zstd</li>
  <li><code>dir {download,upload}</code> — tar+zstd directory caching, keyed by content hash</li>
  <li><code>--exec</code> on every <code>download</code> — run a build script on a miss
      and seed the cache with what it produced; <code>--optional</code> makes a miss exit 0</li>
</ul>
<p>
  Run <code>cacher --help</code> for the full surface. Read