@@ 6,6 6,30 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
+### Added
+- `--exec <script>` on `download`, `dir download` and `docker download` —
+ on a cache miss the script runs through `sh -c` and its result seeds
+ the cache. Generalises the `--url` and `--pull` fallbacks to anything
+ expressible as a shell command, which collapses the
+ restore-or-build `if`/`fi` block CI manifests repeat per cache:
+
+ ```sh
+ cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
+ --hash-from go.sum --exec 'go mod download'
+ ```
+
+ The destination directory is created before the script runs; the seed
+ upload is best-effort; a failing script is fatal and propagates its own
+ exit status.
+- `--optional` on `download` and `dir download` — a cache miss exits 0
+ instead of 1/3, so `set -e` manifests no longer need a trailing
+ `|| echo "cache miss"`.
+- `install.sh`, published next to the binaries: detects the platform,
+ verifies against `checksums.txt`, installs into `~/.local/bin` and, on
+ builds.sr.ht, appends the `PATH` export to `~/.buildenv`. Reduces the
+ bootstrap task to
+ `curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh`.
+
## [0.1.2] — 2026-05-26
### Changed
@@ 28,11 28,26 @@ Same collapse for docker images:
cacher docker download "$key" "$image:tag" --pull
```
+…and for a directory you build yourself, where the fallback is your own
+build script:
+
+```sh
+cacher dir download "$key" ~/scss --exec 'git clone … && cp -r …'
+```
+
Releases and changelog: [bigbes.pages.srht.bigb.es/ci-cacher](https://bigbes.pages.srht.bigb.es/ci-cacher/).
## Install
```sh
+# One-liner: detects the platform, verifies against checksums.txt,
+# installs into ~/.local/bin (override with CACHER_BINDIR). On
+# builds.sr.ht it also appends the PATH export to ~/.buildenv, so the
+# whole bootstrap task is this single line.
+curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
+```
+
+```sh
# Pre-built binary (latest tag). Substitute your platform:
# cacher-linux-amd64, cacher-linux-arm64,
# cacher-darwin-amd64, cacher-darwin-arm64
@@ 87,6 102,22 @@ Cache HIT: pulls from S3. Cache MISS: GETs the URL, verifies the optional
sha256, writes to the destination, **and uploads to S3** so the next run
hits.
+When the artifact isn't a URL away but a command away, `--exec` is the
+same shape — the script runs only on a miss and must leave the
+destination behind, which is then cached:
+
+```sh
+cacher download "bin/tool-$VER" ~/.local/bin/tool \
+ --exec 'go build -o ~/.local/bin/tool ./cmd/tool'
+```
+
+`--optional` turns a miss into exit 0 instead of an error, for caches
+whose absence just means a cold build:
+
+```sh
+cacher dir download "$KEY_GOC" ~/.cache/go-build --optional
+```
+
```sh
cacher upload my-key /path/to/artifact # skip if present
cacher upload my-key /path/to/artifact --force # overwrite
@@ 108,8 139,16 @@ cacher docker download "docker/dxflrs-garage-v2.3.0.tar.zst" \
dxflrs/garage:v2.3.0 --pull
```
-For images you build locally, drive the cache by hand — same primitives,
-key by Dockerfile content:
+For images you build locally, `--exec` replaces `--pull` — key by
+Dockerfile content, and the build only runs on a miss:
+
+```sh
+cacher docker download "images/{hash}.tar.zst" myimage:latest \
+ --hash-from Dockerfile \
+ --exec 'docker build -t myimage:latest .'
+```
+
+The primitives are still there if you want the branch by hand:
```sh
KEY=$(cacher key "images/{hash}.tar.zst" --hash-from Dockerfile)
@@ 134,18 173,30 @@ when nothing changed:
```sh
# Go module cache, keyed by go.sum:
-KEY=$(cacher key "go-mod/{hash}.tar.zst" --hash-from go.sum)
-cacher dir download "$KEY" ~/go/pkg/mod 2>/dev/null || {
- go mod download
- cacher dir upload "$KEY" ~/go/pkg/mod
-}
+cacher dir download "go-mod/{hash}.tar.zst" ~/go/pkg/mod \
+ --hash-from go.sum --exec 'go mod download'
# Lua rocks tree, keyed by rockspec:
-KEY=$(cacher key "rocks/{hash}.tar.zst" --hash-from project-scm-1.rockspec)
-cacher dir download "$KEY" .rocks 2>/dev/null || {
- tt rocks install ...
- cacher dir upload "$KEY" .rocks
-}
+cacher dir download "rocks/{hash}.tar.zst" .rocks \
+ --hash-from project-scm-1.rockspec --exec 'tt rocks install ...'
+```
+
+`--exec` runs only on a miss, with `sh -c`, stdout and stderr wired
+straight to the build log. The destination directory is created before
+the script starts, so the script can write into it without its own
+`mkdir -p`. On success the tree is packed and uploaded; if the upload
+fails the build still continues, because the content the script produced
+is already on disk. If the script itself fails, `cacher` exits with the
+script's own status and nothing is cached.
+
+When restore and save can't be one command — because the tree is only
+final several tasks later, as with a Go build cache — use the pair:
+
+```sh
+KEY=$(cacher key "gocache/{hash}.tar.zst" --hash-from go.sum)
+cacher dir download "$KEY" ~/.cache/go-build --optional # miss is fine
+... build, test, package ...
+cacher dir upload "$KEY" ~/.cache/go-build # no-op if present
```
### Key derivation
@@ 189,10 240,7 @@ sources:
- https://git.example.com/myproject
tasks:
- install_cacher: |
- mkdir -p ~/.local/bin
- curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \
- -o ~/.local/bin/cacher
- chmod +x ~/.local/bin/cacher
+ curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh
- cacher_init: |
cacher init \
--endpoint https://s3.example.com --region garage --bucket cache \
@@ 203,14 251,9 @@ tasks:
--url https://go.dev/dl/go1.26.3.linux-amd64.tar.gz
mkdir -p ~/.local && tar -xzf /tmp/go.tar.gz -C ~/.local
- cache_gomod: |
- KEY="gomod/$(sha256sum myproject/go.sum | cut -c1-16).tar.gz"
- if cacher download "$KEY" /tmp/gomod.tar.gz; then
- mkdir -p ~/go && tar -xzf /tmp/gomod.tar.gz -C ~/go
- else
- cd myproject && go mod download && cd ..
- tar -czf /tmp/gomod.tar.gz -C ~/go pkg/mod
- cacher upload "$KEY" /tmp/gomod.tar.gz
- fi
+ cacher dir download "gomod/{hash}.tar.zst" ~/go/pkg/mod \
+ --hash-from myproject/go.sum \
+ --exec 'cd myproject && go mod download'
- cache_postgres: |
cacher docker download "docker/postgres-16.tar.zst" \
postgres:16 --pull
@@ 258,12 301,13 @@ default and works identically on MinIO and AWS S3:
## Exit codes
-| Code | Meaning |
-|------|-----------------------------------------------------------|
-| `0` | Success |
-| `1` | `exists` returned false / key missing |
-| `2` | Operational error (credentials, network, permission, …) |
-| `3` | `download` cache miss with no `--url` fallback |
+| Code | Meaning |
+|------|------------------------------------------------------------|
+| `0` | Success (including a miss under `--optional`) |
+| `1` | `exists` returned false / key missing |
+| `2` | Operational error (credentials, network, permission, …) |
+| `3` | `download` cache miss with no `--url`/`--exec` fallback |
+| *n* | The `--exec` script's own exit status, propagated verbatim |
So shell can branch on `exists` cleanly:
@@ 291,7 335,9 @@ handled by the shell version at all — `cacher dir` closes that gap.
`--pull` on `cacher docker download` collapses the docker HIT/MISS
branch into one call; the equivalent for files exists too via
-`cacher download --url`.
+`cacher download --url`, and `--exec` generalises both to any fallback
+you can express as a shell command — which is what removes the last
+`if`/`fi` block from a CI manifest.
## License
@@ 107,7 107,15 @@
</tbody>
</table>
<p>
- All four hashes plus filenames are also available in a single
+ The <a href="install.sh">install.sh</a> bootstrap picks the right asset for
+ the host, verifies it against <a href="checksums.txt">checksums.txt</a>, and
+ installs into <code>~/.local/bin</code> — on builds.sr.ht it also appends the
+ <code>PATH</code> export to <code>~/.buildenv</code>, so a CI manifest needs
+ exactly one line:
+</p>
+<pre><code>curl -sSL https://bigbes.pages.srht.bigb.es/ci-cacher/install.sh | sh</code></pre>
+<p>
+ By hand, all four hashes plus filenames are also available in a single
<a href="checksums.txt">checksums.txt</a> for piping into <code>sha256sum -c</code>:
</p>
<pre><code>wget https://bigbes.pages.srht.bigb.es/ci-cacher/cacher-linux-amd64 \
@@ 135,6 143,8 @@ cacher download "$key" "$out" --url "$url"</code></pre>
<li><code>exists</code> / <code>list</code> / <code>delete</code> / <code>key</code> — management + shell helpers</li>
<li><code>docker {exists,download,upload}</code> — streamed save/load via zstd</li>
<li><code>dir {download,upload}</code> — tar+zstd directory caching, keyed by content hash</li>
+ <li><code>--exec</code> on every <code>download</code> — run a build script on a miss
+ and seed the cache with what it produced; <code>--optional</code> makes a miss exit 0</li>
</ul>
<p>
Run <code>cacher --help</code> for the full surface. Read