~bigbes/sourcehut-root

ref: 72581ff94e21863373e2c72a01eb75332cb22f32 sourcehut-root/skills/sourcehut-ci/references/integrations.md -rw-r--r-- 15.6 KiB
72581ff9 — Eugene Blikh docs(sourcehut-ci): fix factual errors found in full audit 2 days ago

#Integrations and Build Submission

Builds.sr.ht is a job runner. Multiple things can submit jobs to it: git.sr.ht on push, the project hub on patch email, the web form, hut, the GraphQL API. This file covers how each works.

#git.sr.ht auto-submission

When you push to a git.sr.ht repo, the post-update hook walks the new refs in the push, parses any matching manifests out of each new commit's tree, and fires a GraphQL submit mutation against builds.sr.ht for each.

The default discovery glob is comma-separated and covers both extensions:

.build.yml,.builds/*.yml,.build.yaml,.builds/*.yaml
  • A single .build.yml/.yaml at the repo root is the common case.
  • .builds/*.yml (or .yaml) supports multiple manifests. Each becomes an independent job.
  • The hook caps total submissions at 4 per push, summed across every ref in the push. So a push that touches three branches each carrying two manifests still gets at most 4 jobs total; the rest are silently dropped. When more than 4 are present in a single ref's manifest set, the hook reads them into a Go map and iterates — so "chosen randomly" is the practical effect.
  • Override the discovery pattern per-push with git push -o submit=".sourcehut/*.yml" (fnmatch syntax, comma-separated).

Before submission, the hook rewrites the manifest:

  1. sources: injection. For each entry whose basename equals the pushed repo's name, the URL is replaced with the canonical clone URL plus #<commit-sha> of the pushed commit. If no entry matches, the clone URL is appended to sources:. Private repos get the SSH form (git+ssh://git@<host>/~<owner>/<repo>); public repos get HTTPS. The commit-sha pin guarantees the build runs against exactly what you pushed.
  2. environment: augmentation. BUILD_SUBMITTER=git.sr.ht and GIT_REF=<pushed ref> are added (and overwrite any same-named keys you set in the manifest).
  3. shell: true is stripped, with a Notice: removing 'shell: true' from build manifest line in the log. Use shell: true only for manual submissions, never for auto-submitted manifests.

The note attached to each build (used as the email/IRC subject line by post-build triggers) is generated from the commit. It looks like:

[abc1234][0] — [Author Name][1]

    First line of the commit message

[0]: https://<git-host>/~<owner>/<repo>/commit/<sha>
[1]: mailto:author@example.com

so the dashboard always tells you which commit and who made it, even before you click through.

#Push options

git push -o <option> lets you control behavior per-push without changing the manifest:

Option Effect
skip-ci Skip build submission entirely for this push. Push itself succeeds normally.
submit=<glob> Override the default manifest path glob (.build.yml,.builds/*.yml). Comma-separated, fnmatch(3) style.
debug Print the push UUID. Useful for support tickets.
description=<text> Set the repository's description.
visibility=public|unlisted|private Set the repository's visibility.
git push -o skip-ci
git push -o submit=".sourcehut/*.yml"
git push -o description="My cool project"

To make a push option permanent for a repo:

git config --add push.pushOption submit=".sourcehut/*.yml"

All push options (recognized or not) are also forwarded to repo webhooks, so custom tooling can hook into them.

#In-manifest submission control

The submitter: block in .build.yml lets the git.sr.ht push hook filter by ref at submission time, so non-matching pushes don't even create a build:

submitter:
  git.sr.ht:
    enabled: true          # optional, default true
    allow-refs:
      - refs/heads/master
      - refs/heads/main
      - "refs/tags/*"

Semantics as implemented in update-hook/submitter.go:

  • enabled: tri-state. Unset or true → permissive. false → skip immediately; allow-refs is not consulted. Writing enabled: true is redundant.
  • allow-refs: list of fnmatch(3)-style glob patterns matched against the full ref string (e.g. refs/heads/master, refs/tags/v1.0). Matched with no flags, so * does cross /. Absent/empty → no filtering; non-empty → submit only if at least one pattern matches.
  • The block is interpreted by the git.sr.ht push hook, not builds.sr.ht. Manual submission via hut builds submit, the web submit form, or a direct GraphQL submit ignores submitter: entirely — the runner does not re-evaluate it.
  • A hub.sr.ht: sibling (used by patchset auto-testing) is interpreted by hub.sr.ht's separate codebase, not the git push hook; the git hook silently drops unknown sub-keys when it re-serializes the manifest before sending it to builds.sr.ht.
  • The filter fires after the manifest is parsed from the tree but before the GraphQL submit mutation, so non-matching refs log a skip: ref X disabled by rule line and never count toward the 4-job push cap.

This is better than complete-build for known-at-push-time filters because it doesn't waste a VM boot.

#Built-in environment for git.sr.ht-submitted builds

  • GIT_REF — e.g. refs/heads/master, refs/tags/v1.0
  • BUILD_SUBMITTERgit.sr.ht

BUILD_REASON is not set by the git.sr.ht hook (only hub.sr.ht sets it, to patchset). Older notes claiming BUILD_REASON=git_push are wrong; check the env yourself: env | grep BUILD_.

Use these in tasks:

tasks:
  - check: |
      case "$GIT_REF" in
        refs/heads/main) ;;
        refs/tags/*) ;;
        *) complete-build ;;
      esac

#Mailing list patch testing

When a project is associated with a mailing list via the project hub, patches sent to the list are tested automatically.

Setup:

  1. Create a project on hub.sr.ht.
  2. Link both the repo and the list to the project.
  3. Patches with the subject prefix [PATCH <reponame>] get auto-tested against <reponame>.

The patch is applied on top of the target branch, builds run, and the result is posted back to the email thread as a reply.

Key behaviors:

  • Secrets are disabled for patchset-triggered builds. Don't rely on them.
  • Manifest used is the one in the target repo, not the patchset.
  • $BUILD_REASON=patchset, $PATCHSET_ID and $PATCHSET_URL are set.

Disable patch testing for a specific manifest:

submitter:
  hub.sr.ht:
    enabled: false

The default subject prefix for a repo can be set with git config format.subjectPrefix 'PATCH project-name'.

#hut CLI

hut is the de-facto SourceHut CLI (from ~xenrox). It uses OAuth tokens — interactive via hut init, or in builds via the ~/.config/hut/config that the worker provisions when the manifest has an oauth: directive. Full CI reference in references/hut.md.

Common commands:

# Submit a build from a manifest
hut builds submit .build.yml

# Submit and stream the log
hut builds submit --follow .build.yml

# Show a recent build
hut builds show <job-id>

# List your secrets (without revealing values)
hut builds secret list

# Cancel a job
hut builds cancel <job-id>

# Publish to pages
hut pages publish -d example.com site.tar.gz

# List your published sites
hut pages list

# Unpublish a site
hut pages unpublish -d example.com

hut init runs the OAuth flow for local use. In CI, no init is needed — the worker writes /home/build/.config/hut/config with the minted token and per-service origins as soon as the manifest has an oauth: directive (and secrets are enabled). See references/hut.md for the full mechanism, install instructions per image, and the scope-to-command map.

#Web submission

https://builds.sr.ht/submit — paste a manifest, click submit, watch it run. The fastest iteration loop for designing a manifest. No commit history pollution.

#GraphQL API

Endpoint: https://builds.sr.ht/query. Authentication via OAuth 2.0 bearer token.

Key mutations (user-callable):

  • submit($manifest: String!, $tags: [String!], $note: String, $secrets: Boolean, $execute: Boolean, $visibility: Visibility): Job! — submit a new job. execute: false keeps the job pending so you can createGroup it. Requires JOBS:RW.
  • start($jobID: Int!): Job — queue a pending job. Requires JOBS:RW.
  • cancel($jobId: Int!): Job — cancel a running/pending job. Requires JOBS:RW.
  • update($jobId: Int!, $visibility: Visibility!, $tags: [String!]): Job — change tags/visibility after submission.
  • createGroup($jobIds: [Int!]!, $triggers: [TriggerInput!], $execute: Boolean, $note: String): JobGroup! — group pending jobs with shared completion triggers.
  • startGroup($groupId: Int!): JobGroup — start a deferred job group.
  • shareSecret($uuid: String!, $user: String!): Secret! — share a secret with another user. Requires SECRETS:RW.

What's not user-callable: createArtifact, claim, updateJob, updateTask. These are decorated @worker in the schema and only accessible to the builds.sr.ht runner itself — there is no way for a user to upload an artifact post-hoc via the API. If artifacts: won't work for you, upload to your own storage or uploadArtifact (the git.sr.ht mutation, OBJECTS:RW) which attaches files to git tags.

The playground at https://builds.sr.ht/graphql is the easiest way to explore the live schema for your instance.

#Multi-image builds (the matrix substitute)

There's no matrix syntax. Use separate files in .builds/:

.builds/
├── alpine.yml
├── debian.yml
├── freebsd.yml
└── archlinux.yml

Each file is a standalone manifest. Up to 4 are submitted per push. Each runs independently as its own job; there's no "fan-in" job. To aggregate results, either:

  1. Group them after submission via the API — submit all 4 with execute: false, then createGroup them with a shared trigger, then startGroup. The trigger fires when all members finish.
  2. Use a finalizing job triggered by each — each job's triggers: includes a webhook to your own aggregator service.

In practice, separate independent jobs and reading the dashboard is what most projects do.

#Post-build triggers

Only two trigger actions exist in current builds.sr.ht: email and webhook. The Python manifest parser (buildsrht/manifest.py:TriggerAction) accepts only these enum values, and the Go worker (worker/triggers.go:ProcessTriggers) dispatches only these two — anything else logs Unknown trigger action 'X' and is skipped. Older docs and blog posts mention irc and job triggers; they were removed (or never landed) and do not work today.

#email

triggers:
  - action: email
    condition: failure
    to: "Me <me@example.com>"
    cc: "Other <them@example.com>"
    in_reply_to: "<some-message-id@example.com>"   # optional

Subject: [<tags>] build <status>. Body lists task statuses with ✓/✗ glyphs and a link to the job page. Sender is taken from instance config (builds.sr.ht::worker/trigger-from).

#webhook

triggers:
  - action: webhook
    condition: always
    url: https://example.com/sr-ht-hook

POSTs a JSON payload to url with Content-Type: application/json. Headers:

  • X-Payload-Nonce — random 8-byte nonce, hex.
  • X-Payload-Signature — Ed25519 signature of the body. The public key is published per-instance; on builds.sr.ht.org it's at https://meta.sr.ht/.well-known/webhook-pubkey and equivalents.

Body shape (matches the JobStatus struct in worker/triggers.go):

{
  "id": 1234567,
  "status": "success",
  "setup_log": "http://runner.example/logs/1234567/log",
  "note": "abc1234 — Author Name\n\n    Commit subject\n\n...",
  "runner": "runner.example",
  "owner": { "canonical_name": "~user", "name": "user" },
  "tasks": [
    { "name": "build", "status": "success",
      "log": "http://runner.example/logs/1234567/build/log" }
  ]
}

Client timeout is 10 seconds. The receiver's response body is read up to 2 KiB and copied into the build log, which is handy for debugging the receiver.

#Substitutes for irc and job

  • IRC notification: point action: webhook at an HTTP-to-IRC bridge you run (matterbridge, ZNC webhook plugin, a custom 30-line script). builds.sr.ht has never owned IRC-client connection state, and rewriting the trigger to do so isn't on the roadmap.
  • Chained deploy ("deploy after tests pass"):
    • Same-job final task: simplest — add a deploy: task after test:. Use submitter.allow-refs so it only runs on the right branch. Downside: oauth: is set once per job, so the deploy task sees the same broad scope.
    • Webhook → external submitter: have a small service receive the webhook trigger, then call hut builds submit deploy.yml (or the submit GraphQL mutation) to launch the deploy with a narrow oauth:.
    • Job groups (see below): submit both jobs with execute: false, group them with a shared completion trigger, then startGroup. This is what hub.sr.ht does for patchset auto-testing.

#Job groups

Job groups bundle multiple jobs with shared triggers that fire after every member finishes. Created via the GraphQL createGroup mutation:

mutation {
  createGroup(jobIds: [123, 124, 125], triggers: [
    { type: EMAIL, condition: FAILURE, email: { to: "me@example.com" } }
  ]) { id }
}

The TriggerInput schema accepts type: EMAIL and type: WEBHOOK, but the worker (worker/triggers.go:processJobGroupTriggers) only dispatches EMAIL — the webhook branch is a TODO comment. So group-level webhook triggers will log "Unknown trigger action" today; use per-job webhooks if you need that.

Typical setup:

  1. Submit each job with submit(manifest: ..., execute: false) → returns pending job IDs.
  2. createGroup(jobIds: [...], triggers: [...], execute: false) → returns the group ID.
  3. startGroup(groupId: ...) → all jobs start.

Practical uses:

  • Fan-out across architectures, get one summary email when the whole matrix finishes.
  • hub.sr.ht's patchset auto-tester: it groups all per-manifest jobs for a single patchset and attaches an EMAIL trigger that replies to the patchset's email thread with the aggregate result.

There's no manifest-level "this job is in a group" directive. Groups are formed by the API caller (often hut, hub.sr.ht, or your own script). A job can only belong to one group, and once startGroup runs the group is immutable.

#Tagging builds

When submitting via API, you can pass tags:

mutation {
  submit(manifest: "...", tags: ["nightly", "release-candidate"]) { id }
}

Tags become:

  • Filters in the dashboard: builds.sr.ht/~user/?search=tag:nightly
  • Status badges: a per-tag badge URL on the dashboard page

git.sr.ht-submitted builds don't auto-tag, but you can submit manually-tagged builds via hut:

hut builds submit --tags nightly,release-candidate .build.yml

#Things that look like features but aren't

Required status checks: there's no concept of "this build must pass before merge". SourceHut's contribution model is via patches on a mailing list, not PR merges. Build results post back to the patch thread; reviewers see them inline.

Status badges in the GitHub sense: there are badge images, but they're per-tag, not per-branch. Use a release tag to badge a stable status.

Workflow dispatch / manual builds: the closest equivalent is the web submit form. There's no "click to re-run with parameters" UI on the job page (you can resubmit a job, but the manifest stays identical).

Caching between builds: no built-in cache. Workarounds: rsync to a cache server you control, or commit caches to a separate repo.

Concurrent build limits: not exposed as a user-facing limit. Free accounts share worker capacity with everyone; if you have an unusual workload, the sourcehut admins email you about it (see the rizinorg/rizin GitHub issue for an example of what that conversation looks like).