test: trigger hook to debug build dispatch
test: explicit submit option
test: debug push hook output
test: verify auto-submit on push
c-accel: add compile_flags.txt for clangd
Without the Tarantool include path, clangd can't find <module.h> and
the entire file cascades into undefined-symbol diagnostics. The actual
make build is unaffected — only the editor experience.
List the four common Tarantool include paths (macOS Homebrew, the
Cellar-style symlink, /usr/local, /usr/include). Missing dirs are
silently ignored by the compiler, so a single file works for both
macOS and Linux.
c-accel: fix strdup on glibc with -std=c99
strdup is POSIX, not ISO C99, so glibc's <string.h> only exposes it when
_POSIX_C_SOURCE >= 200809. With -std=c99 (strict mode) gcc otherwise
treats strdup as an implicit-int function, which on 64-bit Linux
truncates the returned pointer to int and yields warnings + likely
crashes. macOS happens to declare strdup unconditionally so the issue
only surfaces on the srht.bigb.es Ubuntu builder.
Define _POSIX_C_SOURCE at the top of c_runtime.c before any include.
beads: sync mq7 closure to issues.jsonl
Passive export catch-up after closing tarantool-protobuf-mq7
(descriptor -> C plan compiler) — committed in 5c70555 but the .jsonl
export hadn't been refreshed.
ci: install Go 1.26 toolchain + cmake on builds.srht
Ubuntu noble's `golang` apt package is too old to satisfy go.mod's
toolchain directive — `go build` aborted with "toolchain not available".
Drop the apt package, install the official Go 1.26.3 tarball into
~/.local/go, and update go.mod / bench/go/go.mod accordingly. Also add
cmake + build-essential so `tt rocks install luatest` can build its
`checks` dependency.
c-accel: descriptor -> C plan compiler (bd-mq7)
First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.
What the plan carries (per docs/specs/c_accel_strategy.md):
* Per-field records: field_number, wire_type, kind, repeated/packed/
optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
sub_plan_idx (1-based into the sub-plans table), enum_ref
(luaL_ref for enum descriptor), oneof_idx back-pointer.
* Map fields capture map_key_kind, map_value_kind, and the value's
sub_plan_idx when the value is a message.
* Oneofs as a parallel array of {name, member_indices[]}, with
fields' oneof_idx pointing back to their group.
* WKT override detection: when desc.encode/desc.decode are set, the
plan flips has_override=1 and skips field-walk entirely.
* Extension range hooks captured (proto2 scaffolding for bd-3i).
* Field-name cache as a Lua table referenced via luaL_ref, so
encode/decode can do lua_rawgeti instead of re-interning C strings.
Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.
Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.
Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
- module surface + ABI version + KIND/WIRE constants
- Address: 4 fields, names, kinds, wire types, tag bytes, optional
- Person: 14 fields, scalars/enum/message/map/repeated/packed
- Sub-plan resolution + Person.friends self-reference cycle break
- Idempotent compile (second call returns cached plan)
- Result.outcome oneof: 3 members, oneof_idx back-pointers
- WKT Timestamp: has_override=true, field-walk skipped
Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).
Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.
Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
c-accel: arch prereqs — compat contract, C-side strategy, build scaffolding
Three companion specs under docs/specs/ formalize the boundaries
established in docs/c-accel.md, unblocking bd-mq7 (descriptor → C
plan compiler):
* c_accel_compat.md (bd-47e) — pinpoints what must stay byte-equal
between PB_ENABLE_C unset and =1: public surface, generated
module wrappers, 64-bit cdata, WKT shapes, unknown fields,
extensions, errors. Calls out the lazy-view exclusion.
* c_accel_strategy.md (bd-z7x) — pb_plan struct layout, field-name
luaL_ref caching, 4 KB stack-backed pb_buf, cached per-field
stack indices (the 2× win from spike Phase B), sub-buffer over
backpatching, map/oneof/unknown handling.
* c_accel_build_packaging.md (bd-wky) — where the C module lives
(runtime/pb/c/), how it builds, what the rockspec gains, the CI
matrix shape.
Scaffolding that lands now:
* runtime/pb/init.lua — PB_ENABLE_C=1 opt-in pcall hook; the
loaded module (or nil) is exposed as pb.c_runtime for
introspection. Silent fallback when the module is absent.
* Justfile — `build-c` / `clean-c` recipes (stub erroring cleanly
until bd-ra6 lands runtime/pb/c/), new lua_cpath constant,
LUA_CPATH wired through `test` and `test-one`.
* .builds/{pure-lua,c-enabled}.yml — sourcehut CI manifests, one
per activation mode (sourcehut has no matrix; parallel jobs go
in separate files). ubuntu/noble images.
* .sourcehut/conformance.yml — outside .builds/ so it doesn't
auto-submit; trigger manually with `hut builds submit` before
releases.
* .gitignore — runtime/pb/c_runtime.{so,dylib} and runtime/pb/c/*.o.
745/745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1
(silent fallback verified).
Closes bd-47e, bd-z7x, bd-wky. Unblocks bd-mq7.
docs: C acceleration architecture note (pf6 deliverable)
Records the architecture chosen for the C-accelerated encode/decode
path, based on the bench/c_accel/ spike results (commit d4dbd2f).
Closes the umbrella decision task pf6; the breakdown of follow-on
implementation work lives in Beads under bd-ra6 and its twelve
sub-issues (3a–3l).
Key decisions captured:
- Ship S3 (generic C runtime, descriptor-walking, one C call per
message) as bd-ra6. Within ±15% of hand-written codegen in the
spike, and beats it at scale on encode.
- Defer S4 (codegen-emitted C, bd-c0i) with explicit revival
criteria — ≤15% headroom doesn't earn codegen + maintenance +
distribution complexity.
- Drop S2 (per-primitive FFI) — structurally worse than pure Lua;
the FFI boundary cost (~60-75 ns/call) is the same order of
magnitude as the pure-Lua varint helpers it would replace.
User-facing contract:
- PB_ENABLE_C=1 env var is the only activation switch, default
off. require('pb') returns the same Lua surface in both modes;
zero behavior change for existing installs.
- No Lua-level toggle. Single knob, evaluated once at module load.
Parity verification:
- No separate Lua-vs-C diff harness. Existing test suites
(luatest, conformance, interop fixtures) run twice in CI — once
with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Parity is
guaranteed by transitivity through the reference outputs.
beads-tarantool-protobuf-pf6
tooling: add Beads workspace + beads agent skill
Checks in the portable parts of the Beads issue tracker that
this project now uses as its durable source of truth (per the
PLAN.md retirement). The Dolt DB internals (.beads/embeddeddolt/,
.beads/backup/, runtime sockets/locks/sync-state) stay
gitignored by .beads/.gitignore; what lands here are:
.beads/.gitignore — Beads-managed ignore rules
.beads/README.md — workspace README from bd init
.beads/config.yaml — workspace config (id, etc.)
.beads/metadata.json — workspace metadata
.beads/issues.jsonl — passive JSONL export of issues
.beads/interactions.jsonl — passive log of state changes
The JSONL files are the human-readable / diffable surface; the
authoritative store remains the Dolt DB synced via refs/dolt/data
on the git remote (see Beads SYNC_CONCEPTS.md).
Also adds .agents/skills/beads/ — the beads agent skill that
tells subagents to use `bd` for task tracking instead of local
TODO files. Mirrors the rule already in CLAUDE.md but in a form
other agents and IDEs can pick up.
Closes the implementation side of bd-2nl (which removed the
.git/info/exclude rule that previously hid .beads/ from Git).
bench: alloc decomposition probe
Adds bench/alloc_probe.lua, a surgical probe that strips the
encode path apart and measures KB-delta per primitive operation.
Lets us attribute the ~136 B/op encode floor (captured in the
perf-analysis-2026-05-18 memo) to a specific source: result
table, intermediate varint strings, output string, etc.
Runs with GC stopped so allocations accumulate; divides by N to
get bytes-per-op. Uses unique input bytes per iteration where
appropriate to eliminate string-interning noise.
Complements bench/profile.lua (where time goes) with where
*memory* goes — the two together informed which bd perf items
were worth landing (the bytes-saved had to translate to a real
allocation rate change, not just a hot-line attribution).
bench: jit.p profile driver for hot encode/decode paths
Adds bench/profile.lua, a one-shot driver around LuaJIT's
sampling profiler (jit.p). Runs Person_encode and Person_decode
against the 1 KB fixture — the size where decode MB/s halves and
encode MB/s stops climbing — and prints a function-and-line
breakdown of where wall time actually goes.
Used during the M5 perf-investigation arc to produce the
attributions captured in the jit-p-profile-2026-05-18 memo and
referenced by several bd issues (h8v, u39, gcy, 4kj, aah, bgu).
`tarantool bench/profile.lua` runs both passes; `encode` /
`decode` arguments limit to one phase. Sample rate is 4 ms and
the loop runs 200 000 iterations so each phase produces ~500
samples — enough to attribute single-percent line cost.
bench: starwing cross-runtime comparison harness
Adds bench/starwing_bench.lua, a sibling to bench/bench.lua that
runs the same Person and BenchPayload payloads through Starwing's
lua-protobuf (installed as a rock at .rocks/lib/tarantool/pb.so)
instead of our pure-Lua pb runtime. Same payload shapes, same
iteration counts, same measurement loop — so the two outputs
line up column-for-column.
Loading approach: strip runtime/ off package.path so the `pb`
module name resolves to starwing's C module. We use
package.loadlib to grab `luaopen_pb` from the .so and inject
into package.loaded before require('pb') has a chance to hit the
Lua-path loader; starwing's pb.option('int64_as_number') matches
the regime our bench/bench.lua exercises (bench payloads stay in
the Lua number range, avoiding per-call cdata allocation noise).
bench/starwing/{hello.pb,proto2_basic.pb} are the FileDescriptor
binaries starwing's pb.load() consumes; produced by mainline
protoc --descriptor_set_out from the same .proto sources we
generate Lua from.
Mirrors bench/go/'s role for Go (`bench: Go cross-runtime
comparison harness`, 07eef11) but for the in-Tarantool C-module
alternative — answers "how do we compare to the C-module
incumbent on the same host" rather than "how do we compare to
mainline C++/Go on a different host."
bench/jit_trace: harden mcode arena + jit.off the listener
Two infra fixes for the trace-stability gate, both reproduced on
Tarantool 3.8.0-entrypoint / LuaJIT 2.1.0-beta3 / macOS arm64.
1. Default JIT mcode arena (sizemcode=32K, maxmcode=512K) is too
small for our hot-path codegen footprint. Roughly 1 in 10 runs
the gate reports every check as 'stops=0' with no diagnostic —
indistinguishable from a real JIT topology regression. jit.v
shows 'failed to allocate mcode memory at hello_pb.lua:890'
(the packed lucky_numbers varint loop). Calling
jit.opt.start('sizemcode=64','maxmcode=4096') at the top of
the gate gives the arena enough headroom to hold both encoder
and decoder bodies for the proto3 + proto2 fixtures.
2. The trace listener callback itself can become hot enough to
be JIT-compiled, which then races with the function being
recorded (recording-while-recording). Reproduced with a fat
cb that appends event tuples to a table: starts go up but
stops drop to ~0. Calling jit.off(cb) on the listener
prevents this. The current cb body happens to dodge this
because its branches keep call sites polymorphic — but it's
fragile; any future extension (per-event timing, pc context)
would re-trigger the bug. Adds the jit.off as defense.
Verified 20/20 consecutive runs now report 37/37 passing (was
intermittently 0/37 before). Also drops a now-stale "(PLAN.M6)"
reference from the header comment.
beads-tarantool-protobuf-3o2
docs: retire PLAN.md in favor of Beads issue tracking
Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.
References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.
The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.
Closes the goal of bd-8k2.
bench: C-acceleration spike — measure four Lua↔C boundaries
Adds bench/c_accel/ with hello.Person codecs at four boundary
strategies and a harness comparing them across 10B–100KB:
S1 pure Lua (full mode) — existing baseline
S2 prim.c + prim_ffi.lua — per-primitive FFI calls
S3 generic_codec.c — one C call per message, descriptor-walking
S4 person_codec.c — hand-written, no dispatch, upper bound
Plus ffi_probe.lua, a microbenchmark decomposing FFI call cost.
Headline numbers (×L = speedup vs pure-Lua full mode):
ENCODE S2 S3 S4
10B 0.99 3.29 3.79
100B 0.93 3.12 3.46
1KB 0.82 5.33 4.94
10KB 0.60 3.53 2.86
100KB 0.63 3.49 2.85
DECODE S2 S3 S4
10B 0.29 2.59 2.54
100B 0.33 2.94 2.95
1KB 0.39 7.22 7.62
10KB 0.38 9.64 10.23
100KB 0.37 10.87 11.33
S3 lands within ±15% of S4 at every size and beats it on encode
at 1KB+: the descriptor-walking loop is uniformly branch-predictable;
hand-written has more divergent per-field paths.
S2 loses to pure Lua at every size ≥1KB on encode and at every size
on decode. FFI cost decomposition (ffi_probe.lua):
bare FFI call into ffi.load lib 33 ns
pointer-returning FFI 73 ns
+ v_out[0] read + tonumber 118 ns
ffi.C.memcmp (for comparison) 60 ns
ffi.cast(const uint8_t*, str) 156 ns
ffi.string(p, 32) 23 ns
pure-Lua varint decode 75 ns
read_varint in tight traced loop 59 ns (floor)
Per-call FFI dispatch into a ffi.load'd lib is ~60–75 ns — the
same order of magnitude as pure-Lua varint decode. The "win" from
going to C only materializes when you cross the boundary ONCE per
message, not per primitive.
C encode plateaus at ~2.5–2.9 GB/s from 1KB upward; bottleneck
moves to Lua table reads and output string allocation. C decode
hits 2.5 GB/s at 100KB, while pure-Lua decode hits a per-byte
cliff (158 k msg/s @ 1KB → 2.3 k @ 100KB).
Architecture implication for pf6: the generic C runtime (ra6) gets
the full perf envelope; codegen-emitted per-message C (c0i) earns
≤15% headroom and goes the wrong way at scale. Per-primitive FFI
is a non-starter.
Required impl pattern in any future ra6: cache per-field stack
indices for repeated/packed arrays for the duration of
decode_message — the naive lazy-getfield variant was 2× slower
than hand-written at 100KB (see bench/c_accel/README.md).
beads-tarantool-protobuf-04c
bench: record u39 (table.new pre-sized result) post-mortem
Attempted emitting table_new(0, N) for decode result tables. Tests
passed, JIT passed, but small-payload decode (10B, 100B) regressed
15-18% because the table.new upvalue-call cost exceeds any rehash
savings at that scale, and the small case never reaches the first
rehash anyway. Large-payload decode flat. Confirms drm's claim that
alloc shape is not the bottleneck on the bench corpus.
Code change reverted; only the writeup lands.
beads-tarantool-protobuf-u39
codegen: local counter per repeated field on decode
Replaces `list[#list + 1] = val` with `_n_<fname> = _n_<fname> + 1;
list[_n_<fname>] = val` in every generated M.X_decode repeated-field
append site. One counter local per repeated non-map field, declared at
function entry. Counters survive across loop iterations so out-of-order
wire entries for the same field continue past the existing length
without re-scanning.
Profile attributed 6.2% of hello.Person 1KB decode to the `#list + 1`
re-traversal (26-email Person paid 26 list scans per decode).
Bench (Person full decode, msgs/s, median-of-3 vs post-4kj):
1KB +5.0%, 10KB +5.0%, 100KB +7.1%. Encode flat. Tests 745/745.
JIT 37/37, 0 bridges.
beads-tarantool-protobuf-cch