c_runtime: encode singular scalars (ra6 3b)
New entry pb.c_runtime.encode(plan, msg) -> string. Covers int32/64,
uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double,
enum (number or by_name lookup), string, bytes. Repeated, map, and
message-typed fields are silently skipped — those land in 3d/3e.
Buffer is a 4KB stack scratch promoted to a lua_newuserdata on
overflow, so a mid-encode luaL_error doesn't leak: the userdata is
GC'd at the unwind point.
Proto3 zero-suppression mirrors mode=full byte-for-byte: empty strings,
zero ints/fixed/enum skipped, +0.0 double skipped while -0.0 is emitted
via type-pun u64 equality (matches the `1/v == -math.huge` guard on
the Lua side). Proto3-optional fields bypass suppression. Plans with
desc.encode overrides (WKT) are rejected here — bd-rmf scope.
Tests: 16 cases per codegen mode (full + runtime), covering the bd-y1n
acceptance (Person {name='x', age=42, balance=-7,
user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full)
plus per-kind sweep and heap-grow path at 8KB. Suite: 806/806 with
PB_ENABLE_C=1, 748/748 + 58 skipped without.
Closes bd-y1n
lsp: foundational types + annotate pb runtime entry + lazy views
Add lua-language-server scaffolding so editors and LLM assistants get
parameter / return types for the public pb runtime. Pure metadata --
no runtime behavior change. 748 tests still pass.
- .luarc.json at repo root: declare LuaJIT runtime, project require
paths (so require('full.hello.hello_pb') resolves), ignored
directories (.rocks, generated dylibs, bench/starwing third-party).
- runtime/pb/_types.lua (@meta): foundational @class declarations
for the descriptor contract (pb.Descriptor, pb.Field, pb.Enum-
Descriptor, pb.OneofDescriptor), the public pb module surface
(pb.Module), lazy view classes (pb.MessageView / pb.ArrayView /
pb.MapView), gRPC transport contract (pb.GrpcTransport plus
Service / Method descriptors), and the json / text / wire
sub-modules. Mirrors docs/codegen.md's descriptor shape.
- runtime/pb/init.lua: @type pb.Module annotation on the returned
table so hover on pb.encode / pb.decode_lazy / pb.parse picks up
the signatures from _types.lua.
- runtime/pb/lazy.lua: @class annotations on the three view locals
so lua-language-server merges them with the pb.MessageView /
pb.ArrayView / pb.MapView declarations in _types.lua.
Next phases: annotate codec/grpc/json (phase 2), emit per-message
@class blocks from the plugin codegen (phase 3).
bench(c_accel): drop defensive lua_type checks; refresh numbers
person_codec.c had a per-element lua_type(L,-1)==LUA_TSTRING check
inside the emails loop, plus field-level lua_type checks that
generic_codec.c skipped. At 100KB that's ~2700 extra C calls per
message in the hot path, which made S4 (hand-written) look slower
than S3 (generic) at 1KB+ and led to a wrong "branch prediction on
divergent paths" hypothesis in the original README.
Replaced the per-element check with no check and the field-level
lua_type checks with lua_isnil to match generic_codec.c's semantics.
S3 and S4 are now within +/-5% at every size, which strengthens the
ra6 architecture call (ship the generic one-call codec; codegen-
emitted per-message C buys nothing).
Also added bench/c_accel/compile_flags.txt so clangd resolves
<module.h> and friends -- mirrors runtime/pb/c/compile_flags.txt.
fix(wire): cap the Lua-number varint fast path at 2^51 (x86_64 corruption)
encode_varint had a fast path for Lua numbers in [2^28, 2^53) that emitted
each byte via bit.band(n, 0x7f) / math.floor(n / 128). bit.band routes
through LuaJIT's number->int32 conversion, which on x86_64 uses the
magic-number trick (add 2^52 + 2^51, read the low bits). That is exact only
while n + 2^52 + 2^51 < 2^53, i.e. n < 2^51; above it the addition rounds to
an even double and silently drops low bits, corrupting the varint.
arm64 LuaJIT uses an exact FP->int instruction, so the bug was invisible on
Apple-Silicon dev machines and only surfaced on x86_64 (a 64-bit lease ID in
tarantool-etcd round-tripped 3041234677171912 -> 3041234677171940 over gRPC,
breaking lease lookups). Cap the fast path at 2^51; values in [2^51, 2^53)
now fall through to the exact uint64 cdata loop.
Adds test/wire_varint_test.lua pinning the round-trip at the boundaries.
test: verify build dispatch after hook fix
test: trigger hook to debug build dispatch
test: explicit submit option
test: debug push hook output
test: verify auto-submit on push
c-accel: add compile_flags.txt for clangd
Without the Tarantool include path, clangd can't find <module.h> and
the entire file cascades into undefined-symbol diagnostics. The actual
make build is unaffected — only the editor experience.
List the four common Tarantool include paths (macOS Homebrew, the
Cellar-style symlink, /usr/local, /usr/include). Missing dirs are
silently ignored by the compiler, so a single file works for both
macOS and Linux.
c-accel: fix strdup on glibc with -std=c99
strdup is POSIX, not ISO C99, so glibc's <string.h> only exposes it when
_POSIX_C_SOURCE >= 200809. With -std=c99 (strict mode) gcc otherwise
treats strdup as an implicit-int function, which on 64-bit Linux
truncates the returned pointer to int and yields warnings + likely
crashes. macOS happens to declare strdup unconditionally so the issue
only surfaces on the srht.bigb.es Ubuntu builder.
Define _POSIX_C_SOURCE at the top of c_runtime.c before any include.
beads: sync mq7 closure to issues.jsonl
Passive export catch-up after closing tarantool-protobuf-mq7
(descriptor -> C plan compiler) — committed in 5c70555 but the .jsonl
export hadn't been refreshed.
ci: install Go 1.26 toolchain + cmake on builds.srht
Ubuntu noble's `golang` apt package is too old to satisfy go.mod's
toolchain directive — `go build` aborted with "toolchain not available".
Drop the apt package, install the official Go 1.26.3 tarball into
~/.local/go, and update go.mod / bench/go/go.mod accordingly. Also add
cmake + build-essential so `tt rocks install luatest` can build its
`checks` dependency.
c-accel: descriptor -> C plan compiler (bd-mq7)
First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.
What the plan carries (per docs/specs/c_accel_strategy.md):
* Per-field records: field_number, wire_type, kind, repeated/packed/
optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
sub_plan_idx (1-based into the sub-plans table), enum_ref
(luaL_ref for enum descriptor), oneof_idx back-pointer.
* Map fields capture map_key_kind, map_value_kind, and the value's
sub_plan_idx when the value is a message.
* Oneofs as a parallel array of {name, member_indices[]}, with
fields' oneof_idx pointing back to their group.
* WKT override detection: when desc.encode/desc.decode are set, the
plan flips has_override=1 and skips field-walk entirely.
* Extension range hooks captured (proto2 scaffolding for bd-3i).
* Field-name cache as a Lua table referenced via luaL_ref, so
encode/decode can do lua_rawgeti instead of re-interning C strings.
Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.
Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.
Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
- module surface + ABI version + KIND/WIRE constants
- Address: 4 fields, names, kinds, wire types, tag bytes, optional
- Person: 14 fields, scalars/enum/message/map/repeated/packed
- Sub-plan resolution + Person.friends self-reference cycle break
- Idempotent compile (second call returns cached plan)
- Result.outcome oneof: 3 members, oneof_idx back-pointers
- WKT Timestamp: has_override=true, field-walk skipped
Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).
Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.
Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
c-accel: arch prereqs — compat contract, C-side strategy, build scaffolding
Three companion specs under docs/specs/ formalize the boundaries
established in docs/c-accel.md, unblocking bd-mq7 (descriptor → C
plan compiler):
* c_accel_compat.md (bd-47e) — pinpoints what must stay byte-equal
between PB_ENABLE_C unset and =1: public surface, generated
module wrappers, 64-bit cdata, WKT shapes, unknown fields,
extensions, errors. Calls out the lazy-view exclusion.
* c_accel_strategy.md (bd-z7x) — pb_plan struct layout, field-name
luaL_ref caching, 4 KB stack-backed pb_buf, cached per-field
stack indices (the 2× win from spike Phase B), sub-buffer over
backpatching, map/oneof/unknown handling.
* c_accel_build_packaging.md (bd-wky) — where the C module lives
(runtime/pb/c/), how it builds, what the rockspec gains, the CI
matrix shape.
Scaffolding that lands now:
* runtime/pb/init.lua — PB_ENABLE_C=1 opt-in pcall hook; the
loaded module (or nil) is exposed as pb.c_runtime for
introspection. Silent fallback when the module is absent.
* Justfile — `build-c` / `clean-c` recipes (stub erroring cleanly
until bd-ra6 lands runtime/pb/c/), new lua_cpath constant,
LUA_CPATH wired through `test` and `test-one`.
* .builds/{pure-lua,c-enabled}.yml — sourcehut CI manifests, one
per activation mode (sourcehut has no matrix; parallel jobs go
in separate files). ubuntu/noble images.
* .sourcehut/conformance.yml — outside .builds/ so it doesn't
auto-submit; trigger manually with `hut builds submit` before
releases.
* .gitignore — runtime/pb/c_runtime.{so,dylib} and runtime/pb/c/*.o.
745/745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1
(silent fallback verified).
Closes bd-47e, bd-z7x, bd-wky. Unblocks bd-mq7.
docs: C acceleration architecture note (pf6 deliverable)
Records the architecture chosen for the C-accelerated encode/decode
path, based on the bench/c_accel/ spike results (commit d4dbd2f).
Closes the umbrella decision task pf6; the breakdown of follow-on
implementation work lives in Beads under bd-ra6 and its twelve
sub-issues (3a–3l).
Key decisions captured:
- Ship S3 (generic C runtime, descriptor-walking, one C call per
message) as bd-ra6. Within ±15% of hand-written codegen in the
spike, and beats it at scale on encode.
- Defer S4 (codegen-emitted C, bd-c0i) with explicit revival
criteria — ≤15% headroom doesn't earn codegen + maintenance +
distribution complexity.
- Drop S2 (per-primitive FFI) — structurally worse than pure Lua;
the FFI boundary cost (~60-75 ns/call) is the same order of
magnitude as the pure-Lua varint helpers it would replace.
User-facing contract:
- PB_ENABLE_C=1 env var is the only activation switch, default
off. require('pb') returns the same Lua surface in both modes;
zero behavior change for existing installs.
- No Lua-level toggle. Single knob, evaluated once at module load.
Parity verification:
- No separate Lua-vs-C diff harness. Existing test suites
(luatest, conformance, interop fixtures) run twice in CI — once
with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Parity is
guaranteed by transitivity through the reference outputs.
beads-tarantool-protobuf-pf6
tooling: add Beads workspace + beads agent skill
Checks in the portable parts of the Beads issue tracker that
this project now uses as its durable source of truth (per the
PLAN.md retirement). The Dolt DB internals (.beads/embeddeddolt/,
.beads/backup/, runtime sockets/locks/sync-state) stay
gitignored by .beads/.gitignore; what lands here are:
.beads/.gitignore — Beads-managed ignore rules
.beads/README.md — workspace README from bd init
.beads/config.yaml — workspace config (id, etc.)
.beads/metadata.json — workspace metadata
.beads/issues.jsonl — passive JSONL export of issues
.beads/interactions.jsonl — passive log of state changes
The JSONL files are the human-readable / diffable surface; the
authoritative store remains the Dolt DB synced via refs/dolt/data
on the git remote (see Beads SYNC_CONCEPTS.md).
Also adds .agents/skills/beads/ — the beads agent skill that
tells subagents to use `bd` for task tracking instead of local
TODO files. Mirrors the rule already in CLAUDE.md but in a form
other agents and IDEs can pick up.
Closes the implementation side of bd-2nl (which removed the
.git/info/exclude rule that previously hid .beads/ from Git).
bench: alloc decomposition probe
Adds bench/alloc_probe.lua, a surgical probe that strips the
encode path apart and measures KB-delta per primitive operation.
Lets us attribute the ~136 B/op encode floor (captured in the
perf-analysis-2026-05-18 memo) to a specific source: result
table, intermediate varint strings, output string, etc.
Runs with GC stopped so allocations accumulate; divides by N to
get bytes-per-op. Uses unique input bytes per iteration where
appropriate to eliminate string-interning noise.
Complements bench/profile.lua (where time goes) with where
*memory* goes — the two together informed which bd perf items
were worth landing (the bytes-saved had to translate to a real
allocation rate change, not just a hot-line attribution).
bench: jit.p profile driver for hot encode/decode paths
Adds bench/profile.lua, a one-shot driver around LuaJIT's
sampling profiler (jit.p). Runs Person_encode and Person_decode
against the 1 KB fixture — the size where decode MB/s halves and
encode MB/s stops climbing — and prints a function-and-line
breakdown of where wall time actually goes.
Used during the M5 perf-investigation arc to produce the
attributions captured in the jit-p-profile-2026-05-18 memo and
referenced by several bd issues (h8v, u39, gcy, 4kj, aah, bgu).
`tarantool bench/profile.lua` runs both passes; `encode` /
`decode` arguments limit to one phase. Sample rate is 4 ms and
the loop runs 200 000 iterations so each phase produces ~500
samples — enough to attribute single-percent line cost.
bench: starwing cross-runtime comparison harness
Adds bench/starwing_bench.lua, a sibling to bench/bench.lua that
runs the same Person and BenchPayload payloads through Starwing's
lua-protobuf (installed as a rock at .rocks/lib/tarantool/pb.so)
instead of our pure-Lua pb runtime. Same payload shapes, same
iteration counts, same measurement loop — so the two outputs
line up column-for-column.
Loading approach: strip runtime/ off package.path so the `pb`
module name resolves to starwing's C module. We use
package.loadlib to grab `luaopen_pb` from the .so and inject
into package.loaded before require('pb') has a chance to hit the
Lua-path loader; starwing's pb.option('int64_as_number') matches
the regime our bench/bench.lua exercises (bench payloads stay in
the Lua number range, avoiding per-call cdata allocation noise).
bench/starwing/{hello.pb,proto2_basic.pb} are the FileDescriptor
binaries starwing's pb.load() consumes; produced by mainline
protoc --descriptor_set_out from the same .proto sources we
generate Lua from.
Mirrors bench/go/'s role for Go (`bench: Go cross-runtime
comparison harness`, 07eef11) but for the in-Tarantool C-module
alternative — answers "how do we compare to the C-module
incumbent on the same host" rather than "how do we compare to
mainline C++/Go on a different host."