codegen: localize wire.* upvalues per generated message function
Capture each _encode/_decode body, scan for wire.<name> refs, and
rewrite refs that appear >=2 times to bare locals with a
"local X = wire.X" prelude. Single-use refs stay as wire.X — without
the threshold the prelude TGETS outweighed the in-body saving on
sparse small-message decode.
Measured (median-of-3, shapes bench full mode):
- scalar-heavy: enc +38.5%, dec +39.2%
- packed-int32x100: enc +51.1%, dec +38.2%
- map-strxi32-*: enc +5.5%, dec +6-7%
- nested/oneof/wkt: +1-5% (within ~5% variance band)
- bench.lua Person small sizes: neutral
closes kot
beads: close 4ql (ibuf encoder not viable) + memory note on FFI boundary tax
Hand-coded single-pass backpatched Person_encode_ibuf wins by 1.7x at
10-100B but loses 1.4-2.7x at 1KB-100KB. Crossover at ~26 emails:
per-field ffi.copy boundaries scale linearly while Person_encode pays
one bulk table.concat memcpy regardless of count. Three ibuf attempts
total (per-byte alloc, two-pass reserve, single-pass backpatch), same
root cause each time. Not revivable without LuaJIT FFI sinking or a
cdata-string API contract.
beads: close bgu (ffi.string decode disproven) + memory note on buffer-reuse trap
Two-round microbench shows ffi.string(scratch + off, len) loses 2-2.7x
to buf:sub even with a pre-allocated stable scratch cdata and ffi.copy
amortized over 26 strings. The per-call pointer-arith cdata crosses
the ffi.string frame and can't be sunk — same root cause as a6n.
beads: close a6n + memory note on ffi.cast cdata allocation
a6n
beads: close ra6, 43t + drop deferred c0i dep from 43t
ra6 (generic C runtime codec): all 16 sub-tasks (3a-3l) plus rc8 strict-
decode parity bug closed. Umbrella issue closed as scope-complete.
43t (parity gate): closed after Justfile + bench.lua wiring. The c0i
dependency was removed first — c0i is deferred indefinitely per the
04c spike conclusion, so keeping it as a blocker would prevent 43t
from ever closing despite the parity work being complete.
Also added the c-runtime-parity-gate-43t persistent memory recording
the parity strategy and initial perf snapshot vs Lua-runtime.
bd-ra6, bd-43t
c_runtime: wire encode/decode dispatch + conformance-c harness
pb.encode / pb.decode lazy-compile desc.c_plan on first call and route
to pb.c_runtime.encode/decode when PB_ENABLE_C=1 loaded the module.
Eager compile at finalize_message time fails on codegen's
forward-declared descriptors — sub-messages don't have .fields yet —
so compilation is deferred until first encode/decode, by which time
the whole module table is populated and sub-plan chase resolves.
Full-mode codegen wrappers (M.<Type>_{encode,decode}) gain the same
lazy-compile prologue, bypassing the inline body when the C runtime
is loaded. Runtime-mode wrappers already call pb.encode and pick up
dispatch centrally.
Harness side: the conformance Docker image now installs tarantool-dev
+ build-essential so the C runtime can be built in-container; a new
`just conformance-c` recipe builds runtime/pb/c_runtime.so inside the
container, runs the suite with PB_ENABLE_C=1, then cleans the .so to
keep the bind mount free of foreign-platform binaries. The runner
script also pre-populates package.loaded.pb (avoids
.rocks/lib/tarantool/pb.so from starwing lua-protobuf masking ours)
and orders package.cpath by jit.os so mixed .dylib/.so trees from
host/container interleavings don't cross-load.
Verified:
- just test → 748 pass, 291 C-conditional skipped
- PB_ENABLE_C=1 just test → 1026/1039 pass; 13 fails are strict-decode
gaps in the C decoder (bd-rc8)
- just conformance-c → 2729/2806 binary suite pass; 77 unexpected
failures match the same gap categories
(illegal wire-type 6/7, field-num 0/over,
overlong tag varint, UTF-8 rejection,
message merge for oneof/repeated)
Strict-decode parity tracked in bd-rc8; this commit closes the wiring
half of bd-43t (conformance gate for C paths).
c_runtime: WKT override-hook passthrough (ra6 3k)
A plan whose descriptor carries desc.encode / desc.decode now
dispatches through those overrides instead of erroring. Top-level
encode/decode, sub-message fields, and map<,message> values all
check the override refs and call them with the same contract the
pure-Lua codec uses: encode(value) -> body bytes; decode(buf) ->
value. runtime/pb/wkt.lua is unmodified.
ra6 3k
c_runtime: unknown-fields capture + re-emission (ra6 3j)
c_runtime: map<K,V> encode + decode dispatch (ra6 3h)
Encode walks the user map with lua_next (the documented JIT exception
per CLAUDE.md — map hot paths can't avoid hash iteration). Each entry
goes into a stack-backed sub-buffer with synthetic tag(1,key) +
tag(2,value); proto3-elides default key and default value independently.
Map<,message> resolves its value sub-plan once and recurses through
encode_body. Decode reads the entry payload bounded, dispatches inner
id=1/id=2 (skipping anything else per spec), and lua_rawsets into a
lazy-created result map table; missing key or value defaults to the
proto3 zero. Reuses the existing list_stack_idx[] slot for the lazy
map cache since a field is either repeated or map, never both.
12 new tests cover round-trip for ages_by_nickname (string->int32),
nickname_by_age (int32->string), and addresses_by_label (string->
message), plus default-elision, multi-key correctness, empty maps,
and unknown-inner-id tolerance. Full suite 964/964 with PB_ENABLE_C=1.
bd-asz
c_runtime: oneof encode + decode dispatch (ra6 3g)
Encode resolves the active member per oneof group (last-non-nil wins in
declaration order via plan->oneofs[].member_indices), skips non-active
members, and force-emits the active branch so default values like
text="" still carry presence.
Decode clears sibling result-table entries after writing any field with
oneof_idx >= 0, mirroring codec.lua's oneof_siblings handling — wire
last-wins. Sibling-clear runs for both the scalar/string/enum/bytes arm
and the sub-message arm.
20 new tests in test/c_runtime_oneof_test.lua cover all three Result
branches across both codegen modes: byte-equal vs full.hello reference,
round-trip preservation of the active branch and absence of siblings,
default-value emission for active branches, and last-wins on both
encode (multi-branch input) and decode (multi-occurrence wire bytes).
Acceptance per bd-w3u
c_runtime: 64-bit cdata fidelity tests (ra6 3l)
Adds the c_int64.Wide fixture (one singular field per 64-bit kind)
plus an 18-test luatest group that round-trips each kind past 2^53
through the C runtime in both codegen modes. Confirms encode accepts
both LuaJIT int64_t/uint64_t cdata and Lua numbers, and decode
surfaces values >DBL_INT_MAX as cdata (matching msgpackffi /
net.box / box.tuple / built-in protobuf convention).
The C runtime already had the dispatch — to_int64_at / to_uint64_at
flow through luaL_toint64 / luaL_touint64 for cdata inputs, and
dec_push_one calls luaL_pushint64 / luaL_pushuint64 for every 64-bit
kind. This change pins the behavior under acceptance.
Closes tarantool-protobuf-awv (ra6 3l)
c_runtime: repeated string + repeated message acceptance at 1KB/10KB/100KB (ra6 3f)
The repeated string and repeated message (self-reference) dispatch
landed implicitly with ra6 3e — encode_repeated_field already routes
PB_KIND_MESSAGE through encode_submessage_field and unpacked
string/bytes through encode_one_field, and decode_body's cached
list_stack_idx[] handles every repeated element type. What was
missing was the formal 3f acceptance: explicit-size byte-equality at
1KB / 10KB / 100KB for Person.emails and Person.friends, against
mode=full.
Adds 4 acceptance tests (encode + decode × emails + friends), each
running both codegen modes. Sizing pins each case into a ±30% band
around the named target so a future schema/wire shift fails loudly
instead of silently drifting.
bd-exy
c_runtime: repeated + packed scalar encode/decode (ra6 3e)
Add repeated dispatch to the C-runtime encode/decode loop. Encode
side: encode_repeated_field walks Lua arrays via lua_objlen + per-
index rawgeti, dispatches on element kind. Packed numerics build
their payload in a stack-backed sub-buffer then emit `tag(LEN) +
varint(len) + body`; unpacked emit `tag + value` per element via
encode_one_field with force_emit=1 to bypass zero-suppression;
strings/bytes flow through the same path (never packable); repeated
messages reuse encode_submessage_field per element.
Decode side: per-field stack-slot cache (list_stack_idx[]) tied to
list_count[] avoids the per-element lua_getfield(result, name) round
trip the c-accel spike measured at 2x slower at 100KB. On first hit
for a repeated field we lua_createtable + write result[name] AND dup-
push the list onto the stack; subsequent hits lua_rawseti through the
cached absolute stack index. Lists stay valid across recursive sub-
message decodes because each child decode_body cleans up its own
scratch back to the caller's frame.
Packed/unpacked symmetry on read: a wt==LEN payload for any packable
scalar is decoded as a packed blob regardless of the schema's packed
flag, and a per-element-tagged stream is decoded element-by-element
even on a schema that defaults to packed — per proto3 reader rules.
New test/proto/c_repeated.proto fixture carries packed + explicit-
unpacked + repeated string/bytes + repeated message branches. The
encode and decode tests round-trip at 10/100/1000 elements per
branch. The two existing "skip repeated and map" marker tests
collapse to "skip map" — only map fields remain out of scope for
3e (bd-asz / 3h lands them next). 854 → 896 passing tests.
bd-jc9
c_runtime: encode/decode singular sub-messages (ra6 3d)
Refactor encode_lua/decode_lua into encode_body/decode_body so the
field-walk loop is callable recursively, then dispatch the MESSAGE
kind into a per-side sub-handler. Repeated and map fields still skip
at the field-walk level — 3e (jc9) and 3h (asz) land them next.
encode_submessage_field force-establishes the parent enc_buf's
heap_idx via a no-op ebuf_grow before recursing. Without that the
final ebuf_reserve on the parent could land its new userdata above
sub-encode's leaked stack slots, making the closing lua_settop drop
the parent's heap.
decode_submessage_field bounds the inner read by temporarily
shrinking c->len to the sub-message end offset; the wire-prim
helpers already bounds-check against c->len, so a malformed inner
payload can't over-read into the outer message's bytes.
New fixture test/proto/c_nested.proto carries a 5-level singular
chain (L1->L2->L3->L4->L5) for the depth test. The two existing
"skip message" tests are renamed to "skip repeated and map" — sub-
messages now encode and decode end-to-end.
bd-hwe
c_runtime: decode singular scalars (ra6 3c)
New entry pb.c_runtime.decode(plan, bytes) -> table, symmetric in
scope with 3b: int32/64, uint32/64, sint32/64, bool, fixed32/64,
sfixed32/64, float, double, enum, string, bytes. Repeated, map,
message-typed, and unknown tags skip by wire type — 3d/3e/3i/wyp
will extend later.
Result shape mirrors mode=full pure-Lua decode byte-for-byte: int64
family (int64/uint64/sint64/fixed64/sfixed64) push Tarantool cdata
via luaL_pushint64 / luaL_pushuint64; everything else lands as Lua
number/string/boolean. Pre-sized via lua_createtable(0, n_fields).
Field lookup is a linear scan over plan->fields by field_number; a
tag-keyed dispatch table is a future optimization.
Tests: 26 cases per codegen mode (full + runtime), covering the
bd-mz6 acceptance (round-trip of bd-y1n's Person payload matches
mode=full Person_decode shape-for-shape) plus per-kind coverage,
proto3-optional presence, fixed64 cdata, -0.0 sign preservation
(built via cdata to dodge LuaJIT literal-folding to +0.0),
skip-by-wire-type for out-of-scope shapes, unknown-tag skip, WKT
override rejection, and truncated input. Suite: 832/832 with
PB_ENABLE_C=1, 748/748 + 84 skipped without.
Closes bd-mz6
c_runtime: encode singular scalars (ra6 3b)
New entry pb.c_runtime.encode(plan, msg) -> string. Covers int32/64,
uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double,
enum (number or by_name lookup), string, bytes. Repeated, map, and
message-typed fields are silently skipped — those land in 3d/3e.
Buffer is a 4KB stack scratch promoted to a lua_newuserdata on
overflow, so a mid-encode luaL_error doesn't leak: the userdata is
GC'd at the unwind point.
Proto3 zero-suppression mirrors mode=full byte-for-byte: empty strings,
zero ints/fixed/enum skipped, +0.0 double skipped while -0.0 is emitted
via type-pun u64 equality (matches the `1/v == -math.huge` guard on
the Lua side). Proto3-optional fields bypass suppression. Plans with
desc.encode overrides (WKT) are rejected here — bd-rmf scope.
Tests: 16 cases per codegen mode (full + runtime), covering the bd-y1n
acceptance (Person {name='x', age=42, balance=-7,
user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full)
plus per-kind sweep and heap-grow path at 8KB. Suite: 806/806 with
PB_ENABLE_C=1, 748/748 + 58 skipped without.
Closes bd-y1n