codegen: localize wire.* upvalues per generated message function
Capture each _encode/_decode body, scan for wire.<name> refs, and
rewrite refs that appear >=2 times to bare locals with a
"local X = wire.X" prelude. Single-use refs stay as wire.X — without
the threshold the prelude TGETS outweighed the in-body saving on
sparse small-message decode.
Measured (median-of-3, shapes bench full mode):
- scalar-heavy: enc +38.5%, dec +39.2%
- packed-int32x100: enc +51.1%, dec +38.2%
- map-strxi32-*: enc +5.5%, dec +6-7%
- nested/oneof/wkt: +1-5% (within ~5% variance band)
- bench.lua Person small sizes: neutral
closes kot
beads: close 4ql (ibuf encoder not viable) + memory note on FFI boundary tax
Hand-coded single-pass backpatched Person_encode_ibuf wins by 1.7x at
10-100B but loses 1.4-2.7x at 1KB-100KB. Crossover at ~26 emails:
per-field ffi.copy boundaries scale linearly while Person_encode pays
one bulk table.concat memcpy regardless of count. Three ibuf attempts
total (per-byte alloc, two-pass reserve, single-pass backpatch), same
root cause each time. Not revivable without LuaJIT FFI sinking or a
cdata-string API contract.
beads: close bgu (ffi.string decode disproven) + memory note on buffer-reuse trap
Two-round microbench shows ffi.string(scratch + off, len) loses 2-2.7x
to buf:sub even with a pre-allocated stable scratch cdata and ffi.copy
amortized over 26 strings. The per-call pointer-arith cdata crosses
the ffi.string frame and can't be sunk — same root cause as a6n.
beads: close a6n + memory note on ffi.cast cdata allocation
a6n
codegen: inline 1-byte LEN fast path for string/bytes decode
Singular and repeated string/bytes fields in generated full-mode _decode
now read the length byte and dispatch in-line instead of calling
wire.decode_string / wire.decode_bytes. For length < 128 (the common
short-string RPC case), the path stays inside the parent JIT trace:
no child-trace stitch, no per-element function frame, utf8_len lookup
hoisted to a generated-file upvalue.
The original a6n design — one ffi.cast(U8CP, buf) at the top of each
_decode — was abandoned: the cdata wrapper is 24 bytes per call and
LuaJIT can't sink the allocation because the pointer local lives
across wire.decode_* call frames. Net regression at small sizes
(+11-16%) overwhelmed the single-byte-read savings.
Bench (hello.Person full-mode decode):
size ns/op before ns/op after delta
10B 419 390 -6.9%
100B 515 478 -7.2%
1KB 9004 8042 -10.7%
10KB 64644 56745 -12.2%
100KB 625638 537750 -14.0%
Zero allocation impact across all sizes. Tests: 1043 pass, 37 JIT
trace gates pass. (a6n)
lsp: annotate codec / grpc / json / wkt public surface (74c)
Phase-2 of the LSP / LLM affordance work (phase-1 landed annotations
on init.lua + lazy.lua). Adds ---@param / ---@return on the
public-surface entry points so editors and LLM assistants see typed
signatures on hover.
- codec.lua: encode_message, decode_message, encode_field,
compile_writers, compile_readers, merge_message.
- grpc.lua: loopback, multiplex, new_stream_pair, wrap_call,
wrap_server_stream, wrap_server_view.
- json.lua: M.encode, M.decode. New pb.JsonEncodeOpts and
pb.JsonDecodeOpts @class blocks in _types.lua document the opts
fields the implementation actually consults (use_proto_names,
emit_defaults / always_emit_zero_value alias, indent;
ignore_unknown_fields on decode).
- wkt.lua: register, lookup, any_pack, any_unpack. New
pb.AnyMessage @class for the {type_url, value} shape.
Also corrects two pre-existing signature lies in _types.lua:
- pb.register is `(desc): pb.Descriptor`, not `(full_name, desc)` —
the implementation has always derived the key from desc.name and
all callers pass a single arg.
- pb.any.pack is `(desc, t, type_url_prefix?)`, not `(t, type_url)`.
Pure metadata — `just test` stays at 752 and `just test-c` at 1043.
bd-74c
beads: close ra6, 43t + drop deferred c0i dep from 43t
ra6 (generic C runtime codec): all 16 sub-tasks (3a-3l) plus rc8 strict-
decode parity bug closed. Umbrella issue closed as scope-complete.
43t (parity gate): closed after Justfile + bench.lua wiring. The c0i
dependency was removed first — c0i is deferred indefinitely per the
04c spike conclusion, so keeping it as a blocker would prevent 43t
from ever closing despite the parity work being complete.
Also added the c-runtime-parity-gate-43t persistent memory recording
the parity strategy and initial perf snapshot vs Lua-runtime.
bd-ra6, bd-43t
c_runtime: parity gate — test-c / test-all / bench-c (43t)
Adds the C-acceleration parity gate that bd-43t mandates: every test
asserts against a reference output (golden bytes, txtpb, conformance
result), so passing the same suite under both `PB_ENABLE_C=1` and the
default Lua codec proves Lua ≡ C by transitivity. No separate diff
harness needed.
Justfile recipes:
- `test-c` — same luatest suite with PB_ENABLE_C=1 (1043 tests;
unlocks the c_runtime_* groups via build-c)
- `test-all` — `test` + `test-c` (parity gate)
- `bench-c` — bench.lua under PB_ENABLE_C=1; runtime column is
relabelled `c-runtime`. Also tidies build-c — the stale
"directory doesn't exist yet" branch goes away now that
ra6 has landed.
bench.lua: extends package.cpath so `require('pb.c_runtime')` finds
runtime/pb/c_runtime.{so,dylib} when invoked outside the Justfile.
Detects `PB_ENABLE_C=1`, renames `runtime` → `c-runtime` in the
output, and refuses --baseline / --compare (alloc shape differs
between codecs by design — would noise the gate).
README: documents the parity strategy and lists `test-all`,
`conformance-c`, `bench-c` as the dual-codec entry points.
Scope notes: c-generated column dropped (c0i is deferred per
docs/c-accel.md); starwing column already lives in
bench/starwing_bench.lua.
bd-43t
c_runtime: strict-decode parity with pure-Lua codec (rc8)
The C decoder accepted wire types 6/7, field number 0, field numbers
beyond 29 bits, overlong tag varints, invalid UTF-8 in string fields,
and unmatched proto2 SGROUPs — each of which the pure-Lua decoder
already rejected. It also replaced (instead of merged) when a singular
message field appeared more than once on the wire, dropping
sub-message scalars from the prior occurrence.
decode_body: reject wt 6/7 / field 0 / fn > 2^29-1 / overlong tag
varint up front; track egroup_seen so a group body that runs off the
end of the buffer fails loudly. dec_push_kind STRING: port wire.lua's
RFC 3629 validator (utf8.len equivalent) — covers singular, repeated,
oneof, map-key, and map-value via the same code path. Singular message
dispatch in decode_body and decode_extension_into: look up an existing
prev table at result[name] and merge via the new
merge_subresult_into, which ports codec.lua's merge_message (recursive
sub-message, repeated concat, map last-wins per key, skip for WKT
custom-decode).
Regression coverage in test/conformance_test.lua's conformance.core
group: pre-existing tests already pinned the wire-type, tag, UTF-8
(singular/repeated/oneof), and merge cases — those now also exercise
the C path under PB_ENABLE_C=1. Two gaps remained — map-key/value
UTF-8 and proto2 group balancing — both covered now via four new
tests, with proto2 routed through a TestAllTypesProto2 helper.
PB_ENABLE_C=1 just test: 1043/1043 (baseline 1039 + 4 new).
PB_ENABLE_C=1 just conformance-c: 0 unexpected failures (was 77).
just conformance (pure-Lua path): unchanged — no regression.
Closes rc8
c_runtime: wire encode/decode dispatch + conformance-c harness
pb.encode / pb.decode lazy-compile desc.c_plan on first call and route
to pb.c_runtime.encode/decode when PB_ENABLE_C=1 loaded the module.
Eager compile at finalize_message time fails on codegen's
forward-declared descriptors — sub-messages don't have .fields yet —
so compilation is deferred until first encode/decode, by which time
the whole module table is populated and sub-plan chase resolves.
Full-mode codegen wrappers (M.<Type>_{encode,decode}) gain the same
lazy-compile prologue, bypassing the inline body when the C runtime
is loaded. Runtime-mode wrappers already call pb.encode and pick up
dispatch centrally.
Harness side: the conformance Docker image now installs tarantool-dev
+ build-essential so the C runtime can be built in-container; a new
`just conformance-c` recipe builds runtime/pb/c_runtime.so inside the
container, runs the suite with PB_ENABLE_C=1, then cleans the .so to
keep the bind mount free of foreign-platform binaries. The runner
script also pre-populates package.loaded.pb (avoids
.rocks/lib/tarantool/pb.so from starwing lua-protobuf masking ours)
and orders package.cpath by jit.os so mixed .dylib/.so trees from
host/container interleavings don't cross-load.
Verified:
- just test → 748 pass, 291 C-conditional skipped
- PB_ENABLE_C=1 just test → 1026/1039 pass; 13 fails are strict-decode
gaps in the C decoder (bd-rc8)
- just conformance-c → 2729/2806 binary suite pass; 77 unexpected
failures match the same gap categories
(illegal wire-type 6/7, field-num 0/over,
overlong tag varint, UTF-8 rejection,
message merge for oneof/repeated)
Strict-decode parity tracked in bd-rc8; this commit closes the wiring
half of bd-43t (conformance gate for C paths).
c_runtime: proto2 — required, defaults, groups, extensions (ra6 3i)
Plumb proto2 semantic surface through the C plan and codec:
* required: encode-time `required field missing` error with full path
(`<msg>.<field>`); set scalars/enums force-emit so zero still reaches
the wire (matches build_required_writer).
* groups: kind='group' compiles to PB_KIND_MESSAGE with is_group=1; tag
uses SGROUP and a pre-encoded EGROUP closer; encode walks regular
body bytes into a sub-buf and brackets with SGROUP+body+EGROUP (no
length prefix). Decode adds a stop_group_id to decode_body so the
inner walk terminates on the matching EGROUP, with id-mismatch as a
hard error per spec. Unknown-tag skip (`dec_skip_with_id`) recurses
through SGROUP bodies to the matching EGROUP.
* extensions: walk `desc.extensions_list` at plan-compile time, cache
each ext's full_name; encode iterates the cached array and emits any
present in `data._extensions[full_name]`; decode probes unregistered
tags against `plan->extensions` before falling through to
`_unknown_fields`, routing matched bytes into result._extensions.
Defaults: presence-tracked optional fields stay nil-on-absent in the
decoded table; the descriptor's `default_value` is surfaced for callers
(JSON, text) but never auto-materialized at decode — same as codec.lua.
36 new tests in test/c_runtime_proto2_test.lua exercise required missing
+ zero-emit, presence-tracked defaults, singular/repeated groups, and
extension round-trip — each asserts byte-equality with mode=full pure-Lua
output across both codegen modes.
ra6 3i
c_runtime: WKT override-hook passthrough (ra6 3k)
A plan whose descriptor carries desc.encode / desc.decode now
dispatches through those overrides instead of erroring. Top-level
encode/decode, sub-message fields, and map<,message> values all
check the override refs and call them with the same contract the
pure-Lua codec uses: encode(value) -> body bytes; decode(buf) ->
value. runtime/pb/wkt.lua is unmodified.
ra6 3k
c_runtime: unknown-fields capture + re-emission (ra6 3j)
c_runtime: map<K,V> encode + decode dispatch (ra6 3h)
Encode walks the user map with lua_next (the documented JIT exception
per CLAUDE.md — map hot paths can't avoid hash iteration). Each entry
goes into a stack-backed sub-buffer with synthetic tag(1,key) +
tag(2,value); proto3-elides default key and default value independently.
Map<,message> resolves its value sub-plan once and recurses through
encode_body. Decode reads the entry payload bounded, dispatches inner
id=1/id=2 (skipping anything else per spec), and lua_rawsets into a
lazy-created result map table; missing key or value defaults to the
proto3 zero. Reuses the existing list_stack_idx[] slot for the lazy
map cache since a field is either repeated or map, never both.
12 new tests cover round-trip for ages_by_nickname (string->int32),
nickname_by_age (int32->string), and addresses_by_label (string->
message), plus default-elision, multi-key correctness, empty maps,
and unknown-inner-id tolerance. Full suite 964/964 with PB_ENABLE_C=1.
bd-asz
c_runtime: oneof encode + decode dispatch (ra6 3g)
Encode resolves the active member per oneof group (last-non-nil wins in
declaration order via plan->oneofs[].member_indices), skips non-active
members, and force-emits the active branch so default values like
text="" still carry presence.
Decode clears sibling result-table entries after writing any field with
oneof_idx >= 0, mirroring codec.lua's oneof_siblings handling — wire
last-wins. Sibling-clear runs for both the scalar/string/enum/bytes arm
and the sub-message arm.
20 new tests in test/c_runtime_oneof_test.lua cover all three Result
branches across both codegen modes: byte-equal vs full.hello reference,
round-trip preservation of the active branch and absence of siblings,
default-value emission for active branches, and last-wins on both
encode (multi-branch input) and decode (multi-occurrence wire bytes).
Acceptance per bd-w3u
c_runtime: 64-bit cdata fidelity tests (ra6 3l)
Adds the c_int64.Wide fixture (one singular field per 64-bit kind)
plus an 18-test luatest group that round-trips each kind past 2^53
through the C runtime in both codegen modes. Confirms encode accepts
both LuaJIT int64_t/uint64_t cdata and Lua numbers, and decode
surfaces values >DBL_INT_MAX as cdata (matching msgpackffi /
net.box / box.tuple / built-in protobuf convention).
The C runtime already had the dispatch — to_int64_at / to_uint64_at
flow through luaL_toint64 / luaL_touint64 for cdata inputs, and
dec_push_one calls luaL_pushint64 / luaL_pushuint64 for every 64-bit
kind. This change pins the behavior under acceptance.
Closes tarantool-protobuf-awv (ra6 3l)