runtime: pb.decode_unsafe + M.<Name>_decode_unsafe in runtime mode (58u)
Completes the unsafe-decode story 6bb started in full mode. Runtime mode
now exposes the same API via parallel `f._reader_unsafe` closures
compiled in pb.finalize_message against a swapped scalar table where
`string` maps to the bytes handler (no utf8_len). build_reader /
build_repeated_reader / decode_one are now parameterized on
(scalar_tbl, decode_msg_fn, decode_group_fn) so the same builders emit
both reader shapes. decode_message_unsafe, decode_group_unsafe, and
decode_extension_unsafe are literal clones of their safe twins with
three substitutions (documented in codec.lua): the _reader field, the
scalar table in slow paths, and the sub-message / group / extension
dispatchers. Tests in test/decode_unsafe_test.lua are now parameterized
over both modes (14 cases, including a map<string, int32> invalid-key
case that exercises the decode_one map-fallback path).
Runtime-mode microbench shows ~8% throughput vs validating decode on
the string-heavy 1KB Person; smaller than full mode's ~20% because the
descriptor dispatch + closure indirection swamp utf8_len, but still a
net win and the perf-cost-of-validating story is now consistent across
modes. Conformance 3240/3240 + JIT trace 37/37 still pass.
Closes 58u, also closes b12 (already fixed in 2656c97; never closed).
kyt still tracks unifying _decode_unsafe with C accel.
codegen: emit <Msg>_decode_unsafe for trusted-source decoding (6bb)
Full-mode codegen now emits a sister <Msg>_decode_unsafe(buf) alongside
<Msg>_decode that drops the per-string utf8_len check (singular,
repeated, map keys/values, extensions, and the >=128-byte fallback all
route through wire.decode_bytes). Sub-messages recurse into their own
_decode_unsafe so nested strings also bypass; WKTs continue to call the
normal pb.wkt.<Name>_decode (no _unsafe twin, no string-validation hot
path). C runtime dispatch is skipped because it validates today (kyt).
Use this when re-decoding bytes from a trusted producer — your own
encoder over typed RPC, JSON/text round-trips, in-process pipelines —
where the spec-required utf8.len check on every string is duplicate
work. Microbench on a string-heavy 1KB Person (26 emails) shows
~20% throughput vs _decode; conformance suite still passes (3240/3240)
because _decode itself is unchanged.
Runtime mode does not yet expose _decode_unsafe (compiled f._reader
closures capture handler.decode by value, so a runtime swap wouldn't
reach them); tracked in 58u.
c_runtime: strict-decode parity with pure-Lua codec (rc8)
The C decoder accepted wire types 6/7, field number 0, field numbers
beyond 29 bits, overlong tag varints, invalid UTF-8 in string fields,
and unmatched proto2 SGROUPs — each of which the pure-Lua decoder
already rejected. It also replaced (instead of merged) when a singular
message field appeared more than once on the wire, dropping
sub-message scalars from the prior occurrence.
decode_body: reject wt 6/7 / field 0 / fn > 2^29-1 / overlong tag
varint up front; track egroup_seen so a group body that runs off the
end of the buffer fails loudly. dec_push_kind STRING: port wire.lua's
RFC 3629 validator (utf8.len equivalent) — covers singular, repeated,
oneof, map-key, and map-value via the same code path. Singular message
dispatch in decode_body and decode_extension_into: look up an existing
prev table at result[name] and merge via the new
merge_subresult_into, which ports codec.lua's merge_message (recursive
sub-message, repeated concat, map last-wins per key, skip for WKT
custom-decode).
Regression coverage in test/conformance_test.lua's conformance.core
group: pre-existing tests already pinned the wire-type, tag, UTF-8
(singular/repeated/oneof), and merge cases — those now also exercise
the C path under PB_ENABLE_C=1. Two gaps remained — map-key/value
UTF-8 and proto2 group balancing — both covered now via four new
tests, with proto2 routed through a TestAllTypesProto2 helper.
PB_ENABLE_C=1 just test: 1043/1043 (baseline 1039 + 4 new).
PB_ENABLE_C=1 just conformance-c: 0 unexpected failures (was 77).
just conformance (pure-Lua path): unchanged — no regression.
Closes rc8
c_runtime: proto2 — required, defaults, groups, extensions (ra6 3i)
Plumb proto2 semantic surface through the C plan and codec:
* required: encode-time `required field missing` error with full path
(`<msg>.<field>`); set scalars/enums force-emit so zero still reaches
the wire (matches build_required_writer).
* groups: kind='group' compiles to PB_KIND_MESSAGE with is_group=1; tag
uses SGROUP and a pre-encoded EGROUP closer; encode walks regular
body bytes into a sub-buf and brackets with SGROUP+body+EGROUP (no
length prefix). Decode adds a stop_group_id to decode_body so the
inner walk terminates on the matching EGROUP, with id-mismatch as a
hard error per spec. Unknown-tag skip (`dec_skip_with_id`) recurses
through SGROUP bodies to the matching EGROUP.
* extensions: walk `desc.extensions_list` at plan-compile time, cache
each ext's full_name; encode iterates the cached array and emits any
present in `data._extensions[full_name]`; decode probes unregistered
tags against `plan->extensions` before falling through to
`_unknown_fields`, routing matched bytes into result._extensions.
Defaults: presence-tracked optional fields stay nil-on-absent in the
decoded table; the descriptor's `default_value` is surfaced for callers
(JSON, text) but never auto-materialized at decode — same as codec.lua.
36 new tests in test/c_runtime_proto2_test.lua exercise required missing
+ zero-emit, presence-tracked defaults, singular/repeated groups, and
extension round-trip — each asserts byte-equality with mode=full pure-Lua
output across both codegen modes.
ra6 3i
c_runtime: WKT override-hook passthrough (ra6 3k)
A plan whose descriptor carries desc.encode / desc.decode now
dispatches through those overrides instead of erroring. Top-level
encode/decode, sub-message fields, and map<,message> values all
check the override refs and call them with the same contract the
pure-Lua codec uses: encode(value) -> body bytes; decode(buf) ->
value. runtime/pb/wkt.lua is unmodified.
ra6 3k
c_runtime: unknown-fields capture + re-emission (ra6 3j)
c_runtime: map<K,V> encode + decode dispatch (ra6 3h)
Encode walks the user map with lua_next (the documented JIT exception
per CLAUDE.md — map hot paths can't avoid hash iteration). Each entry
goes into a stack-backed sub-buffer with synthetic tag(1,key) +
tag(2,value); proto3-elides default key and default value independently.
Map<,message> resolves its value sub-plan once and recurses through
encode_body. Decode reads the entry payload bounded, dispatches inner
id=1/id=2 (skipping anything else per spec), and lua_rawsets into a
lazy-created result map table; missing key or value defaults to the
proto3 zero. Reuses the existing list_stack_idx[] slot for the lazy
map cache since a field is either repeated or map, never both.
12 new tests cover round-trip for ages_by_nickname (string->int32),
nickname_by_age (int32->string), and addresses_by_label (string->
message), plus default-elision, multi-key correctness, empty maps,
and unknown-inner-id tolerance. Full suite 964/964 with PB_ENABLE_C=1.
bd-asz
c_runtime: oneof encode + decode dispatch (ra6 3g)
Encode resolves the active member per oneof group (last-non-nil wins in
declaration order via plan->oneofs[].member_indices), skips non-active
members, and force-emits the active branch so default values like
text="" still carry presence.
Decode clears sibling result-table entries after writing any field with
oneof_idx >= 0, mirroring codec.lua's oneof_siblings handling — wire
last-wins. Sibling-clear runs for both the scalar/string/enum/bytes arm
and the sub-message arm.
20 new tests in test/c_runtime_oneof_test.lua cover all three Result
branches across both codegen modes: byte-equal vs full.hello reference,
round-trip preservation of the active branch and absence of siblings,
default-value emission for active branches, and last-wins on both
encode (multi-branch input) and decode (multi-occurrence wire bytes).
Acceptance per bd-w3u
c_runtime: 64-bit cdata fidelity tests (ra6 3l)
Adds the c_int64.Wide fixture (one singular field per 64-bit kind)
plus an 18-test luatest group that round-trips each kind past 2^53
through the C runtime in both codegen modes. Confirms encode accepts
both LuaJIT int64_t/uint64_t cdata and Lua numbers, and decode
surfaces values >DBL_INT_MAX as cdata (matching msgpackffi /
net.box / box.tuple / built-in protobuf convention).
The C runtime already had the dispatch — to_int64_at / to_uint64_at
flow through luaL_toint64 / luaL_touint64 for cdata inputs, and
dec_push_one calls luaL_pushint64 / luaL_pushuint64 for every 64-bit
kind. This change pins the behavior under acceptance.
Closes tarantool-protobuf-awv (ra6 3l)
c_runtime: repeated string + repeated message acceptance at 1KB/10KB/100KB (ra6 3f)
The repeated string and repeated message (self-reference) dispatch
landed implicitly with ra6 3e — encode_repeated_field already routes
PB_KIND_MESSAGE through encode_submessage_field and unpacked
string/bytes through encode_one_field, and decode_body's cached
list_stack_idx[] handles every repeated element type. What was
missing was the formal 3f acceptance: explicit-size byte-equality at
1KB / 10KB / 100KB for Person.emails and Person.friends, against
mode=full.
Adds 4 acceptance tests (encode + decode × emails + friends), each
running both codegen modes. Sizing pins each case into a ±30% band
around the named target so a future schema/wire shift fails loudly
instead of silently drifting.
bd-exy
c_runtime: repeated + packed scalar encode/decode (ra6 3e)
Add repeated dispatch to the C-runtime encode/decode loop. Encode
side: encode_repeated_field walks Lua arrays via lua_objlen + per-
index rawgeti, dispatches on element kind. Packed numerics build
their payload in a stack-backed sub-buffer then emit `tag(LEN) +
varint(len) + body`; unpacked emit `tag + value` per element via
encode_one_field with force_emit=1 to bypass zero-suppression;
strings/bytes flow through the same path (never packable); repeated
messages reuse encode_submessage_field per element.
Decode side: per-field stack-slot cache (list_stack_idx[]) tied to
list_count[] avoids the per-element lua_getfield(result, name) round
trip the c-accel spike measured at 2x slower at 100KB. On first hit
for a repeated field we lua_createtable + write result[name] AND dup-
push the list onto the stack; subsequent hits lua_rawseti through the
cached absolute stack index. Lists stay valid across recursive sub-
message decodes because each child decode_body cleans up its own
scratch back to the caller's frame.
Packed/unpacked symmetry on read: a wt==LEN payload for any packable
scalar is decoded as a packed blob regardless of the schema's packed
flag, and a per-element-tagged stream is decoded element-by-element
even on a schema that defaults to packed — per proto3 reader rules.
New test/proto/c_repeated.proto fixture carries packed + explicit-
unpacked + repeated string/bytes + repeated message branches. The
encode and decode tests round-trip at 10/100/1000 elements per
branch. The two existing "skip repeated and map" marker tests
collapse to "skip map" — only map fields remain out of scope for
3e (bd-asz / 3h lands them next). 854 → 896 passing tests.
bd-jc9
c_runtime: encode/decode singular sub-messages (ra6 3d)
Refactor encode_lua/decode_lua into encode_body/decode_body so the
field-walk loop is callable recursively, then dispatch the MESSAGE
kind into a per-side sub-handler. Repeated and map fields still skip
at the field-walk level — 3e (jc9) and 3h (asz) land them next.
encode_submessage_field force-establishes the parent enc_buf's
heap_idx via a no-op ebuf_grow before recursing. Without that the
final ebuf_reserve on the parent could land its new userdata above
sub-encode's leaked stack slots, making the closing lua_settop drop
the parent's heap.
decode_submessage_field bounds the inner read by temporarily
shrinking c->len to the sub-message end offset; the wire-prim
helpers already bounds-check against c->len, so a malformed inner
payload can't over-read into the outer message's bytes.
New fixture test/proto/c_nested.proto carries a 5-level singular
chain (L1->L2->L3->L4->L5) for the depth test. The two existing
"skip message" tests are renamed to "skip repeated and map" — sub-
messages now encode and decode end-to-end.
bd-hwe
c_runtime: decode singular scalars (ra6 3c)
New entry pb.c_runtime.decode(plan, bytes) -> table, symmetric in
scope with 3b: int32/64, uint32/64, sint32/64, bool, fixed32/64,
sfixed32/64, float, double, enum, string, bytes. Repeated, map,
message-typed, and unknown tags skip by wire type — 3d/3e/3i/wyp
will extend later.
Result shape mirrors mode=full pure-Lua decode byte-for-byte: int64
family (int64/uint64/sint64/fixed64/sfixed64) push Tarantool cdata
via luaL_pushint64 / luaL_pushuint64; everything else lands as Lua
number/string/boolean. Pre-sized via lua_createtable(0, n_fields).
Field lookup is a linear scan over plan->fields by field_number; a
tag-keyed dispatch table is a future optimization.
Tests: 26 cases per codegen mode (full + runtime), covering the
bd-mz6 acceptance (round-trip of bd-y1n's Person payload matches
mode=full Person_decode shape-for-shape) plus per-kind coverage,
proto3-optional presence, fixed64 cdata, -0.0 sign preservation
(built via cdata to dodge LuaJIT literal-folding to +0.0),
skip-by-wire-type for out-of-scope shapes, unknown-tag skip, WKT
override rejection, and truncated input. Suite: 832/832 with
PB_ENABLE_C=1, 748/748 + 84 skipped without.
Closes bd-mz6
c_runtime: encode singular scalars (ra6 3b)
New entry pb.c_runtime.encode(plan, msg) -> string. Covers int32/64,
uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double,
enum (number or by_name lookup), string, bytes. Repeated, map, and
message-typed fields are silently skipped — those land in 3d/3e.
Buffer is a 4KB stack scratch promoted to a lua_newuserdata on
overflow, so a mid-encode luaL_error doesn't leak: the userdata is
GC'd at the unwind point.
Proto3 zero-suppression mirrors mode=full byte-for-byte: empty strings,
zero ints/fixed/enum skipped, +0.0 double skipped while -0.0 is emitted
via type-pun u64 equality (matches the `1/v == -math.huge` guard on
the Lua side). Proto3-optional fields bypass suppression. Plans with
desc.encode overrides (WKT) are rejected here — bd-rmf scope.
Tests: 16 cases per codegen mode (full + runtime), covering the bd-y1n
acceptance (Person {name='x', age=42, balance=-7,
user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full)
plus per-kind sweep and heap-grow path at 8KB. Suite: 806/806 with
PB_ENABLE_C=1, 748/748 + 58 skipped without.
Closes bd-y1n
fix(wire): cap the Lua-number varint fast path at 2^51 (x86_64 corruption)
encode_varint had a fast path for Lua numbers in [2^28, 2^53) that emitted
each byte via bit.band(n, 0x7f) / math.floor(n / 128). bit.band routes
through LuaJIT's number->int32 conversion, which on x86_64 uses the
magic-number trick (add 2^52 + 2^51, read the low bits). That is exact only
while n + 2^52 + 2^51 < 2^53, i.e. n < 2^51; above it the addition rounds to
an even double and silently drops low bits, corrupting the varint.
arm64 LuaJIT uses an exact FP->int instruction, so the bug was invisible on
Apple-Silicon dev machines and only surfaced on x86_64 (a 64-bit lease ID in
tarantool-etcd round-tripped 3041234677171912 -> 3041234677171940 over gRPC,
breaking lease lookups). Cap the fast path at 2^51; values in [2^51, 2^53)
now fall through to the exact uint64 cdata loop.
Adds test/wire_varint_test.lua pinning the round-trip at the boundaries.
c-accel: descriptor -> C plan compiler (bd-mq7)
First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.
What the plan carries (per docs/specs/c_accel_strategy.md):
* Per-field records: field_number, wire_type, kind, repeated/packed/
optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
sub_plan_idx (1-based into the sub-plans table), enum_ref
(luaL_ref for enum descriptor), oneof_idx back-pointer.
* Map fields capture map_key_kind, map_value_kind, and the value's
sub_plan_idx when the value is a message.
* Oneofs as a parallel array of {name, member_indices[]}, with
fields' oneof_idx pointing back to their group.
* WKT override detection: when desc.encode/desc.decode are set, the
plan flips has_override=1 and skips field-walk entirely.
* Extension range hooks captured (proto2 scaffolding for bd-3i).
* Field-name cache as a Lua table referenced via luaL_ref, so
encode/decode can do lua_rawgeti instead of re-interning C strings.
Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.
Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.
Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
- module surface + ABI version + KIND/WIRE constants
- Address: 4 fields, names, kinds, wire types, tag bytes, optional
- Person: 14 fields, scalars/enum/message/map/repeated/packed
- Sub-plan resolution + Person.friends self-reference cycle break
- Idempotent compile (second call returns cached plan)
- Result.outcome oneof: 3 members, oneof_idx back-pointers
- WKT Timestamp: has_override=true, field-walk skipped
Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).
Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.
Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
docs: retire PLAN.md in favor of Beads issue tracking
Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.
References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.
The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.
Closes the goal of bd-8k2.
bench,codec: pin proto2 paths, kill pairs() on extension hot path
JIT trace gate gains 12 proto2-specific checks (required encode/decode,
groups singular + repeated, extension encode/decode). The extension-encode
check fired NYI on bytecode 72 (ISNEXT) — pairs() over the new
extensions_by_full_name hash is the same trace-abort that gates map
encode. Fix: register_extension now also appends to extensions_list,
an array view. codec.encode_message, text.emit_message and
json.encode_message all switch to ipairs over the list. Hash tables
stay around for O(1) lookups in decode (extensions_by_id) and bracket-
name resolution (extensions_by_full_name).
Inline (full-mode) codegen learns to walk extensions too: before this
commit the inline encoder skipped t._extensions entirely (only the
runtime codec walked it), so a generated _encode silently dropped any
extension data set on the message. Add the array-walk after the field
loop and a decode_extension dispatch in the unknown-tag branch — the
inline path now matches runtime byte-for-byte.
bench/bench.lua parameterizes over a FIXTURES list so the baseline can
cover both hello.Person and a new proto2_basic.BenchPayload fixture
(required + group + extension + repeated). baseline.json restructured
under "schemas": [{schema, results}, …]; compare walks both. Fresh
numbers committed.
740/740 luatest cases pass; JIT gate 37/37; conformance 2806 binary+JSON
and 434 text-format, both 0 failures.
conformance: dispatch proto2 TestAllTypesProto2
Register the proto2 test message in MESSAGE_REGISTRY and the
Any/JSON registry. The harness now picks up 1312 previously-skipped
binary + JSON cases; whole suite jumps to 2805 ✓ / 0 failures (was
1493 ✓ with proto2 fully skipped).
Text-format suite still has 16 unexpected failures, all bound up
in proto2 group-field text syntax (CamelCase submessage label, no
mandatory `:` before `{`, extension-bracketed group form). Tackled
in the next commit.
The conformance unit test that pinned the proto2-skipped behavior
now targets an editions test type instead so the skip-path still
has a hard assertion.
codec: proto2 groups (SGROUP/EGROUP wire format)
Plugin: detect protoreflect.GroupKind, emit kind='group' in the
field descriptor and a SGROUP opening tag in inline codegen. The
group body bypasses the LEN-prefix path entirely — encoder emits
start-tag, body bytes, end-tag in three slots; decoder calls
pb.codec.decode_group(desc, buf, pos, field_id) which reads inner
tags until the matching EGROUP id.
Wire layer already understood SGROUP/EGROUP for skip_field; the new
decode_group reuses the per-tag dispatch from decode_message but
stops on EGROUP instead of end-of-buffer. Repeated groups bracket
each element with its own SGROUP/EGROUP pair.
Runtime parser now desugars `optional|required|repeated group Name
= id { body }` into (a) a nested message named Name and (b) a
synthetic field of kind=group whose lowercased name is `name` — so
source-parsed proto2 schemas behave the same as build-time codegen.
Text format renders groups under the submessage's capitalized name
(`SingleGroup { ... }`) rather than the lowercase field name,
matching mainline protoc's convention.
Adds a vendored, MessageSet-stripped test_messages_proto2.proto in
test/conformance/proto/ (header comment documents the patch).
10 new luatest cases (group_*) cover singular + repeated + text +
descriptor kind, across both codegen modes. 739 tests pass.