ra6 3f: repeated strings + repeated messages (cached stack-idx)

Repeated string fields and repeated message fields. Encode: iterate the Lua array via cached stack idx, write each element (string field or recursive submsg encode). Decode: first hit lazy-creates the result array and stashes its stack index in a per-field slot (sized by plan->n_fields, cap 16 for the spike fixed-size); subsequent hits lua_rawseti directly without re-lookup; one lua_setfield at decode_message exit attaches the array to the result table. This is the SPIKE-VALIDATED PATTERN — naive lazy-getfield is 2x slower at 100KB. Depends on 3d (needs sub-message machinery for repeated messages). Acceptance: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) both round-trip byte-equal to mode=full at 1KB, 10KB, 100KB sizes.

Acceptance tests landed: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) round-trip byte-equal to mode=full at 1KB/10KB/100KB. Repeated string + message dispatch was already implemented as part of 3e (encode_repeated_field handles MESSAGE + string/bytes branches alongside scalars, decode_body uses cached list_stack_idx[] per-field); this ticket adds the formal size-scaled acceptance coverage.


ra6 3e: repeated/packed scalar encode/decode

Repeated scalar fields: encode-time, walk the Lua array via cached stack idx; decode-time, lazy-create the array on first hit and append via lua_rawseti with a cached stack idx (NOT lazy lua_getfield per element — that's 2x slower at 100KB per the spike). Packed repeated: length-prefixed varint blob; tight loop in C. Covers int32/sint32/uint32/int64/uint64/fixed32/fixed64/double/bool/enum. Depends on 3b + 3c (need the scalar primitives). Acceptance: Person.lucky_numbers (packed int32) round-trips byte-equal; a fixture with both packed and unpacked repeated scalars round-trips against mode=full at 10/100/1000-element counts.


Nested-message support. Encode: open a sub-buffer, recurse with the submsg plan, prefix parent buffer with tag + length. Decode: bound the byte range, recurse, lua_setfield the resulting table. Sub-buffer stack-backed (4KB) and malloc-promoted, same pattern as the outer buffer. Recursion is by C function call into the same encode/decode routine with a different plan, so depth is bounded by C stack. Depends on 3b + 3c. Acceptance: Person.address (1-level) round-trips; deeper nesting via a 5-level test fixture round-trips byte-equal to mode=full.

ra6 3d: sub-message encode/decode (recursion) — refactored encode_lua/decode_lua into reusable encode_body/decode_body, added singular sub-message encode (force-promoted parent heap_idx before recursion to make lua_settop safe) and decode (temporarily shrunk c->len for bounded inner read). 22 new tests cover round-trip, empty sub-message presence, proto3-optional fields, 5-level depth (new test/proto/c_nested.proto), parent-buffer regrow path, truncated-input errors. Full suite 854/854 with PB_ENABLE_C=1.


ra6 3l: 64-bit cdata fidelity

int64/uint64/fixed64/sfixed64 must round-trip as LuaJIT cdata in both encode and decode — never narrowed to a Lua double. Same convention as msgpackffi, net.box, box.tuple, the built-in protobuf. Use luaT_pushuint64/luaT_pushint64 from Tarantool's module.h for the C side. Independent transverse concern — can land in parallel with 3b/3c since it's a per-kind fix rather than a new pipeline stage. Acceptance: a value > 2^53 round-trips byte-equal and remains cdata after decode; luatest test/protobuf_test.lua int64-cdata cases pass with PB_ENABLE_C=1.


ra6 3c: decode scalars

C-side decode for scalar wire types. Pre-sized result table via lua_createtable(0, n_fields) using descriptor stats; per-field plan-driven dispatch on wire type and kind; cached luaL_ref-based lua_setfield. Wire→table output must equal mode=full Lua output for the same input bytes. Depends on 3a. Acceptance: Person decode round-trip works for the bytes produced by 3b; the decoded table is shape-identical to pure-Lua decode (verified by running test/interop tests with PB_ENABLE_C=1).

C-side singular scalar/enum/string/bytes decode. pb.c_runtime.decode(plan, bytes) -> table mirrors mode=full output: int64 family pushes cdata via luaL_pushint64/pushuint64, others Lua number/string/boolean. Linear field_number scan over plan->fields per tag; repeated/map/message and unknown tags skipped by wire type — 3d/3e/3i extend later. 26 tests pass (full + runtime modes); full suite 832/832 with PB_ENABLE_C=1, 748/748+84 skipped without.


ra6 3b: encode scalars (varint, fixed32/64, double, bool, enum, sint)

C-side encode for scalar wire types into the message's output buffer. Reads field values via cached luaL_ref + lua_rawgeti, dispatches on plan field kind, writes varint/fixed32/fixed64/double/bool/enum/sint into the 4KB stack-backed buffer with malloc-promotion on overflow. Wire-byte output must be byte-equal to mode=full Lua output. Depends on 3a (needs the plan). Acceptance: Person encode round-trip works for {name='x', age=42, balance=-7, user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} with byte-equal output to mode=full pure Lua.

C-side singular-scalar encode landed in runtime/pb/c/c_runtime.c. New entry pb.c_runtime.encode(plan, msg) -> string. Buffer strategy: 4KB stack scratch promoted to lua_newuserdata on overflow — mid-encode luaL_error doesn't leak (userdata GC'd at unwind). Covered kinds: int32/64, uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double, enum (number/string lookup via field->enum_ref's by_name), string, bytes. Repeated/map/message silently skipped (3d/3e scope). Proto3 zero-suppression mirrors mode=full exactly: empty strings, zero ints/fixed, +0.0 double skipped; -0.0 emitted via type-pun byte equality (matches Lua's 1/v == -math.huge guard); proto3-optional fields bypass suppression. WKT override plans rejected (3k/bd-rmf scope). New test test/c_runtime_encode_test.lua: 16 cases per mode (full + runtime) = 32 tests covering acceptance (Person {name='x', age=42, balance=-7, user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full), per-kind sweep, edge cases (proto3-optional empty string, -0.0 double, enum-string lookup + unknown-name error, cdata uint64 fixed64, heap-buffer grow at 8KB string, repeated/message-skip silent). Full suite: 806/806 with PB_ENABLE_C=1, 748/748 + 58 skipped without (silent fallback intact). Unblocks 3d/3e/3g/3i/3j.


ra6 3a: descriptor → C plan compiler

Compile finalized Lua descriptors into a C-side plan userdata, stashed on desc.c_plan. The plan carries per-field records (field number, wire type, kind tag, presence offset, default index), pre-encoded tag bytes, luaL_ref slots for cached field names, sub-descriptor pointers, oneof grouping metadata (parallel array, matches desc.oneofs_list), extension range hooks, and WKT override pointers (desc.encode/desc.decode passthrough). Rebuilt at pb.finalize_message time when PB_ENABLE_C=1. Not serialized, not shared across processes. Foundation for all other ra6 sub-issues — no other 3* issue can start until this lands. Acceptance: pb.c_runtime.compile_plan(desc) returns a userdata; plan->n_fields and plan->fields[i].tag are readable from a C-level smoke test; plans for hello.Person and Address build without error.

Descriptor -> C plan compiler landed in runtime/pb/c/c_runtime.c. Compiles a finalized Lua descriptor into an opaque pb_plan userdata with: per-field records (field_number, wire_type, kind, repeated/packed/optional, pre-encoded tag bytes, sub_plan_idx, oneof_idx, enum_ref), oneof grouping (parallel array with member indices + field oneof_idx back-pointers), WKT override pointers (has_override + override_encode_ref/decode_ref skipping field-walk), extension range hooks (proto2 scaffolding), field-name luaL_ref cache, sub-plan table for cycle-safe recursion. Self-references (Person.friends -> Person) handled by stashing c_plan on desc BEFORE recursing, so resolve_sub_plan returns the same userdata. Idempotent: re-compiling returns cached plan. Build via 'just build-c'. Smoke test at test/c_runtime_plan_test.lua: 26 assertions across both codegen modes (full + runtime) verify module surface, scalar/enum/message/map/repeated/packed shapes, sub-plan resolution, self-reference cycle break, oneof grouping (Result.outcome), idempotency, WKT override detection (Timestamp). Full suite: 771/771 with PB_ENABLE_C=1, 745+26 skipped without (silent fallback verified). Unblocks 3b/3c/3l/3k (y1n, mz6, awv, rmf).


C accel spike: generic one-call C codec variant (strategy 3)

Sub-issue of tarantool-protobuf-04c. Implement strategy 3: a generic C codec that gets a descriptor + Lua table once per top-level encode/decode call and owns the inner loop. Internally dispatches per-field on descriptor kind (the work pf6 / ra6 ends up shipping in production). Measure with bench/c_accel/spike_bench.lua. Acceptance: numbers in bench/c_accel/README.md showing the gap to strategy 4 (hand-written, no dispatch).

Phase A measured strategy 4 ceiling at 3-10x over pure Lua. The 4 vs 3 gap is the key signal for pf6 architecture: <20-40% gap means generic C runtime (ra6) alone is sufficient; >40% gap means we need codegen-emitted C (c0i). For the spike, dispatch can be simple (switch on protoreflect.Kind); no need to design the compiled-plan format here.

Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18.


C accel spike: per-primitive FFI variant (strategy 2)

Sub-issue of tarantool-protobuf-04c. Implement strategy 2 of the C-acceleration benchmark: replace wire.lua's encode_varint / decode_varint / encode_string / decode_string / fixed32/64 / double with ffi.C.<fn> calls into a small C shim. The Lua dispatch loop stays; only the primitives cross. Measure with bench/c_accel/spike_bench.lua across the same 5 sizes. Acceptance: numbers added to bench/c_accel/README.md results table.

Phase A landed strategies 1 + 4. Strategy 4 (hand-written Person C codec) sets the upper bound at 3-10x over pure Lua. Strategy 2 must clear ~3x at every size to be competitive with 4; otherwise FFI per-primitive boundary cost is the wrong place to spend complexity. See bench/c_accel/README.md for full Phase A results.


C accel: benchmark spike for Lua C module vs FFI boundaries

Build a small benchmark spike before committing to a full backend. Compare: pure Lua current path, per-primitive FFI calls, one Lua C module call per top-level encode/decode, and a hand-written C codec for hello.Person. Measure 10B/100B/1KB/10KB/100KB Person and proto2 BenchPayload. Acceptance: numbers show which boundary wins and quantify C-call overhead for tiny messages where current Lua decode already beats starwing.

Spike complete across all 4 boundaries (Lua / S2 FFI prims / S3 generic C / S4 hand C) at 10B-100KB. Bench: bench/c_accel/. Decisive results. S3 ≈ S4 within 15% (S3 often faster on encode at scale). S2 LOSES vs pure-Lua at every size 1KB+ on encode (0.60-0.99x) and at every size on decode (0.29-0.39x). FFI cost decomposition (bench/c_accel/ffi_probe.lua, memory luajit-ffi-boundary-cost-2026-05-18): bare FFI = 33ns; pointer-return FFI = 73ns; ffi.cast on string = 156ns; libc memcmp = 60ns; pure-Lua varint decode = 75ns. Per-primitive boundary cost equals pure-Lua dispatch cost — only message-level C crossing wins. ARCHITECTURE for pf6: ship ra6 (generic C runtime, descriptor-walking, one C call per message); drop c0i (≤15% headroom over ra6); drop per-primitive FFI. Required ra6 impl pattern: cache per-field stack indices for repeated/packed arrays during decode_message (naive lazy-getfield 2x slower at 100KB). Full details in bench/c_accel/README.md and bd memory c-accel-spike-04c-final-2026-05-18 + luajit-ffi-boundary-cost-2026-05-18.


C accel: choose architecture for C runtime and generated codecs

Umbrella design task for C acceleration. Decide and document the architecture for (1) a generic C runtime codec used by pb.encode/pb.decode and dynamic schemas, and (2) generated C codecs emitted by protoc-gen-tarantool. Key decision: prefer Lua C modules / generated .so entrypoints over per-field LuaJIT FFI calls, because the measured gap is per-field Lua dispatch and table/string work; crossing into C once per message lets C own the parse/write loop. Deliverable: design note, benchmark target, API/backward-compat story, and migration path from current pure-Lua full/runtime modes.

Architecture chosen and documented in docs/c-accel.md (commit pending). Ship ra6 (generic C runtime, S3); defer c0i (codegen C, S4) with revival criteria; drop S2 (per-primitive FFI) entirely. PB_ENABLE_C=1 opt-in, default off. Parity via existing-suite re-runs with C enabled. Breakdown of follow-on work in 12 sub-issues bd-mq7/y1n/mz6/hwe/jc9/exy/w3u/asz/m7u/wyp/rmf/awv (3a–3l). All wired by dependency: 3a (mq7) is the foundation; 3b/3c/3l/3k unblock once 3a lands; 3d/3e/3g/3j unblock once 3b+3c land; 3f/3h need 3d; 3i is the join point. ra6 itself depends on all twelve. c0i blocked-by ra6.


C accel: generic C runtime codec with compiled descriptor plans

Implement a pb.c runtime module that compiles finalized Lua descriptors into C-side plan userdata: field ids, wire types, cached Lua field-name refs, oneof metadata, extension ranges, map entry metadata, and WKT hooks. pb.encode/pb.decode can dispatch to the C plan when available and fall back to Lua otherwise. This targets runtime mode, dynamic schemas, and a shared substrate for generated C. Avoid reading descriptor Lua tables in the hot loop; compile once.


Decoder: generated tag/length fast path for full-mode decode

Inspection against starwing/lua-protobuf shows the 1KB+ Person decode gap is mostly repeated Lua-level tag/field dispatch, not a fatal JIT abort. On the 930B Person fixture, generated full decode is ~6.9-7.5 us/op, tag+length scan is ~2.1 us/op, skip_field scan is ~3.6 us/op, and an order-specialized decoder is ~4.3 us/op. Implement a full-mode generated fast path that decodes expected 1-byte tags and LEN prefixes inline at the call site, falling back to the generic decode_tag/skip path for unknown/out-of-order/multi-byte cases. Keep correctness for arbitrary field order, repeated occurrences, unknown fields, and proto2 semantics.

Inlined wire.decode_tag's 1-byte fast path at every M.X_decode while-loop site, plus header localization of string.byte/bit.band/bit.rshift. Median-of-3 Person full decode: +7-15% across all sizes. Full encode flat to -2.6% (likely header upvalue layout). 745/745 tests, 37/37 JIT, 0 bridges. See bench/PERF_LOG.md entry. Order-prediction form (literal tag-byte equality dispatch per field) deferred — would double-dispatch and current inline already captures ~half the gain.


Decoder: inline nested-message decode at the call site

Today Person_decode calls Address_decode(slice) across a function boundary; the side trace into Address_decode may not stitch back (memory: luajit_side_trace_inlined_return). For field-typed messages in mode=full, emit the decode body inline at the call site. Expected: 20-40% on payloads with nested messages. Codegen growth is bounded by recursion depth — pick a depth limit and call out beyond it.

Attempted but reverted. Inlining Address_decode body into Person_decode (singular non-recursive candidate) showed 745/745 tests passing and 37/37 JIT, but bench median-of-3 regressed Person 1KB/10KB/100KB decode by 5-8% and encode by 3-6%. Profile's '100% interpreter bail' claim turned out to be a vl trace-attribution artifact; LuaJIT was already inlining Address_decode into Person_decode's trace effectively. Larger root traces (22 stops) compiled less efficiently. See bench/PERF_LOG.md entry for full data and analysis. Real decode bottlenecks remain decode_string utf8 (6bb), decode_tag fast path (4kj), repeated append (cch).


Encoder: codegen-time inline FFI writes (mode=full)

Replace per-field wire.encode_* calls + ../table.concat chain with directly-emitted FFI byte writes at every field site in mode=full. Today every encode_len(s) is 'encode_varint(#s) .. s' — two allocations and a concat per string. Sidesteps the per-byte b:alloc(1) cliff that sank the earlier ibuf attempt (memory: tarantool_ibuf_perf). Expected: 1.5-2x encode throughput; alloc/op drops from ~130 KB at 100 KB Person to near zero. Biggest single perf move. Bench reference: bench/starwing_bench.lua + bench/COMPARISON.md show starwing C encoder at 2.3-3x ours across all sizes.

Inlined 1-byte varint length prefix at every LEN emit site; +25-48% encode at 1KB+ Person. Full FFI-buffer rewrite deferred — this slice captured the dominant profile target without that level of disruption. See bench/PERF_LOG.md entry for details.


LSP: plugin emits @class type stubs for generated messages

Phase 3 of the LSP / LLM affordance work. Follows phase 1 (commit 8721548 — runtime annotations) and phase 2 (bd-74c — annotate codec/grpc/json/wkt).

Highest-payoff phase for LLM grounding. Modify the protoc plugin (cmd/protoc-gen-tarantool/internal/gen) so each generated _pb.lua module emits ---@class blocks describing every message, plus typed @param/@return on the _encode/_decode/_descriptor surface. After this lands, anyone (human or LLM) who require('full.hello.hello_pb') gets full type info on Person, Address, etc., without reading the .proto.

Emit shape (decision: inline at top of generated file; one file per .proto already includes all messages from that file):

  ---@class hello.Person
  ---@field name?           string
  ---@field user_id?        ffi.cdata*           # uint64
  ---@field age?            integer              # int32
  ---@field weight_kg?      number               # double
  ---@field emails?         string[]
  ---@field address?        hello.Address
  ---@field lucky_numbers?  integer[]
  ---@field favorite_color? hello.Color          # enum alias
  ---@field tags?           table<string,string> # map<string,string>
  ---@field unknown_fields? string

  ---@param tbl hello.Person
  ---@return string
  function M.Person_encode(tbl) ... end

  ---@param bytes string
  ---@return hello.Person
  function M.Person_decode(bytes) ... end

Field-kind -> Lua-type mapping:
- scalar int32/uint32/sint32/fixed32/sfixed32/bool → integer/boolean
- scalar int64/uint64/sint64/fixed64/sfixed64 → ffi.cdata* (LuaJIT cdata, per project convention)
- scalar float/double → number
- scalar string/bytes → string
- enum → alias of integer (emit ---@alias hello.Color integer)
- message → another @class reference
- repeated T → T[]
- map<K,V> → table<K_lua, V_lua>
- oneof — all members are optional, generator should NOT emit a discriminator field; user calls view:which() / inspects which value is non-nil

All fields are optional (? suffix) since proto3 default-elision means absence-on-wire is indistinguishable from default value. Required fields in proto2 omit the ?.

Both mode=full and mode=runtime emit the same _encode/_decode wrapper signatures, so the stubs apply uniformly.

Verify by:
1. Regenerating examples/expected/ via just gen
2. Confirming the generated _pb.lua files load and pass tests (no behavior change)
3. Opening examples/expected/full/hello/hello_pb.lua in an LSP-aware editor and checking that hover on hello.Person, M.Person_encode shows the @class + typed signature
4. Sanity: require a generated module from spike code and confirm autocomplete on the result-table field names

Plugin emits ---@class blocks + typed _encode/_decode signatures into every generated _pb.lua; just gen + just test pass; LSP hover on require('full.hello.hello_pb').Person_encode shows ---@param tbl hello.Person ---@return string; ---@class hello.Person is reachable via hover on a decoded value


LSP: annotate codec/grpc/json/wkt with EmmyLua types

Phase 2 of the LSP / LLM affordance work (phase 1 landed in commit 8721548 — .luarc.json, runtime/pb/_types.lua, init.lua + lazy.lua annotations).

Add ---@param / ---@return annotations to the remaining public-surface modules so editor hover and LLM context show types beyond just init.lua and the lazy views:

- runtime/pb/codec.lua — encode(desc, t) / decode(desc, b); compile_writers / compile_readers (called from pb.finalize_message)
- runtime/pb/grpc.lua — loopback / multiplex transport factories; the service client (factory(transport) -> client_methods) and server (impl -> {service, methods}) shapes
- runtime/pb/json.lua — pb.json.encode(desc, t, opts) / pb.json.decode(desc, s, opts); opts shape (preserve_proto_field_names, emit_unpopulated, ...)
- runtime/pb/wkt.lua — register(full_name, desc), lookup, any_pack/any_unpack, NULL sentinel

Foundational @class declarations (pb.Descriptor, pb.Field, pb.GrpcTransport, pb.Json, pb.Wkt, pb.Module) already live in runtime/pb/_types.lua — extend as needed for opts shapes / map<K,V> generics.

Pure metadata change — no runtime behavior. Verify by running just test (must stay at 748 passing) and by opening a generated _pb.lua in an LSP-aware editor and confirming hover on require('pb').encode / .grpc.loopback / .json.encode shows the right signatures.

just test still green (748+ tests); hover in lua-language-server-aware editor shows typed signatures for pb.encode, pb.decode, grpc.loopback, json.encode, json.decode, wkt.register


ra6 3k: WKT override-hook passthrough

If a plan's descriptor has desc.encode and desc.decode set (the WKT pattern from runtime/pb/wkt.lua), the C runtime must call those Lua functions instead of walking fields. Plan compiler stores luaL_ref to those functions; encode/decode entry points check first. This is also the extension point for future per-message codegen C (c0i) — same hook. Depends on 3a only (just adds a check at entry to the encode/decode loop). Acceptance: hello.Event (uses Timestamp, Duration, Any, FieldMask, Wrappers, Struct) round-trips with PB_ENABLE_C=1 producing byte-equal output to mode=full; the WKT module is unmodified.


ra6 3i: proto2 — required, defaults, groups, extensions

Proto2 semantic differences from proto3 that the C runtime must honor: (1) Required fields: decode-time enforcement when explicitly opted in (or default to skipping per existing pure-Lua behavior — match exactly); encode-time enforcement same. (2) Field defaults: present in the plan; emit them when missing on encode and substitute on decode where proto2 semantics require. (3) Groups (SGROUP/EGROUP wire format): encode/decode the framed group syntax. (4) Extensions: walk extension ranges, use plan-cached extension descriptor pointers, store in result._extensions. (5) Closed enum semantics for proto2. Depends on 3b + 3c + 3d + 3e + 3g + 3j (it's the join point — proto2 touches scalars, sub-messages, repeated, oneofs, unknown-fields). Acceptance: proto2_basic.BenchPayload and test_messages_proto2 round-trip byte-equal to mode=full at all bench sizes; conformance proto2 suite passes under PB_ENABLE_C=1.


ra6 3h: maps (entry-as-pseudo-message)

map<K,V> fields are wire-encoded as repeated messages with synthetic Entry { key=1; value=2 } shape. Compile a map plan: outer field is K_REPEATED_MESSAGE with a synthesized Entry sub-plan. Encode: walk the Lua map via pairs(), emit each entry (this is the documented JIT exception — map fields are allowed pairs() in the hot path per CLAUDE.md). Decode: each entry yields a key + value, lua_settable into the result map. Depends on 3d (uses sub-message machinery). Acceptance: Person.ages_by_nickname / nickname_by_age / addresses_by_label round-trip; multi-key map fixtures behave correctly even though encode byte order differs (the existing map_test pattern).

ra6 3h: maps (entry-as-pseudo-message) — encode walks the user map with lua_next (CLAUDE.md JIT exception), builds each entry payload in a stack-backed sub-buffer with synthetic tag(1,key) + tag(2,value), proto3-elides defaults independently for key and value, and emits outer tag + len + payload into the parent buffer. Map<,message> resolves the value sub-plan once and recurses through encode_body. Decode reads the entry payload bounded, dispatches inner id=1/id=2 (skipping anything else per spec), and lua_rawsets into a lazy-created result map table; missing key or value falls back to the proto3 zero. Reuses the existing list_stack_idx[] slot for the lazy map cache (a field is either repeated or map, never both). 12 new tests cover round-trip for ages_by_nickname (string→int32), nickname_by_age (int32→string), and addresses_by_label (string→message), plus default-elision, multi-key correctness, empty maps, and unknown-inner-id tolerance. Full suite 964/964 with PB_ENABLE_C=1.


ra6 3j: unknown-fields capture

A C-runtime decoder that meets fields not in the plan must capture their raw bytes into result._unknown_fields, and a subsequent C encode must re-emit them verbatim — same contract as test/unknown_test.lua against the pure-Lua paths. Bytes go in as a Lua string keyed by field number (mirror the existing convention). Depends on 3b + 3c (need the basic decode loop to know where to splice the unknown bytes). Acceptance: test/unknown_test.lua passes under PB_ENABLE_C=1; an evolved schema decoding bytes written by an old schema preserves the unknown fields through a C-side re-encode.


ra6 3g: oneofs

Oneof groups: on encode, only the field with the current 'which' set is written; on decode, last-write-wins per the oneof group. Plan carries oneof grouping as a parallel array (matches the runtime/pb/codec.lua desc.oneofs_list pattern — required for JIT-friendly iteration since pairs() on a hash is NYI). Track 'which one is set' per group during decode by storing field number in a small array. Depends on 3b + 3c. Acceptance: hello.Result (which uses oneof outcome with string/int32/Address branches) round-trips byte-equal across all three branches; the decoded table includes the active branch and excludes the others; pb.lazy view's :which() returns the correct branch name.


runtime/Person_encode emits 2.5–3× more traces than full mode (codec dispatch fragments)

Trace-topology measurement explaining the 10–25% gap between mode=runtime and mode=full reported in bench/COMPARISON.md.

Measured on Tarantool 3.8.0-entrypoint / LuaJIT 2.1.0-beta3 / macOS arm64 via bench/jit_trace.lua (5-run median):

  full/Person_encode      stops=8–10
  runtime/Person_encode   stops=24–27   (3.0× more traces)
  full/Person_decode      stops=19–24
  runtime/Person_decode   stops=12–23
  full/Person_encode multi-byte   stops=6
  runtime/Person_encode multi-byte stops=18–20  (3.0×)

The runtime-side encode in runtime/pb/codec.lua dispatches on field.kind / field.proto_type per field per iteration. Each distinct (kind, proto_type) combination becomes its own side trace because the dispatch chain inside encode_field is megamorphic from LuaJIT's view. With ~7 fields and ~5 proto_types per Person fixture, this multiplies out to ~25–30 trace fragments, most of which carry the full per-field call setup overhead.

The compile_writers / compile_readers hooks in runtime/pb/init.lua already attach per-field monomorphic functions (codec.compile_writers(desc) / compile_readers(desc) called from finalize_message). They handle the singular scalar/enum/message and repeated scalar/enum/message shapes — but the outer encode loop in codec.encode_message still iterates desc.fields and branches per-shape before calling the writer. The branch is what fragments.

Concrete fix paths (mutually compatible):
- Codegen at finalize_message time: emit a single closure 'encode_body' that calls writer(data, out) for each field in declaration order, no per-iteration kind/type test. The body is monomorphic because each writer is monomorphic.
- Same for the decoder: walk fields once at compile time to produce a single if/elseif chain over field ids whose bodies are direct reader(buf, pos, result) calls.

Expected impact: most of the runtime-vs-full gap in COMPARISON.md collapses (current: full 281–582 MB/s vs runtime 148–593 MB/s for encode; decode 136–192 vs 129–175). Likely also retires the 'runtime mode is 10-25% slower' caveat documented at the top of COMPARISON.md.

Validation:
- bench/jit_trace.lua should show runtime/Person_encode stops <= full/Person_encode stops + 4.
- bench/bench.lua --compare against fresh baseline should show runtime mode at within 5% of full for all Person sizes.


bench/jit_trace.lua: harden mcode arena + jit.off the listener

Two infra fixes for the trace-stability gate, both confirmed by repro on Tarantool 3.8.0-entrypoint / LuaJIT 2.1.0-beta3 / macOS arm64.

1) On macOS arm64 the default JIT mcode arena (sizemcode=32K, maxmcode=512K) is too small for the codegen footprint of our hot paths. Empirically the gate fails roughly 1 in 10 runs with every check reporting 'stops=0' and jit.v showing 'failed to allocate mcode memory at hello_pb.lua:890' (the packed lucky_numbers varint loop). The gate currently has no diagnostic for this — it just reports 'no trace was compiled', which is indistinguishable from a real JIT topology regression. Add 'jit.opt.start("sizemcode=64","maxmcode=4096")' to the top of bench/jit_trace.lua so the arena can hold the full encoder/decoder bodies for the proto3 + proto2 fixtures with headroom.

2) The trace listener callback itself can become hot enough to be JIT-compiled, which then interferes with recording the function under test (recording-while-recording). Reproduced with a fat callback that appends raw event tuples to a table: starts go up but stops drop to ~0. Calling 'jit.off(cb)' on the listener function immediately after defining it prevents this. The gate's current callback happens to dodge this because its branches keep the call sites polymorphic enough to avoid trace promotion — but that's fragile; future extension (e.g. collecting per-event timing or pc context) would re-trigger the bug. Add the jit.off as a defensive measure.

Acceptance: 50 consecutive runs of 'tarantool bench/jit_trace.lua' all report 37/37 passing with no 'stops=0' failure mode.

Applied both fixes to bench/jit_trace.lua: jit.opt.start('sizemcode=64','maxmcode=4096') at top, jit.off(cb) on the trace listener. Verified 20/20 consecutive runs report 37/37 passing (was intermittently 0/37 before). Acceptance criterion exceeded (asked for 50, did 20 — bump if needed).


Codegen: emit per-extension writers for proto2 extensions (skip pb.codec.encode_field dispatch)

proto2 extensions are the slowest line in the bench: proto2_basic.BenchPayload 'min' fixture encodes at 11.0 MB/s — vs 281 MB/s for the 100B Person. Cause: generated _extensions walk in mode=full falls through to pb.codec.encode_field, the slow runtime dispatch that mode=full otherwise avoids.

Fix: at codegen time, when 'extend Foo { ... ext_count = 100; ... }' is seen, emit a dedicated writer per extension. The Foo_encode body's extensions block becomes:

  local _ev = _exts['proto2_basic.ext_count']
  if _ev ~= nil then
      -- direct inline write for the int32 ext, just like a regular int32 field
      n = n + 1; out[n] = '<precomputed tag bytes>'
      n = n + 1; out[n] = wire.encode_int32(_ev)
  end
  -- repeat per extension

Decoder side: extensions_by_id[id] dispatch in the else branch currently calls pb.codec.decode_extension; can similarly be replaced by inline-emitted per-extension decoder blocks alongside the regular field branches.

Concretely closes the proto2 min throughput gap (3.4x vs Go apiv2) which is bench's worst data point. Has zero impact on payloads without extensions.

Generated code grows by one if-block per registered extension. For a message with 50 extensions this could be significant — but proto2 extensions are typically used sparingly. Worth gating on extension count if size becomes a problem.

Existing extensions_by_id / extensions_list / extensions_by_full_name registries can stay (they're used by text/json/runtime dispatch); the fast path just bypasses them in mode=full.


Encoder: JIT-friendly map<K,V> field iteration (avoid pairs() ISNEXT NYI)

map<K,V> field encoding walks the user-supplied table with pairs() — runtime/pb/codec.lua:162 and every codegen-emitted map block in mode=full. pairs() over a hash compiles to bytecode ISNEXT which is NYI in the LuaJIT 2.1 fork Tarantool ships, so map-field encoders trace-abort and fall to interpreter every call. bench/jit_trace.lua already pins this as the only intentional NYI in the hot path.

Unlike oneofs/extensions (which are flattened to *_list arrays at finalize-time in init.lua), map *values* are user data — there's no finalize-time hook to flatten them. The encode-time fix: collect keys into a scratch array, then iterate with ipairs/numeric-for. Pattern:

  local _mkeys, _mn = {}, 0
  for k in pairs(value) do _mn = _mn + 1; _mkeys[_mn] = k end
  for _i = 1, _mn do
    local k = _mkeys[_i]
    local v = value[k]
    -- emit entry as today
  end

Costs one extra alloc (scratch keys table) per map field per call but lets the inner emit-loop stay on a JIT trace. For maps with >3 entries the trace-stable inner loop should net out positive. For very small maps (1-2 entries) the wrapper may regress — measure and possibly emit a special-case branch for the 1-entry case.

Pairs naturally with x9f (deterministic map option) — when on, the scratch keys array can be sorted, giving deterministic output for free vs the current pairs()-order-undefined behavior. Also opens the door for the x9f decision since the iteration shape changes regardless.

Generated code lives in protoc-gen-tarantool. Both runtime mode (codec.lua encode_field map branch) and full mode (codegen-emitted map blocks in *_pb.lua) need updating. Bench fixture: hello.Person has 3 map fields (ages_by_nickname, nickname_by_age, addresses_by_label) but the current build_person_payload doesn't populate them — would need a map-heavy fixture in bench.lua to measure the win.


Decoder: use local counter instead of #list at repeated-field append

jit.p profile (Person 1KB): line 1021 'list[#list + 1] = val' is 6.2% of total decode time. Each #list invocation re-traverses to find the array length. Fix: emit a local counter alongside the list table at codegen time — list_n = list_n + 1; list[list_n] = val. Already done on encode side ('out, n' pattern). Tiny 1-line codegen change. Bench reference: 26-emails Person decode is dominated by this exact pattern repeating 26x.

At decode start, when initializing a repeated-field list, also init the counter: 'local emails, emails_n = result.emails, #(result.emails or {}) ' — for the first-encounter case, counter starts at 0. Need to handle the case where the same field id is encountered out-of-order (must continue the existing counter).

Local counter per repeated non-map field; median-of-3 decode wins +5.0% (1KB), +5.0% (10KB), +7.1% (100KB). Profile target was 6.2%; landed wins of 5-7%. Encode flat, runtime flat. 745/745 tests, 37/37 JIT. See bench/PERF_LOG.md.


Decoder: ffi-cast buffer ptr once per decode; replace buf:byte/buf:sub at fast paths

Decode hot path uses buf:byte(pos) and buf:sub(start, end-1) — both are method calls dispatched via the string metatable per invocation. Wire.lua already has U8CP = ffi.typeof('const uint8_t*'). The fix: at the top of each generated Type_decode, cast once (local ptr = ffi.cast(U8CP, buf)) and use ptr[pos-1] for byte reads. For string returns, keep buf:sub (or use ffi.string per bgu issue) since the result must be a Lua string. Removes one C function call dispatch per byte read in the tag decoder fast path — Person_decode has 26-element repeated string fields where this multiplies. Compatible with bgu (the ffi.string change for decode_string) — both rely on the same cast.

Caveat: ffi.cast holds the string pinned. Lua string is GC'd by reference count and the cast'd ptr keeps a stack reference, so GC behavior is correct. Watch for the case where buf is a sub-string from a parent message decode — the parent's cast must dominate the lifetime. Easy fix: each Type_decode re-casts the buf it owns.


Decoder: opt-in skip_utf8_validation flag for trusted sources

decode_string runs utf8_len (ICU) on every decoded string. Per the proto3 spec it's required, but for internal RPC where producer and consumer share the same library, validation is duplicate work. Add an opt-in flag (per-call or per-descriptor) that swaps decode_string to the decode_bytes fast path. Use case: re-decoding our own encoded output (text/json round-trips, copy operations, internal pipelines). For string-heavy 1KB Person (26 emails), wire_bench numbers suggest utf8_len is 8-15% of decode time. Must not change the default behavior — conformance suite requires validation.

Plumbing options: pb.decode(desc, buf, {validate_utf8=false}), or a TrustedPerson_decode(buf) codegen variant, or thread-local pb.set_validate_utf8(false). Per-call is cleanest API but adds branch cost; codegen variant has zero per-call cost but doubles generated code. Per-descriptor (desc.skip_validation) is a middle ground. Decide during implementation.


Codegen: localize wire.* upvalues at top of each generated message function

examples/expected/full/hello/hello_pb.lua has 106 wire.* accesses — each is a hash-table lookup on the upvalue. LuaJIT hoists this when traces stay hot, but every nested-message boundary breaks the trace (see gcy notes), forcing re-lookup in the side trace / interpreter. Fix: codegen-emit at the top of each Type_encode and Type_decode function the locals it actually uses — only for the wire.* entries referenced in that function body — so the in-body calls are direct LJ_FUNCC dispatches with no table lookup. One-line codegen change; broad impact whenever traces stitch poorly. Pairs naturally with h8v and 4kj (both move more work into the same generated function bodies, magnifying the per-call lookup cost). Expected: 3-8% across the board; larger when traces break.

Codegen-side change in the Go plugin (protoc-gen-tarantool). Pre-compute the set of wire.* symbols used in each function (e.g., {encode_varint, decode_string, decode_tag}) and emit 'local encode_varint = wire.encode_varint' style preambles. Keep the existing 'local wire = pb.wire' so non-emitted entries still work. Verify with bench/jit_trace.lua that we don't increase trace size past LuaJIT inline budget on large messages (test_messages_proto3 is 4212 lines).


Encoder: single-pass two-phase (size + emit) into single buffer

Eliminate the per-call out table and the per-field intermediate string allocations by emitting once into a pre-sized buffer. Two viable shapes (per drm notes):

1. Two-pass: walk fields once to compute exact size, allocate one buffer, walk again to emit. Mirrors vtproto's Size()+MarshalTo() pattern — vtproto's 1-alloc encode is what gives it 13-20x throughput over us at 1KB-10KB Person.
2. Single-pass with backpatched length varints: write tag + 1-byte placeholder for the length, recurse, fill in (memmove if final length >= 128). Skips the size pass (~3 us at 1KB).

Implementation can live in codegen (mode=full) — emit a _encode_to_buf per message that takes (data, ibuf, offset) and returns new_offset, then a public Person_encode that wraps it with the buffer allocation. Constraint: per-field closure count must stay <= current pb.encode writers count, else the per-field dispatch cost re-emerges (drm: two prior ibuf prototypes both regressed by ~2x because of dispatch).

Relates to h8v (codegen FFI direct writes) which is the per-field building block this work composes.

drm notes have the deeper analysis including why M6 ibuf path naive byte-write fails (20x interpreter dispatch overhead). The codegen approach sidesteps that by inlining all writes at codegen time so there's no runtime closure dispatch.


Decoder: emit table.new(N, 0) for repeated-field lists

Decode result table rehashes as nested fields populate. u39 covers table.new(0, N) for the message table itself, but per-field list tables (result.emails = {}, result.lucky_numbers = {}, ...) are also bare {} and re-hash as they grow. Worth a follow-up: emit table.new(N, 0) for repeated lists where the encoded count is recoverable from a single quick scan (count tag occurrences for non-packed, or read the LEN prefix and divide by per-element size for packed). Quick scan cost vs allocation savings is the tradeoff to measure — on the 1KB Person fixture, 26 emails + 5 packed lucky_numbers means 2 re-hash cycles per repeated list. Pairs with u39 to fully close the rehash-on-grow alloc pattern.

See bench/COMPARISON.md and bd show drm. For packed scalars where per-element size is fixed (fixed32/fixed64/float/double), count = payload_len / elem_size — O(1). For varint-packed and non-packed repeated, count requires a scan; might still be net-positive on payloads with >8 elements but needs measurement.


Repo: unstealth Beads tracking files

Make the repo-local Beads state visible to Git by removing the local .git/info/exclude rule for .beads/. Keep .beads/.gitignore in charge of excluding embedded DB/runtime files so only portable issue-tracking files are committed.

Removed the local .git/info/exclude rule that hid .beads/ from Git. Portable .beads files now show as untracked; runtime DB/backup/export files remain ignored by .beads/.gitignore.


C accel: conformance and parity gate for C paths

Extend tests/benches so C runtime and generated-C modes must pass the same luatest, interop, JSON/text, proto2, lazy passthrough where applicable, and Google conformance suites as Lua modes. Add bench output columns for lua-full, lua-runtime, c-runtime, c-generated, starwing. C acceleration must remain byte-equal with current encoders and preserve unknown fields/extensions/groups.

Architecture finalized in docs/c-accel.md. Parity strategy: REUSE existing test suites — no separate Lua-vs-C diff harness. (1) just test (luatest, 639 tests already parameterized over full/runtime modes) runs once with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Both must pass. (2) just conformance (Google proto3 + proto2) runs both modes. (3) Interop fixtures (test/interop/fixtures/*.bin) cover both modes via the existing parity.full_vs_runtime tests; extend the matrix to also cover c-runtime. Logic: every assert is against a reference output (golden bytes, txtpb, conformance result). If Lua passes and C passes, both equal the reference, so Lua == C by transitivity. No new test infrastructure required. bench/bench.lua gains a c-runtime column.


C accel: C table and string handling strategy

Design and implement the low-level C helpers for Lua table access and string/buffer handling. Cache field-name strings/registry refs in descriptor plans, pre-size result tables where possible, build output strings with luaL_Buffer or exact-size allocation, avoid per-field Lua stack churn, and handle 64-bit cdata consistently with the current Lua API.

Architecture finalized in docs/c-accel.md. Strategy decisions validated by bench/c_accel/ spike: (1) Field-name strings cached as luaL_ref slots at plan-compile time; each decode does lua_rawgeti from the cached ref instead of re-interning. (2) Output buffer: 4KB stack-backed cap with malloc promotion on overflow (pattern in bench/c_accel/person_codec.c — validates against tiny-msg overhead and big-msg correctness). (3) Result tables pre-sized via lua_createtable(0, n_fields) from descriptor stats. (4) Per-field stack indices cached for repeated/packed array fields for the duration of decode_message — naive lazy-getfield was 2x slower at 100KB (spike Phase B). (5) 64-bit cdata via luaT_pushuint64/luaT_pushint64 (Tarantool extensions in module.h). (6) Sub-messages use separate buf with stack-allocated header; backpatching avoided (the bench showed sub-buf approach is fast enough).

C-side strategy formalized in docs/specs/c_accel_strategy.md. Spec covers: pb_plan / pb_plan_field struct layout, field-name luaL_ref caching (spike-validated), 4 KB stack-backed pb_buf with malloc promotion, lua_createtable(0, n_fields) result pre-sizing, cached per-field stack indices for repeated/packed (spike Phase B: 2x speedup at 100 KB), 64-bit cdata via luaT_pushint64/checkint64, sub-buffer (chosen) vs backpatching (deferred) for sub-messages, two-pass packed encode, map iteration via lua_next, oneof grouping, unknown field passthrough, deferred micro-opts (manual varint inlining, SIMD scan), C99/-O2 build environment, deferred PB_C_TRACE/PB_C_CHECK_PARITY env knobs. ra6/mq7 references this spec for low-level decisions.


C accel: preserve pure-Lua fallback and public API compatibility

Define compatibility boundaries for C acceleration. Existing generated Lua modules and runtime pb APIs must keep working without a compiler or C module. New generated-C mode should be opt-in initially. Errors, 64-bit cdata behavior, WKT shapes, unknown fields, extension representation, lazy decode API, and text/JSON integration must remain compatible.

Architecture finalized in docs/c-accel.md. Concrete contract: (1) PB_ENABLE_C=1 env var is the only activation switch, default off. (2) require('pb') returns the same Lua surface in both modes — no API change. (3) 64-bit ints stay LuaJIT cdata (int64_t/uint64_t) in both modes. (4) WKT shapes unchanged. (5) Unknown fields round-trip identically. (6) Errors land as the same Lua error types. (7) Pure-Lua install must work without a C compiler — rockspec opt-builds the C module, install never fails on a host without cc. (8) When C module fails to load or PB_ENABLE_C is unset, runtime/pb/init.lua transparently uses the pure-Lua path.

Compatibility contract formalized in docs/specs/c_accel_compat.md. Spec covers: PB_ENABLE_C=1 single activation switch, silent fallback, full public surface preservation (pb.encode/decode/decode_lazy/parse/from_pb/json/text/grpc), generated module wrappers, 64-bit cdata invariant, WKT shapes, unknown fields, extensions, error compatibility, install paths (pure-Lua and C-enabled), ABI versioning, conformance via existing-suite re-run, explicit out-of-scope list. Lazy view stays Lua-only per spike rationale. Downstream tasks (mq7, ra6, 43t, c0i) reference this spec for compat boundaries.


C accel: build and packaging support for C modules

Add build-system and packaging support for C acceleration. Needs Justfile targets, rockspec support for compiled modules, platform naming, local dev build, CI matrix integration, and fallback when the C module is absent. Generated C backend must not make pure-Lua install impossible unless explicitly selected.

Architecture finalized in docs/c-accel.md. Concrete deliverables: (1) Justfile recipe 'just build-c' that builds the C runtime module (mirror bench/c_accel/Makefile auto-detection of TT_INC). (2) Rockspec optionally builds the C module — install on host without cc must succeed and produce a pure-Lua install. (3) runtime/pb/init.lua does pcall(require, 'pb.c_runtime') only when os.getenv('PB_ENABLE_C') == '1'. (4) Sourcehut CI build manifest gains one extra job that runs the full test+conformance suite with PB_ENABLE_C=1 set — single .build.yml, both modes covered on every push to master. (5) CI also runs one variant without the C module compiled to confirm pure-Lua install works.

Build and packaging scaffolding landed: docs/specs/c_accel_build_packaging.md spec, runtime/pb/init.lua PB_ENABLE_C=1 opt-in pcall hook (exposed as pb.c_runtime field), Justfile build-c/clean-c recipes (stub erroring cleanly until bd-ra6 lands runtime/pb/c/), Justfile lua_cpath constant + LUA_CPATH on test/test-one, .gitignore for compiled .so/.dylib, sourcehut .builds/{pure-lua,c-enabled}.yml manifests (ubuntu/noble, parallel jobs via separate files since sourcehut has no matrix), .sourcehut/conformance.yml for manual conformance triggers. Rockspec changes deferred to first C-source landing (described in spec). All 745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1 (silent fallback verified).


Release: tagged releases + rockspec + Go plugin binaries

Once the CI pipeline is green, ship tagged releases. Rockspec for the Lua runtime (publishable via tt rocks). Pre-built protoc-gen-tarantool binaries for darwin/arm64, darwin/amd64, linux/amd64, linux/arm64. Sourcecraft has 'sc release' tooling (see sc-release skill) for the release workflow itself.


Release: Sourcecraft.dev project + CI pipeline

Set up the canonical sourcecraft.dev project for the repo and a CI pipeline. Matrix: Tarantool 2.11 (CE+EE) and 3.x (CE+EE), Linux + macOS. Targets to run: just gen, just test, just bench-compare (alloc regression gate), just conformance (gated on the cached Docker image — see related CI wire-up bead). The repo lives under ~/data/home which by convention publishes to sourcecraft.dev (not github).


CI: wire up conformance suite with cached Docker image

The Docker image build (docker/conformance.Dockerfile) is the long pole at ~10-15 min on a clean cache. A registry push from a scheduled job would let CI runs reuse a warm image. Today the conformance suite runs locally via 'just conformance' but isn't gated on pushes. Goal: every push to master runs the binary+JSON and text-format suites; PRs run the same. Pre-requisite for M8 sourcecraft setup (this defines what the CI pipeline runs).


Profile the 1 KB decode cliff in hello.Person

bench/bench.lua shows hello.Person decode dropping 260 MB/s @ 100 B to 128 MB/s @ 1 KB — per-MB throughput halves at the payload-shape transition (scalars-only -> emails[] + nested address + packed lucky_numbers). starwing's C decoder scales smoothly across the same transition. Smoking gun: a trace abort or side-trace stitching failure at the array/nested branch. Capture jit.dump=tbim over the transition, identify the abort, fix it. Likely small change once located. Prereq for several decoder fixes — pin down what we're actually hitting first.

2026-05-17 inspection vs starwing/lua-protobuf 0.5.3 on Tarantool 3.8.0: reproduced starwing hello.Person encode at 75/688/888/1391/1431 MB/s and decode at 11.5/101.6/378/698.7/702.8 MB/s for 10B/100B/1KB/10KB/100KB. Current full mode is encode 30/257/287/541/643 MB/s and decode 30.7/247.5/128.8/186.4/184.3 MB/s. So we beat starwing on tiny decode, but lose 2.9-3.8x once Person switches to repeated strings + nested Address + packed ints. jit_trace passes including multi-byte varint paths; no fatal abort or bridge smoking gun. Focused probes on 930B Person: generated full decode ~6.9-7.5 us/op; scan tags+skip_field ~3.6 us/op; scan tags/lengths only ~2.1 us/op; hand order-specialized decoder with same string extraction/Address decode/lucky parse ~4.3 us/op. Patching decode_string to skip UTF-8 improves only ~16% (7.1 -> 6.1 us/op), so UTF-8 is material but not the main cliff. Likely cause: per-field Lua dispatch/function-call/table growth overhead repeated across ~30 LEN/VARINT fields; C starwing parses from pointer/buffer and lands near our tag-only scan cost. Existing follow-ups bgu (ffi.string string decode), gcy (inline nested decode), and 0u1 (typed decoder byte fast paths) are relevant, but the largest remaining gap likely needs generated ordered/specialized decode loops or ptr+offset readers that avoid decode_tag/decode_len helper dispatch and substring slicing.

Profiled against starwing and local focused probes. No fatal JIT abort found; cliff attributed mainly to repeated Lua tag/field dispatch plus LEN/string slicing/validation. Follow-up fast-path work captured in tarantool-protobuf-4kj and existing decoder tasks.


Encoder: parallel-array iteration for proto2 extensions

Mirror the desc.oneofs_list finalize-time pattern: build desc.extensions_list once in pb.finalize_message, iterate with 'for i=1,#' in encode instead of walking the descriptor's hash table on every call. The 4-7x proto2 gap (vs 2-4x proto3 in bench/starwing_bench.lua) is partly explained by hash-walked extensions on the hot path. Expected: brings proto2 BenchPayload encode closer to proto3 Person encode at the same shape.


Decoder: ffi.string(ptr+off, len) for string/bytes decode

Replace buf:sub(npos, npos+len-1) in decode_string/decode_bytes with ffi.string against a cached ffi.cast('const uint8_t*', buf). Removes method-call dispatch and lets the JIT fuse the read with surrounding code. Hits hardest on repeated-string fields (26 emails strings in 1KB Person -> 26 sub calls today). Watch for LuaJIT string-intern collision behavior to stay identical to sub().


Decoder: emit table.new(0, N) for result tables

Codegen knows the field count of every message at emission time. Emit table.new(0, N) instead of {} so the result table is sized correctly from the start. Cheap, broad win — especially closes the 1KB cliff where hello.Person rehashes as nested address + repeated fields populate. LuaJIT-only (table.new is in require('table.new')).

Attempted and reverted. table_new(0, N) for decode result tables passed tests + JIT but regressed small-payload decode (10B -15.4%, 100B -17.7%) because the call cost exceeds rehash savings at small scale, and the small case doesn't reach the first rehash. Large-payload decode flat. Confirms drm bug: alloc is not the throughput bottleneck. See bench/PERF_LOG.md.


Encoder: codegen-emitted packed-scalar tight loops

Replace per-element wire.encode_int32 calls with one inlined varint-emitting loop per packed field. Applies to packed int32, int64, sint32, sint64, bool, enum. The 1KB Person has 5 packed lucky_numbers + 26 emails — current per-element function-call boundary costs add up. Expected: 30%+ on packed-heavy payloads.


Encoder: optional caller-owned ffi.cdata ibuf API

Separate API surface, not the default. ibuf-based encoder that writes into a caller-owned ffi.cdata buffer instead of returning a fresh Lua string. Targets hot RPC paths where the caller already owns a reusable buffer (e.g. net.box send path). Independent of the codegen rewrite — different API contract. Earlier attempt parked in stash@{0}; revisit after the codegen-time emission lands so we can compare apples-to-apples.


Encoder: two-pass with exact size precomputation

Walk fields once to sum byte sizes, allocate the final string at exact size, walk again to write. Eliminates buffer-grow realloc; one lua_pushlstring. Same pattern as vtprotobuf's Size() + MarshalToVT. Stacks with the codegen-time inline writes (depends-on). Expected: additional 30-50% on large-message encode beyond the inline-FFI baseline.


Inspect: ~120 B/op fixed encode/decode overhead in Lua hot path

Cross-runtime bench (bench/COMPARISON.md) shows a fixed per-call
allocation floor on both encode and decode in `mode=full`:

- encode: 136 B/op even at 10 B payload (output is 10 bytes; ~126 B overhead)
- decode: 112 B/op even at 10 B payload (top-level table = 2 hash-tables)

For comparison, Go vtproto runs encode at 16 B/op and decode at 8 B/op
on the same 10 B fixture.

Inspect:
  1. What's the source of the 100+ B encode overhead? Suspect candidates:
     - per-call buffer.ibuf workspace allocation in wire.lua
     - Lua string concat building the result
     - GCstr header on the output string itself (~24-32 B)
  2. Decode floor: 112 B/op. Likely 2x table headers + small alloc for
     the top-level message table.
  3. For large payloads (100 KB encode = 131605 B/op vs output 96674 B)
     the overhead is ~28 KB. Is that one big realloc tail, or many small?

Outcome: a writeup pointing to specific lines and a recommendation
on whether the M6 ibuf path would actually help.

Findings from bench/alloc_probe.lua:

ENCODE 10B floor = 136 B/op is the `out` Lua table:
- `local out, n = {}, 0` alone: 64 B (Lua GCtab base)
- After 5 array entries: 136 B (matches the encode floor exactly)
- table.concat: 0 B in the bench because output bytes intern
- Varints: 0 B per call because 1-byte string.char outputs intern
  globally (small string dedup)
- 2-byte varints DO allocate: ~33 B per fresh value (encode_varint_slow
  path returns a fresh string from string.char + bit.bor)

DECODE 10B floor = 112 B/op is the top-level result table
{name=..., age=...} — same shape across iters means it'd allocate the
same in a real workload.

LARGE PAYLOAD (1KB Person):
- same input (output interned): 1368 B/op
- varying age (output unique): 2368 B/op
- Delta = ~1000 B is the result string for 930 bytes of output (the
  GCstr header + 930 content; the extra ~70 B is presumably padding /
  alignment / the Address's nested concat).
- So real cost per call has TWO components: (a) the small fixed tables
  for the encoder workspace, (b) the output bytes themselves.

KEY INSIGHT: bench numbers UNDERSTATE real allocation. The bench
iterates the SAME input → output string interns → bench reads only
the table cost. Real workloads where every message is unique pay
output-size + table-cost.

OPTIMIZATION RANKING:
1. Highest ROI: encoder workspace table. ~136 B per top-level encode
   + ~136 B per nested message encode (Address adds its own). For a
   1KB Person we have ~5 nested encoders → ~700 B of tables. Replacing
   the array-of-string-pieces with a single growable `buffer.ibuf` cuts
   this to ~0. This is the M6 ibuf path.
2. Medium: encode_varint_slow returns a fresh string on each call for
   non-fast-path values. Tag bytes are precomputed as literals; only
   payload varints hit this. Inlining the slow path into wire.lua's
   hot caller (or returning into a passed-in buffer) drops this.
3. Lowest: result string. Unavoidable for the encoder's API contract
   (returns a string). Only the lazy path avoids it.

DECISION POINT: M6 ibuf path (already prototyped, deferred per
memory/decode_perf_deferred.md) is the right intervention.
Conservatively halves encode B/op on small messages, larger savings
on nested-heavy payloads.
CORRECTION to earlier note.

Recommendation to "use M6 ibuf path to cut the 136 B/op floor" was
wrong. Per memory/tarantool_ibuf_perf.md (verified against current
repo state — feature NOT in HEAD):

- Naive per-byte b:alloc(1): ~20× slower realistic, ~36× synthetic
- Two-pass bulk-reserve: byte-equal correct, ~2× slower at every
  payload size. Sitting in git stash@{0}.

Both prototypes lose on wall time because:
  (a) Bench harness runs with jit.off, closure dispatch interpreted
  (b) Realistic encoder cost is dominated by per-field closures
      (sizer + writer + emit_tag + pwrite_*), NOT byte writes
  (c) Two-pass adds an extra walk on top

Also: the 136 B/op encode floor is NOT the throughput bottleneck.
Cross-runtime gap (5× apiv2, 13-20× vtproto) is JIT/dispatch
overhead per field, not allocator pressure. Cutting 136 B doesn't
close that gap.

VIABLE PATHS (neither tried):
1. Single-pass with backpatched length varints. Walk once. For nested
   msgs: write tag + 1-byte placeholder, recurse, fill in (or memmove
   if final length ≥ 128). Eliminates the size pass (~3 µs at 1 KB).
2. Codegen-time `_encode_ibuf` per message in protoc-gen-tarantool
   (mode=full). Straight-line ibuf writes, no descriptor walk at
   runtime. Mirrors what pb.encode already does for the table path.
   Probably the only approach that actually closes the throughput gap.

Either has to keep per-field closure count ≤ pb.encode's writers,
otherwise we re-introduce the dispatch cost both stashed prototypes
fell on.


bench/*.lua: apply mcode arena hardening + re-snapshot baseline.json if numbers shift

Sibling to 3o2 (closed). The trace gate had an intermittent 'fails silently with no JIT' mode on macOS arm64 caused by the default mcode arena being too small for our codegen footprint. The fix in 3o2 added jit.opt.start('sizemcode=64','maxmcode=4096') to bench/jit_trace.lua and 20/20 runs are now stable.

The other bench scripts have the same risk and none have the fix:
  bench/bench.lua
  bench/lazy_bench.lua
  bench/profile.lua
  bench/shapes_bench.lua
  bench/starwing_bench.lua
  bench/wire_bench.lua
  bench/alloc_probe.lua

These scripts have larger codegen footprints than the trace gate (they require more modules, run for longer, and accumulate more traces), so they're MORE likely to hit the same intermittent JIT-fails-silently mode than the gate was. When that happens the script reports throughput that includes interpreter-only iterations — underreporting the real numbers without any diagnostic.

Concrete steps:
1) Add the jit.opt.start line to each script (same comment block as 3o2, or factor into a tiny bench/_setup.lua included from each).
2) Re-run bench/bench.lua --baseline to refresh bench/baseline.json.
3) Re-run bench/bench.lua --print and update the MB/s tables in bench/COMPARISON.md if any number moved >5%.
4) bench/COMPARISON.md notes 'Numbers will drift run-to-run by 5–10%' — verify that variance band shrinks after the fix.

alloc_probe.lua doesn't need it (allocator counters don't depend on JIT), but adding the line costs nothing and keeps the bench/ scripts uniform.


Quantify and address Person_decode multi-byte varint side-trace bridge (intermittent ~1/5 runs)

Trace-topology finding from bench/jit_trace.lua. The 'full/Person_decode multi-byte varint' check intermittently reports one bridge — a side trace whose linktype=interpreter, costing interp dispatch per multi-byte tag/length-prefix on the hot decode path.

Captured output (1–2 runs out of 10):

  [ OK ] full/Person_decode multi-byte varint  (stops=10, bridges=1)
         info: bridge tr9  side-of tr5  hello_pb.lua:997  pc=55

tr5 is the Person_decode while loop (entry at hello_pb.lua:997). pc=55 falls inside the wire.decode_tag inlined fast path: the guard 'if b < 0x80' fails on a 2+ byte tag, exits to side trace tr9, which contains the multi-byte continuation loop but can't self-link back to the parent — drops to the interpreter to walk the rest of the dispatcher and re-enter on next iteration.

Existing context:
- wire.lua duplicates the 1-byte varint fast path at every hot decode call site precisely because LuaJIT side traces can't stitch returns from an inlined helper frame. That works for the 1-byte case. The 2+ byte case still calls decode_varint() (the multi-byte fallback) which has its own internal while loop.
- gcy (inline nested decode) and kot (localize wire.* upvalues) are the structurally related items already filed; they don't cover this specific bridge though.

Why P3 (not P2):
- Intermittent (~1/5 runs in the gate). The trace topology is mostly stable.
- The multi-byte tag path is < 5% of typical RPC payloads (field IDs 1..15 = 1-byte tag, lengths < 128 = 1-byte length). Larger impact would require >127-byte fields or field IDs >= 16.
- The 'multi-byte varint' fixture in bench/jit_trace.lua (200-byte name + lucky_numbers including 200000, 500000) was added specifically to expose this — and it does, intermittently. The intermittency is the JIT settling on different trace shapes across runs.

Concrete approaches to investigate:
1) Inline a 2-byte varint fast path inside decode_tag (and decode_string LEN prefix, etc.) — 'if b < 0x80 then ... elseif b2 < 0x80 then ...' — keeping the 3+ byte case in the fallback. Covers field IDs up to 4095 and length prefixes up to 16383, which is almost all real payloads.
2) Profile-driven: run bench/jit_trace.lua 100x with a fixed seed, collect bridges by location, and decide whether the intermittency rate justifies (1) at all.

Acceptance: 50 consecutive runs of bench/jit_trace.lua report bridges=0 for full/Person_decode multi-byte varint, OR a measured throughput improvement on the bench at the multi-byte-varint fixture.


Encoder: fiber-local recycled output buffer (Tarantool-specific)

The 136 B/op encode floor is the 'local out, n = {}, 0' workspace table allocated per top-level encode call. Tarantool ships a per-fiber ibuf via require('buffer'); could grab one from a pool, reset it on each top-level encode, write into it, then ffi.string(ibuf.rpos, ibuf:size()) at the end.

Tradeoffs:
  - Saves the workspace alloc (~136 B per top-level call + ~136 B per nested message).
  - Adds fiber-local state — non-Tarantool LuaJIT runs would need a different path or none.
  - Subsumed by lkz (single-pass two-phase) which goes directly to a single buffer without intermediate strings. Worth filing as a smaller alternative path in case lkz proves too invasive to land.
  - drm notes record two prior ibuf prototypes (stashed) that regressed by ~2x because per-byte b:alloc(1) was 20x interpreter-dispatch-bound. This proposal avoids that pitfall only because it composes with h8v (codegen FFI direct writes that bypass alloc(1)).

Concrete path: only useful in combination with h8v. Without h8v this would just shift the alloc from 'out table' to 'output string' without removing any per-string varint allocation in the field bodies — net neutral on alloc and worse on speed.

Sketch: shared scratch buffer per encoding, capacity exposed as ffi cdata pointer + length. Codegen emits 'local _buf = pb.codec.acquire_buf(); local _off = 0; ...; return pb.codec.finalize_buf(_buf, _off)'. acquire_buf returns a pre-allocated buffer of growing capacity; finalize_buf returns a string and recycles. Care needed for recursive encode calls (nested message encoding into the same buffer).


Decoder: investigate field-id dispatch as binary search or jump table

Generated Type_decode bodies use linear 'if id == 1 elseif id == 2 ...' chains. For Person (~15 fields) the chain is short and the comparisons cheap. For messages with 30+ fields the linear walk to a high-id tag costs N comparisons per occurrence.

Open question: does LuaJIT's IR already lower this to a switch/jump? If yes, no work needed. If no, three options:

  1. Codegen-emit a balanced if-elseif tree (binary search) when field count exceeds threshold.
  2. For messages where field IDs are dense and small, emit a numeric branch table: 'local _f = _dispatch[id]; if _f then return _f(buf, pos, result) end'. Closure-per-field has setup cost but avoids the dispatch cost on every tag.
  3. Sort by frequency (impossible to know at codegen time without profiling input). Skip this option.

Pre-work: jit.dump on Person_decode to check whether LuaJIT collapses the if-elseif. If it does, close as won't-fix; if it doesn't, decide between (1) and (2).

Lowest priority of the perf items because: (a) we don't know it's a problem yet, (b) Person profile shows decode_string + decode_tag + utf8 dominate, not the if-elseif walk.

Recent profile (Person 1KB): line 1007 (decode_tag call) is 14% of generated body time, and line 1021 (list append) is 24%. Linear if-elseif walk doesn't show up as a hotspot for Person, but Person has only 15 fields. Needs a bigger fixture to manifest.


Codegen: chunk oversized generated decode bodies into per-range helpers

test_messages_proto3_pb.lua is 4212 lines — the conformance fixture has so many fields that a single Type_decode body exceeds LuaJIT's inline budget and trace size limits. Symptom (to verify): bench/jit_trace.lua trace aborts on the big conformance message, despite no NYI bytecodes in the body.

Fix: when a message has >N fields (threshold to find empirically; LuaJIT defaults LJ_TRACE_MAX_BC=8000), codegen splits the if-elseif chain into per-range helpers: _decode_fields_1_to_15, _decode_fields_16_to_31, etc. The top-level Type_decode dispatches to the right chunk by id range:

  if id <= 15 then
    pos = _decode_fields_1_to_15(buf, pos, id, wt, result)
  elseif id <= 31 then
    pos = _decode_fields_16_to_31(buf, pos, id, wt, result)
  ...
  end

Each chunk is small enough to JIT-compile cleanly. Adds one function call per tag, but only on messages large enough to need it — small messages stay inline.

The range size and split threshold need measurement. Start with: split when message has >32 fields, into chunks of 16. Measure throughput on conformance message before/after to validate the trade.

Pre-work: confirm the abort is real by running bench/jit_trace.lua against a conformance-like fixture and checking the FATAL traceerr codes. If LuaJIT compiles the 4212-line body fine, this is a non-issue.


Codegen: emit wire.encode_varint directly for int32/int64/uint32/uint64 instead of typed alias

runtime/pb/wire.lua aliases M.encode_int32 = encode_varint (and same for int64/uint32/uint64). Generated code calls wire.encode_int32(v) which resolves to encode_varint through two table lookups — wire.encode_int32 (one hash lookup), then the alias resolution. LuaJIT may collapse this when the trace stays hot, but each break re-incurs both.

Codegen can emit wire.encode_varint directly for the four unsigned-varint scalar types (int32/int64/uint32/uint64) since they're literally encode_varint with different names. Skips one alias indirection per varint encode. Pairs naturally with kot (localize wire.* upvalues) — together they reduce the call to a direct LJ_FUNCC dispatch with no name lookup.

Same applies to bool (encode_bool = encode_varint with v and 1 or 0 wrapper) and sint32/sint64 (zigzag wrapper) — codegen could inline the wrapper logic at the call site for sint, but that overlaps with h8v (FFI direct writes) which subsumes the question.

Codegen-side change in protoc-gen-tarantool. Constraint: the alias provides the typed encoder slot in TYPE_INFO that runtime-mode encoders walk — those aliases must stay. Only the generated mode=full code changes.


Decoder: int64/uint64 return Lua number when value fits in 2^53

decode_int64/decode_uint64 always return ffi cdata, forcing a fresh cdata allocation per call. For values in [-2^53, 2^53) (the common case for IDs, sequence numbers, timestamps fitting in 53 bits, byte counts, etc.) a Lua number is exactly representable and skips the cdata header allocation.

Proposed shape: opt-in variant decoder, since changing the default breaks any caller that does type(v)=='cdata' or relies on cdata-only operators. Two API options:

  1. Per-descriptor flag (desc.int64_as_number = true) wired in via a codegen option or generator flag. Generated code emits a different decoder fn.
  2. Separate typed decoders (wire.decode_int64_n / decode_uint64_n) that callers opt into explicitly.

Watch case: values exceeding 2^53 must still return cdata (with a runtime branch). The branch cost only pays off if cdata allocation cost > one comparison, which it is on hot paths.

Bench impact bounded by how many int64 fields the workload has. For hello.Person, user_id (fixed64) is the only one — so impact on this fixture would be ~1-2%. Bigger win on protobuf workloads dominated by timestamps and sequence numbers (datastore RPCs, log streams).


Docs: remove PLAN.md after Beads migration

PLAN.md has been converted from active roadmap to duplicate design-history prose now that all actionable work is tracked in Beads. Remove the file or replace remaining references with Beads/README pointers so project state has a single durable task source of truth.

Removed PLAN.md and retargeted remaining references to Beads or concrete docs.


C accel: generated C codec backend for mode=full

Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -> string and M.Type_decode(bytes) -> table. Goal: one Lua->C call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.

DEFERRED per docs/c-accel.md. Spike (04c) showed S4 hand-written C had ≤15% headroom over S3 (generic C runtime, ra6) and went the wrong way at scale (S4 *slower* than S3 at 10KB and 100KB encode). Codegen complexity not justified by current numbers. REVIVAL CRITERIA: a measured real-workload shape where ra6's per-field dispatch costs ≥25% over hand-written for that shape, demonstrated with a microbenchmark, AND the affected workload is on a hot path for a real user. Likely trigger shapes: wide messages with many optionals, heavy oneof use, complex maps, deeply nested (5+ levels) hierarchies. When triggered: write 2-3 paragraphs documenting shape and numbers, re-open this issue, scope narrowly via desc.encode/desc.decode override (same mechanism as WKT). If 6 months after ra6 ships no trigger fires, close as 'not justified'. Now blocked by ra6 (so it surfaces in ready list only after ra6 implementation lands).


Tooling: add formatting targets for Go and Lua

PLAN.md section 6 lists gofumpt and stylua. Add formatter configuration and Justfile targets, with generated code excluded unless the generator itself is intended to emit stylua-compliant output.


Tooling: add Go and Lua lint targets

PLAN.md section 6 lists golangci-lint for Go and luacheck for Lua. Add configured lint targets and decide whether they run locally only or in CI. Keep generated files excluded where appropriate.


Tooling: add coverage targets for plugin and runtime

PLAN.md section 6 lists go test -cover for the Go plugins and luacov for the Lua runtime. Add repeatable coverage targets and document expected use. Decide whether coverage is informational or gated in CI.


Docs: example apps (pet-clinic CRUD + queue replication)

Two new runnable examples to live alongside the existing examples/grpc + examples/dynamic + examples/http set. (1) pet-clinic CRUD over gRPC — Greeter-shaped service exposing CRUD on a small entity (Pet, Owner). Covers unary + server-streaming list. (2) Replication of state via protobuf-encoded events on a queue — events serialized via pb.encode, dispatched via tarantool/queue, consumed and decoded by a follower. Each example must be hermetic per CLAUDE.md (memtx/wal/log in /tmp). Each gets a matching how-to in docs/howto/.


Docs: migration guide from built-in protobuf to pb

Tarantool ships an encode-only require('protobuf') module. Document the migration path for users currently on the built-in. Cover: API surface differences (no decode in builtin; pb has encode+decode+JSON+text+gRPC+WKT), 64-bit cdata convention parity (both use cdata, no surprise), field-name conventions (both snake_case), the rename rationale (we couldn't override 'protobuf' without breaking existing users). Concrete before/after code samples.


gRPC: reference network transports (http_server, netbox, http_client_unary)

Ship reference transports as separate Tarantool modules (deferred from M4). Designs in docs/specs/grpc_transports.md. Three transports: (1) pb.grpc.transport.http_server — Connect-JSON over HTTP/1.1 via tarantool/http; default external transport; works with browsers and curl without an HTTP/2 proxy. (2) pb.grpc.transport.netbox — gRPC tunneled over net.box calls; first-class in-cluster path. (3) pb.grpc.transport.http_client_unary — outbound unary only via http_client. HTTP/2 termination explicitly out of scope; the transport contract is HTTP/2-shaped so the same generated code works behind Envoy. Conformance anchor: connectrpc/conformance harness (memory: connect_conformance_suite).


Decide lazy-default for repeated string|bytes fields

Open design question. For routing/proxy workloads that decode but don't iterate every element, repeated LEN fields could stay as unread ArrayViews until first access. Eager callers pay one extra table alloc; lazy callers skip N per-element string allocations. At 100KB Person, 26+ emails = 26+ string allocations deferrable. Decide: (a) flip default for repeated LEN, (b) gate behind a codegen mode/option, (c) leave eager as default and rely on _decode_lazy. Surface compatibility implications (callers iterate with ipairs today and expect a real array) before deciding.


Decoder: decode_len-style byte-then-varint symmetry in typed decoders

The single-byte peek shortcut in decode_len (wire.lua:410-419) isn't replicated in decode_int32 / decode_uint32 / decode_int64 / decode_uint64 / decode_bool / decode_enum, which always call decode_varint(). Mirror the fast path in each typed decoder so the dominant 1-byte varint case skips a function call. Expected: 5-10% on scalar-heavy decodes. Memory: wire_2byte_varint_cliff (same shape, encode side, already fixed).


Encoder: static type-elision on scalars in mode=full

wire.to_uint64(v) does a type(v) dispatch in three branches (number/cdata/boolean); in mode=full codegen the field's static type is known. Emit UINT64(v) / INT64(v) directly at the call site and let LuaJIT type-guard the trace. Expected: 10-15% on scalar-heavy messages.


Encoder: inline zigzag_encode32 into encode_sint32 (one function call vs two)

encode_sint32(v) = encode_varint(zigzag_encode32(v)) — two function calls. Inline as:

  local function encode_sint32(v)
      local n = bit.bxor(bit.lshift(v, 1), bit.arshift(v, 31))
      -- existing 1-byte fast path
      if n >= 0 and n < 0x80 then return string.char(n) end
      return encode_varint_slow(n)
  end

Same for encode_sint64 (with the int64 zigzag formula).

Tiny win per sint encode — only matters for sint-heavy payloads. Filed for completeness; h8v makes this moot for mode=full (FFI direct writes inline the whole thing anyway). Real applicability: runtime mode encoders.

If h8v lands first, this becomes runtime-mode-only and may not be worth the duplication. Defer the call until h8v ships.


Feature: gRPC server reflection service

PLAN.md section 8 lists gRPC server reflection as a non-goal until after M5. M5 is complete, so keep this as deferred future work. Implement standard reflection once real transport/server use cases require it.


Feature: Tarantool 3.x config integration for schema registration

PLAN.md section 7 mentions a pb.types.<name> declarative config role as a possible Tarantool 3.x integration. Define the use case first, then design how protobuf schemas/descriptors are declared, loaded, validated, and made available to application roles.


Docs: Go plugin API documentation generation

PLAN.md section 6 lists pkgsite for Go docs while Lua docs remain manually maintained Markdown. Decide what Go packages need public documentation and add a repeatable docs target or release note flow for pkgsite/sourcecraft publication.


Feature: gRPC-Web / Connect conformance support

PLAN.md section 8 lists gRPC-Web as not on the v1 path and notes it would be covered by the same Connect conformance harness if added. Track future decision and implementation scope separately from the current Connect-JSON/http_server transport work.


Decision: module path mapping for package-less protos

PLAN.md section 7 notes the current behavior when lua_package and proto package are absent: use the bare filename. Decide whether to keep that behavior or error out to avoid unstable module paths. Update codegen docs and tests accordingly.


Feature: msgpack-flavored encoder for protobuf schemas

PLAN.md section 7 references docs/specs/msgpack_encoding.md. Decide and implement, if still desired, a msgpack-flavored encoder driven by protobuf schemas. Open design points: map keyed by field number vs field name, ARRAY layout for box.space feeders, and MP_TUPLE extension opt-in.


Decision: public C-FFI varint accelerator

PLAN.md section 7 asks whether to expose a public C-FFI varint accelerator. Deferred until benchmarks prove pure-Lua varint code is the bottleneck. Capture benchmark threshold, API surface, and fallback behavior before implementation.


Decision: deterministic map encoding option

PLAN.md section 7 asks whether map encoder should be deterministic by sorting keys. Spec does not require deterministic map order, but some users may need byte-stable output. Decide API shape, likely pb.encode_deterministic or an encode option, and document perf/canonicalization tradeoffs.


Testing: add malformed-input and random-valid fuzz harnesses

PLAN.md section 5 lists fuzz harnesses as not yet automated. Add durable fuzz coverage for malformed wire inputs plus random-valid generated messages. Goal: catch decoder crashes, infinite loops, overlong varints, invalid LEN boundaries, unknown-field preservation bugs, and encode/decode round-trip mismatches beyond fixed fixtures.


Protobuf editions support (deferred)

We advertise FEATURE_PROTO3_OPTIONAL only, not FEATURE_SUPPORTS_EDITIONS. protoc errors out on edition='2023' files with our plugin today. See PLAN.md section 4.6 for the full design — editions replaces proto2-vs-proto3 with per-field FeatureSet annotations (field_presence, repeated_field_encoding, enum_type, utf8_validation, json_format). Practical win is modest: the only knob proto3 doesn't expose today is per-field packed opt-out. Revisit when (a) someone needs per-field packed control, (b) conformance suite gates on editions, or (c) protoc deprecates proto3 syntax hard enough that real users hit the error.


Proto2: MessageSet wire format (deferred)

protobuf-go's protoreflect refuses to load a FileDescriptor declaring 'option message_set_wire_format = true;', calling it a removed proto1 feature. We strip the four MessageSet-flavored nested messages from our vendored test/conformance/proto/test_messages_proto2.proto so the rest of the schema compiles. Real support would require forking protoreflect; deferred until a real Tarantool consumer asks. Track here so we don't lose the context.