From 3042384b3161fb06cda88cccd98d8f94dd805782 Mon Sep 17 00:00:00 2001 From: Eugene Blikh Date: Tue, 19 May 2026 07:24:50 +0300 Subject: [PATCH] =?UTF-8?q?c-accel:=20arch=20prereqs=20=E2=80=94=20compat?= =?UTF-8?q?=20contract,=20C-side=20strategy,=20build=20scaffolding?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three companion specs under docs/specs/ formalize the boundaries established in docs/c-accel.md, unblocking bd-mq7 (descriptor → C plan compiler): * c_accel_compat.md (bd-47e) — pinpoints what must stay byte-equal between PB_ENABLE_C unset and =1: public surface, generated module wrappers, 64-bit cdata, WKT shapes, unknown fields, extensions, errors. Calls out the lazy-view exclusion. * c_accel_strategy.md (bd-z7x) — pb_plan struct layout, field-name luaL_ref caching, 4 KB stack-backed pb_buf, cached per-field stack indices (the 2× win from spike Phase B), sub-buffer over backpatching, map/oneof/unknown handling. * c_accel_build_packaging.md (bd-wky) — where the C module lives (runtime/pb/c/), how it builds, what the rockspec gains, the CI matrix shape. Scaffolding that lands now: * runtime/pb/init.lua — PB_ENABLE_C=1 opt-in pcall hook; the loaded module (or nil) is exposed as pb.c_runtime for introspection. Silent fallback when the module is absent. * Justfile — `build-c` / `clean-c` recipes (stub erroring cleanly until bd-ra6 lands runtime/pb/c/), new lua_cpath constant, LUA_CPATH wired through `test` and `test-one`. * .builds/{pure-lua,c-enabled}.yml — sourcehut CI manifests, one per activation mode (sourcehut has no matrix; parallel jobs go in separate files). ubuntu/noble images. * .sourcehut/conformance.yml — outside .builds/ so it doesn't auto-submit; trigger manually with `hut builds submit` before releases. * .gitignore — runtime/pb/c_runtime.{so,dylib} and runtime/pb/c/*.o. 745/745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1 (silent fallback verified). Closes bd-47e, bd-z7x, bd-wky. Unblocks bd-mq7. --- .beads/interactions.jsonl | 5 + .beads/issues.jsonl | 26 +- .builds/c-enabled.yml | 34 ++ .builds/pure-lua.yml | 28 ++ .gitignore | 7 + .sourcehut/conformance.yml | 33 ++ Justfile | 30 +- docs/specs/c_accel_build_packaging.md | 229 ++++++++++++++ docs/specs/c_accel_compat.md | 310 ++++++++++++++++++ docs/specs/c_accel_strategy.md | 433 ++++++++++++++++++++++++++ runtime/pb/init.lua | 17 + 11 files changed, 1142 insertions(+), 10 deletions(-) create mode 100644 .builds/c-enabled.yml create mode 100644 .builds/pure-lua.yml create mode 100644 .sourcehut/conformance.yml create mode 100644 docs/specs/c_accel_build_packaging.md create mode 100644 docs/specs/c_accel_compat.md create mode 100644 docs/specs/c_accel_strategy.md diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index 3fd6a9fdd22ef3e6ef2f7af8d38e351bf5d064b4..b791f1c00d2b6950d399a930e40fa3a750c45bbd 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -13,3 +13,8 @@ {"id":"int-7e5b3fc8","kind":"field_change","created_at":"2026-05-18T19:43:59.346895Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-04c","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Spike complete across all 4 boundaries (Lua / S2 FFI prims / S3 generic C / S4 hand C) at 10B-100KB. Bench: bench/c_accel/. Decisive results. S3 ≈ S4 within 15% (S3 often faster on encode at scale). S2 LOSES vs pure-Lua at every size 1KB+ on encode (0.60-0.99x) and at every size on decode (0.29-0.39x). FFI cost decomposition (bench/c_accel/ffi_probe.lua, memory luajit-ffi-boundary-cost-2026-05-18): bare FFI = 33ns; pointer-return FFI = 73ns; ffi.cast on string = 156ns; libc memcmp = 60ns; pure-Lua varint decode = 75ns. Per-primitive boundary cost equals pure-Lua dispatch cost — only message-level C crossing wins. ARCHITECTURE for pf6: ship ra6 (generic C runtime, descriptor-walking, one C call per message); drop c0i (≤15% headroom over ra6); drop per-primitive FFI. Required ra6 impl pattern: cache per-field stack indices for repeated/packed arrays during decode_message (naive lazy-getfield 2x slower at 100KB). Full details in bench/c_accel/README.md and bd memory c-accel-spike-04c-final-2026-05-18 + luajit-ffi-boundary-cost-2026-05-18."}} {"id":"int-09acf873","kind":"field_change","created_at":"2026-05-18T19:44:07.276353Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-80t","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18."}} {"id":"int-4869e8ec","kind":"field_change","created_at":"2026-05-18T19:44:07.459433Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-1eq","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18."}} +{"id":"int-af4c1c4d","kind":"field_change","created_at":"2026-05-18T20:19:41.337001Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-c0i","extra":{"field":"priority","new_value":"3","old_value":"1"}} +{"id":"int-a4bcd105","kind":"field_change","created_at":"2026-05-18T20:23:01.540904Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-pf6","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Architecture chosen and documented in docs/c-accel.md (commit pending). Ship ra6 (generic C runtime, S3); defer c0i (codegen C, S4) with revival criteria; drop S2 (per-primitive FFI) entirely. PB_ENABLE_C=1 opt-in, default off. Parity via existing-suite re-runs with C enabled. Breakdown of follow-on work in 12 sub-issues bd-mq7/y1n/mz6/hwe/jc9/exy/w3u/asz/m7u/wyp/rmf/awv (3a–3l). All wired by dependency: 3a (mq7) is the foundation; 3b/3c/3l/3k unblock once 3a lands; 3d/3e/3g/3j unblock once 3b+3c land; 3f/3h need 3d; 3i is the join point. ra6 itself depends on all twelve. c0i blocked-by ra6."}} +{"id":"int-ba4ac242","kind":"field_change","created_at":"2026-05-19T04:23:56.245983Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-47e","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Compatibility contract formalized in docs/specs/c_accel_compat.md. Spec covers: PB_ENABLE_C=1 single activation switch, silent fallback, full public surface preservation (pb.encode/decode/decode_lazy/parse/from_pb/json/text/grpc), generated module wrappers, 64-bit cdata invariant, WKT shapes, unknown fields, extensions, error compatibility, install paths (pure-Lua and C-enabled), ABI versioning, conformance via existing-suite re-run, explicit out-of-scope list. Lazy view stays Lua-only per spike rationale. Downstream tasks (mq7, ra6, 43t, c0i) reference this spec for compat boundaries."}} +{"id":"int-951d03f7","kind":"field_change","created_at":"2026-05-19T04:24:12.141983Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-z7x","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"C-side strategy formalized in docs/specs/c_accel_strategy.md. Spec covers: pb_plan / pb_plan_field struct layout, field-name luaL_ref caching (spike-validated), 4 KB stack-backed pb_buf with malloc promotion, lua_createtable(0, n_fields) result pre-sizing, cached per-field stack indices for repeated/packed (spike Phase B: 2x speedup at 100 KB), 64-bit cdata via luaT_pushint64/checkint64, sub-buffer (chosen) vs backpatching (deferred) for sub-messages, two-pass packed encode, map iteration via lua_next, oneof grouping, unknown field passthrough, deferred micro-opts (manual varint inlining, SIMD scan), C99/-O2 build environment, deferred PB_C_TRACE/PB_C_CHECK_PARITY env knobs. ra6/mq7 references this spec for low-level decisions."}} +{"id":"int-7ce7c344","kind":"field_change","created_at":"2026-05-19T04:24:13.236096Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-wky","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Build and packaging scaffolding landed: docs/specs/c_accel_build_packaging.md spec, runtime/pb/init.lua PB_ENABLE_C=1 opt-in pcall hook (exposed as pb.c_runtime field), Justfile build-c/clean-c recipes (stub erroring cleanly until bd-ra6 lands runtime/pb/c/), Justfile lua_cpath constant + LUA_CPATH on test/test-one, .gitignore for compiled .so/.dylib, sourcehut .builds/{pure-lua,c-enabled}.yml manifests (ubuntu/noble, parallel jobs via separate files since sourcehut has no matrix), .sourcehut/conformance.yml for manual conformance triggers. Rockspec changes deferred to first C-source landing (described in spec). All 745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1 (silent fallback verified)."}} diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index a297507b0cde826d43f87d399bfe37ecb852eba8..72940b1e4aa8523eda449934ff933974fdb6c701 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,12 +1,23 @@ +{"_type":"issue","id":"tarantool-protobuf-exy","title":"ra6 3f: repeated strings + repeated messages (cached stack-idx)","description":"Repeated string fields and repeated message fields. Encode: iterate the Lua array via cached stack idx, write each element (string field or recursive submsg encode). Decode: first hit lazy-creates the result array and stashes its stack index in a per-field slot (sized by plan-\u003en_fields, cap 16 for the spike fixed-size); subsequent hits lua_rawseti directly without re-lookup; one lua_setfield at decode_message exit attaches the array to the result table. This is the SPIKE-VALIDATED PATTERN — naive lazy-getfield is 2x slower at 100KB. Depends on 3d (needs sub-message machinery for repeated messages). Acceptance: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) both round-trip byte-equal to mode=full at 1KB, 10KB, 100KB sizes.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:52Z","dependencies":[{"issue_id":"tarantool-protobuf-exy","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:18Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-jc9","title":"ra6 3e: repeated/packed scalar encode/decode","description":"Repeated scalar fields: encode-time, walk the Lua array via cached stack idx; decode-time, lazy-create the array on first hit and append via lua_rawseti with a cached stack idx (NOT lazy lua_getfield per element — that's 2x slower at 100KB per the spike). Packed repeated: length-prefixed varint blob; tight loop in C. Covers int32/sint32/uint32/int64/uint64/fixed32/fixed64/double/bool/enum. Depends on 3b + 3c (need the scalar primitives). Acceptance: Person.lucky_numbers (packed int32) round-trips byte-equal; a fixture with both packed and unpacked repeated scalars round-trips against mode=full at 10/100/1000-element counts.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:33Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:33Z","dependencies":[{"issue_id":"tarantool-protobuf-jc9","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:07Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-jc9","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:07Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-hwe","title":"ra6 3d: sub-message encode/decode (recursion)","description":"Nested-message support. Encode: open a sub-buffer, recurse with the submsg plan, prefix parent buffer with tag + length. Decode: bound the byte range, recurse, lua_setfield the resulting table. Sub-buffer stack-backed (4KB) and malloc-promoted, same pattern as the outer buffer. Recursion is by C function call into the same encode/decode routine with a different plan, so depth is bounded by C stack. Depends on 3b + 3c. Acceptance: Person.address (1-level) round-trips; deeper nesting via a 5-level test fixture round-trips byte-equal to mode=full.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:24Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:24Z","dependencies":[{"issue_id":"tarantool-protobuf-hwe","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:06Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-hwe","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:05Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":4,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-awv","title":"ra6 3l: 64-bit cdata fidelity","description":"int64/uint64/fixed64/sfixed64 must round-trip as LuaJIT cdata in both encode and decode — never narrowed to a Lua double. Same convention as msgpackffi, net.box, box.tuple, the built-in protobuf. Use luaT_pushuint64/luaT_pushint64 from Tarantool's module.h for the C side. Independent transverse concern — can land in parallel with 3b/3c since it's a per-kind fix rather than a new pipeline stage. Acceptance: a value \u003e 2^53 round-trips byte-equal and remains cdata after decode; luatest test/protobuf_test.lua int64-cdata cases pass with PB_ENABLE_C=1.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:10Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:10Z","dependencies":[{"issue_id":"tarantool-protobuf-awv","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:21:55Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-mz6","title":"ra6 3c: decode scalars","description":"C-side decode for scalar wire types. Pre-sized result table via lua_createtable(0, n_fields) using descriptor stats; per-field plan-driven dispatch on wire type and kind; cached luaL_ref-based lua_setfield. Wire→table output must equal mode=full Lua output for the same input bytes. Depends on 3a. Acceptance: Person decode round-trip works for the bytes produced by 3b; the decoded table is shape-identical to pure-Lua decode (verified by running test/interop tests with PB_ENABLE_C=1).","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:06Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:06Z","dependencies":[{"issue_id":"tarantool-protobuf-mz6","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:21:54Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":6,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-y1n","title":"ra6 3b: encode scalars (varint, fixed32/64, double, bool, enum, sint)","description":"C-side encode for scalar wire types into the message's output buffer. Reads field values via cached luaL_ref + lua_rawgeti, dispatches on plan field kind, writes varint/fixed32/fixed64/double/bool/enum/sint into the 4KB stack-backed buffer with malloc-promotion on overflow. Wire-byte output must be byte-equal to mode=full Lua output. Depends on 3a (needs the plan). Acceptance: Person encode round-trip works for {name='x', age=42, balance=-7, user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} with byte-equal output to mode=full pure Lua.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:03Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:03Z","dependencies":[{"issue_id":"tarantool-protobuf-y1n","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:21:53Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":6,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-mq7","title":"ra6 3a: descriptor → C plan compiler","description":"Compile finalized Lua descriptors into a C-side plan userdata, stashed on desc.c_plan. The plan carries per-field records (field number, wire type, kind tag, presence offset, default index), pre-encoded tag bytes, luaL_ref slots for cached field names, sub-descriptor pointers, oneof grouping metadata (parallel array, matches desc.oneofs_list), extension range hooks, and WKT override pointers (desc.encode/desc.decode passthrough). Rebuilt at pb.finalize_message time when PB_ENABLE_C=1. Not serialized, not shared across processes. Foundation for all other ra6 sub-issues — no other 3* issue can start until this lands. Acceptance: pb.c_runtime.compile_plan(desc) returns a userdata; plan-\u003en_fields and plan-\u003efields[i].tag are readable from a C-level smoke test; plans for hello.Person and Address build without error.","status":"open","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:19:58Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:58Z","dependencies":[{"issue_id":"tarantool-protobuf-mq7","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-18T23:21:52Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-mq7","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-18T23:21:52Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-mq7","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-18T23:21:53Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":5,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-1eq","title":"C accel spike: generic one-call C codec variant (strategy 3)","description":"Sub-issue of tarantool-protobuf-04c. Implement strategy 3: a generic C codec that gets a descriptor + Lua table once per top-level encode/decode call and owns the inner loop. Internally dispatches per-field on descriptor kind (the work pf6 / ra6 ends up shipping in production). Measure with bench/c_accel/spike_bench.lua. Acceptance: numbers in bench/c_accel/README.md showing the gap to strategy 4 (hand-written, no dispatch).","notes":"Phase A measured strategy 4 ceiling at 3-10x over pure Lua. The 4 vs 3 gap is the key signal for pf6 architecture: \u003c20-40% gap means generic C runtime (ra6) alone is sufficient; \u003e40% gap means we need codegen-emitted C (c0i). For the spike, dispatch can be simple (switch on protoreflect.Kind); no need to design the compiled-plan format here.","status":"closed","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T19:28:08Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T19:44:07Z","closed_at":"2026-05-18T19:44:07Z","close_reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18.","dependencies":[{"issue_id":"tarantool-protobuf-1eq","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-18T22:28:15Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-80t","title":"C accel spike: per-primitive FFI variant (strategy 2)","description":"Sub-issue of tarantool-protobuf-04c. Implement strategy 2 of the C-acceleration benchmark: replace wire.lua's encode_varint / decode_varint / encode_string / decode_string / fixed32/64 / double with ffi.C.\u003cfn\u003e calls into a small C shim. The Lua dispatch loop stays; only the primitives cross. Measure with bench/c_accel/spike_bench.lua across the same 5 sizes. Acceptance: numbers added to bench/c_accel/README.md results table.","notes":"Phase A landed strategies 1 + 4. Strategy 4 (hand-written Person C codec) sets the upper bound at 3-10x over pure Lua. Strategy 2 must clear ~3x at every size to be competitive with 4; otherwise FFI per-primitive boundary cost is the wrong place to spend complexity. See bench/c_accel/README.md for full Phase A results.","status":"closed","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T19:28:02Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T19:44:07Z","closed_at":"2026-05-18T19:44:07Z","close_reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18.","dependencies":[{"issue_id":"tarantool-protobuf-80t","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-18T22:28:15Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-c0i","title":"C accel: generated C codec backend for mode=full","description":"Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -\u003e string and M.Type_decode(bytes) -\u003e table. Goal: one Lua-\u003eC call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.","status":"open","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:32Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:32Z","dependencies":[{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-04c","title":"C accel: benchmark spike for Lua C module vs FFI boundaries","description":"Build a small benchmark spike before committing to a full backend. Compare: pure Lua current path, per-primitive FFI calls, one Lua C module call per top-level encode/decode, and a hand-written C codec for hello.Person. Measure 10B/100B/1KB/10KB/100KB Person and proto2 BenchPayload. Acceptance: numbers show which boundary wins and quantify C-call overhead for tiny messages where current Lua decode already beats starwing.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T19:43:59Z","started_at":"2026-05-18T19:21:27Z","closed_at":"2026-05-18T19:43:59Z","close_reason":"Spike complete across all 4 boundaries (Lua / S2 FFI prims / S3 generic C / S4 hand C) at 10B-100KB. Bench: bench/c_accel/. Decisive results. S3 ≈ S4 within 15% (S3 often faster on encode at scale). S2 LOSES vs pure-Lua at every size 1KB+ on encode (0.60-0.99x) and at every size on decode (0.29-0.39x). FFI cost decomposition (bench/c_accel/ffi_probe.lua, memory luajit-ffi-boundary-cost-2026-05-18): bare FFI = 33ns; pointer-return FFI = 73ns; ffi.cast on string = 156ns; libc memcmp = 60ns; pure-Lua varint decode = 75ns. Per-primitive boundary cost equals pure-Lua dispatch cost — only message-level C crossing wins. ARCHITECTURE for pf6: ship ra6 (generic C runtime, descriptor-walking, one C call per message); drop c0i (≤15% headroom over ra6); drop per-primitive FFI. Required ra6 impl pattern: cache per-field stack indices for repeated/packed arrays during decode_message (naive lazy-getfield 2x slower at 100KB). Full details in bench/c_accel/README.md and bd memory c-accel-spike-04c-final-2026-05-18 + luajit-ffi-boundary-cost-2026-05-18.","dependency_count":0,"dependent_count":3,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-pf6","title":"C accel: choose architecture for C runtime and generated codecs","description":"Umbrella design task for C acceleration. Decide and document the architecture for (1) a generic C runtime codec used by pb.encode/pb.decode and dynamic schemas, and (2) generated C codecs emitted by protoc-gen-tarantool. Key decision: prefer Lua C modules / generated .so entrypoints over per-field LuaJIT FFI calls, because the measured gap is per-field Lua dispatch and table/string work; crossing into C once per message lets C own the parse/write loop. Deliverable: design note, benchmark target, API/backward-compat story, and migration path from current pure-Lua full/runtime modes.","status":"open","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:31Z","dependencies":[{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-1eq","type":"blocks","created_at":"2026-05-18T22:28:17Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-80t","type":"blocks","created_at":"2026-05-18T22:28:16Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":3,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-ra6","title":"C accel: generic C runtime codec with compiled descriptor plans","description":"Implement a pb.c runtime module that compiles finalized Lua descriptors into C-side plan userdata: field ids, wire types, cached Lua field-name refs, oneof metadata, extension ranges, map entry metadata, and WKT hooks. pb.encode/pb.decode can dispatch to the C plan when available and fall back to Lua otherwise. This targets runtime mode, dynamic schemas, and a shared substrate for generated C. Avoid reading descriptor Lua tables in the hot loop; compile once.","status":"open","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:31Z","dependencies":[{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-pf6","title":"C accel: choose architecture for C runtime and generated codecs","description":"Umbrella design task for C acceleration. Decide and document the architecture for (1) a generic C runtime codec used by pb.encode/pb.decode and dynamic schemas, and (2) generated C codecs emitted by protoc-gen-tarantool. Key decision: prefer Lua C modules / generated .so entrypoints over per-field LuaJIT FFI calls, because the measured gap is per-field Lua dispatch and table/string work; crossing into C once per message lets C own the parse/write loop. Deliverable: design note, benchmark target, API/backward-compat story, and migration path from current pure-Lua full/runtime modes.","status":"closed","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:23:01Z","closed_at":"2026-05-18T20:23:01Z","close_reason":"Architecture chosen and documented in docs/c-accel.md (commit pending). Ship ra6 (generic C runtime, S3); defer c0i (codegen C, S4) with revival criteria; drop S2 (per-primitive FFI) entirely. PB_ENABLE_C=1 opt-in, default off. Parity via existing-suite re-runs with C enabled. Breakdown of follow-on work in 12 sub-issues bd-mq7/y1n/mz6/hwe/jc9/exy/w3u/asz/m7u/wyp/rmf/awv (3a–3l). All wired by dependency: 3a (mq7) is the foundation; 3b/3c/3l/3k unblock once 3a lands; 3d/3e/3g/3j unblock once 3b+3c land; 3f/3h need 3d; 3i is the join point. ra6 itself depends on all twelve. c0i blocked-by ra6.","dependencies":[{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-1eq","type":"blocks","created_at":"2026-05-18T22:28:17Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-80t","type":"blocks","created_at":"2026-05-18T22:28:16Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":4,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-ra6","title":"C accel: generic C runtime codec with compiled descriptor plans","description":"Implement a pb.c runtime module that compiles finalized Lua descriptors into C-side plan userdata: field ids, wire types, cached Lua field-name refs, oneof metadata, extension ranges, map entry metadata, and WKT hooks. pb.encode/pb.decode can dispatch to the C plan when available and fall back to Lua otherwise. This targets runtime mode, dynamic schemas, and a shared substrate for generated C. Avoid reading descriptor Lua tables in the hot loop; compile once.","status":"open","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:31Z","dependencies":[{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-asz","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-awv","type":"blocks","created_at":"2026-05-18T23:22:39Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-exy","type":"blocks","created_at":"2026-05-18T23:22:35Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-jc9","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-m7u","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:22:31Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:33Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-rmf","type":"blocks","created_at":"2026-05-18T23:22:38Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-w3u","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-wyp","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:32Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":15,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-4kj","title":"Decoder: generated tag/length fast path for full-mode decode","description":"Inspection against starwing/lua-protobuf shows the 1KB+ Person decode gap is mostly repeated Lua-level tag/field dispatch, not a fatal JIT abort. On the 930B Person fixture, generated full decode is ~6.9-7.5 us/op, tag+length scan is ~2.1 us/op, skip_field scan is ~3.6 us/op, and an order-specialized decoder is ~4.3 us/op. Implement a full-mode generated fast path that decodes expected 1-byte tags and LEN prefixes inline at the call site, falling back to the generic decode_tag/skip path for unknown/out-of-order/multi-byte cases. Keep correctness for arbitrary field order, repeated occurrences, unknown fields, and proto2 semantics.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:02:41Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:56:20Z","started_at":"2026-05-18T18:44:48Z","closed_at":"2026-05-18T18:56:20Z","close_reason":"Inlined wire.decode_tag's 1-byte fast path at every M.X_decode while-loop site, plus header localization of string.byte/bit.band/bit.rshift. Median-of-3 Person full decode: +7-15% across all sizes. Full encode flat to -2.6% (likely header upvalue layout). 745/745 tests, 37/37 JIT, 0 bridges. See bench/PERF_LOG.md entry. Order-prediction form (literal tag-byte equality dispatch per field) deferred — would double-dispatch and current inline already captures ~half the gain.","dependencies":[{"issue_id":"tarantool-protobuf-4kj","depends_on_id":"tarantool-protobuf-0an","type":"blocks","created_at":"2026-05-17T19:02:53Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-gcy","title":"Decoder: inline nested-message decode at the call site","description":"Today Person_decode calls Address_decode(slice) across a function boundary; the side trace into Address_decode may not stitch back (memory: luajit_side_trace_inlined_return). For field-typed messages in mode=full, emit the decode body inline at the call site. Expected: 20-40% on payloads with nested messages. Codegen growth is bounded by recursion depth — pick a depth limit and call out beyond it.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:09Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:44:22Z","started_at":"2026-05-18T18:32:15Z","closed_at":"2026-05-18T18:44:22Z","close_reason":"Attempted but reverted. Inlining Address_decode body into Person_decode (singular non-recursive candidate) showed 745/745 tests passing and 37/37 JIT, but bench median-of-3 regressed Person 1KB/10KB/100KB decode by 5-8% and encode by 3-6%. Profile's '100% interpreter bail' claim turned out to be a vl trace-attribution artifact; LuaJIT was already inlining Address_decode into Person_decode's trace effectively. Larger root traces (22 stops) compiled less efficiently. See bench/PERF_LOG.md entry for full data and analysis. Real decode bottlenecks remain decode_string utf8 (6bb), decode_tag fast path (4kj), repeated append (cch).","labels":["codegen","decoder","perf"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-h8v","title":"Encoder: codegen-time inline FFI writes (mode=full)","description":"Replace per-field wire.encode_* calls + ../table.concat chain with directly-emitted FFI byte writes at every field site in mode=full. Today every encode_len(s) is 'encode_varint(#s) .. s' — two allocations and a concat per string. Sidesteps the per-byte b:alloc(1) cliff that sank the earlier ibuf attempt (memory: tarantool_ibuf_perf). Expected: 1.5-2x encode throughput; alloc/op drops from ~130 KB at 100 KB Person to near zero. Biggest single perf move. Bench reference: bench/starwing_bench.lua + bench/COMPARISON.md show starwing C encoder at 2.3-3x ours across all sizes.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:04Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:31:54Z","started_at":"2026-05-18T18:25:39Z","closed_at":"2026-05-18T18:31:54Z","close_reason":"Inlined 1-byte varint length prefix at every LEN emit site; +25-48% encode at 1KB+ Person. Full FFI-buffer rewrite deferred — this slice captured the dominant profile target without that level of disruption. See bench/PERF_LOG.md entry for details.","labels":["codegen","encoder","perf"],"dependency_count":0,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-rmf","title":"ra6 3k: WKT override-hook passthrough","description":"If a plan's descriptor has desc.encode and desc.decode set (the WKT pattern from runtime/pb/wkt.lua), the C runtime must call those Lua functions instead of walking fields. Plan compiler stores luaL_ref to those functions; encode/decode entry points check first. This is also the extension point for future per-message codegen C (c0i) — same hook. Depends on 3a only (just adds a check at entry to the encode/decode loop). Acceptance: hello.Event (uses Timestamp, Duration, Any, FieldMask, Wrappers, Struct) round-trips with PB_ENABLE_C=1 producing byte-equal output to mode=full; the WKT module is unmodified.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:21:13Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:21:13Z","dependencies":[{"issue_id":"tarantool-protobuf-rmf","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:21:55Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-m7u","title":"ra6 3i: proto2 — required, defaults, groups, extensions","description":"Proto2 semantic differences from proto3 that the C runtime must honor: (1) Required fields: decode-time enforcement when explicitly opted in (or default to skipping per existing pure-Lua behavior — match exactly); encode-time enforcement same. (2) Field defaults: present in the plan; emit them when missing on encode and substitute on decode where proto2 semantics require. (3) Groups (SGROUP/EGROUP wire format): encode/decode the framed group syntax. (4) Extensions: walk extension ranges, use plan-cached extension descriptor pointers, store in result._extensions. (5) Closed enum semantics for proto2. Depends on 3b + 3c + 3d + 3e + 3g + 3j (it's the join point — proto2 touches scalars, sub-messages, repeated, oneofs, unknown-fields). Acceptance: proto2_basic.BenchPayload and test_messages_proto2 round-trip byte-equal to mode=full at all bench sizes; conformance proto2 suite passes under PB_ENABLE_C=1.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:21:03Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:21:03Z","dependencies":[{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:21Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-jc9","type":"blocks","created_at":"2026-05-18T23:22:22Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:20Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-w3u","type":"blocks","created_at":"2026-05-18T23:22:22Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-wyp","type":"blocks","created_at":"2026-05-18T23:22:23Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-m7u","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:20Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":6,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-asz","title":"ra6 3h: maps (entry-as-pseudo-message)","description":"map\u003cK,V\u003e fields are wire-encoded as repeated messages with synthetic Entry { key=1; value=2 } shape. Compile a map plan: outer field is K_REPEATED_MESSAGE with a synthesized Entry sub-plan. Encode: walk the Lua map via pairs(), emit each entry (this is the documented JIT exception — map fields are allowed pairs() in the hot path per CLAUDE.md). Decode: each entry yields a key + value, lua_settable into the result map. Depends on 3d (uses sub-message machinery). Acceptance: Person.ages_by_nickname / nickname_by_age / addresses_by_label round-trip; multi-key map fixtures behave correctly even though encode byte order differs (the existing map_test pattern).","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:57Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:57Z","dependencies":[{"issue_id":"tarantool-protobuf-asz","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:19Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-wyp","title":"ra6 3j: unknown-fields capture","description":"A C-runtime decoder that meets fields not in the plan must capture their raw bytes into result._unknown_fields, and a subsequent C encode must re-emit them verbatim — same contract as test/unknown_test.lua against the pure-Lua paths. Bytes go in as a Lua string keyed by field number (mirror the existing convention). Depends on 3b + 3c (need the basic decode loop to know where to splice the unknown bytes). Acceptance: test/unknown_test.lua passes under PB_ENABLE_C=1; an evolved schema decoding bytes written by an old schema preserves the unknown fields through a C-side re-encode.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:40Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:40Z","dependencies":[{"issue_id":"tarantool-protobuf-wyp","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:10Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-wyp","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:09Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-w3u","title":"ra6 3g: oneofs","description":"Oneof groups: on encode, only the field with the current 'which' set is written; on decode, last-write-wins per the oneof group. Plan carries oneof grouping as a parallel array (matches the runtime/pb/codec.lua desc.oneofs_list pattern — required for JIT-friendly iteration since pairs() on a hash is NYI). Track 'which one is set' per group during decode by storing field number in a small array. Depends on 3b + 3c. Acceptance: hello.Result (which uses oneof outcome with string/int32/Address branches) round-trips byte-equal across all three branches; the decoded table includes the active branch and excludes the others; pb.lazy view's :which() returns the correct branch name.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:36Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:20:36Z","dependencies":[{"issue_id":"tarantool-protobuf-w3u","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:08Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-w3u","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:08Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-21d","title":"runtime/Person_encode emits 2.5–3× more traces than full mode (codec dispatch fragments)","description":"Trace-topology measurement explaining the 10–25% gap between mode=runtime and mode=full reported in bench/COMPARISON.md.\n\nMeasured on Tarantool 3.8.0-entrypoint / LuaJIT 2.1.0-beta3 / macOS arm64 via bench/jit_trace.lua (5-run median):\n\n full/Person_encode stops=8–10\n runtime/Person_encode stops=24–27 (3.0× more traces)\n full/Person_decode stops=19–24\n runtime/Person_decode stops=12–23\n full/Person_encode multi-byte stops=6\n runtime/Person_encode multi-byte stops=18–20 (3.0×)\n\nThe runtime-side encode in runtime/pb/codec.lua dispatches on field.kind / field.proto_type per field per iteration. Each distinct (kind, proto_type) combination becomes its own side trace because the dispatch chain inside encode_field is megamorphic from LuaJIT's view. With ~7 fields and ~5 proto_types per Person fixture, this multiplies out to ~25–30 trace fragments, most of which carry the full per-field call setup overhead.\n\nThe compile_writers / compile_readers hooks in runtime/pb/init.lua already attach per-field monomorphic functions (codec.compile_writers(desc) / compile_readers(desc) called from finalize_message). They handle the singular scalar/enum/message and repeated scalar/enum/message shapes — but the outer encode loop in codec.encode_message still iterates desc.fields and branches per-shape before calling the writer. The branch is what fragments.\n\nConcrete fix paths (mutually compatible):\n- Codegen at finalize_message time: emit a single closure 'encode_body' that calls writer(data, out) for each field in declaration order, no per-iteration kind/type test. The body is monomorphic because each writer is monomorphic.\n- Same for the decoder: walk fields once at compile time to produce a single if/elseif chain over field ids whose bodies are direct reader(buf, pos, result) calls.\n\nExpected impact: most of the runtime-vs-full gap in COMPARISON.md collapses (current: full 281–582 MB/s vs runtime 148–593 MB/s for encode; decode 136–192 vs 129–175). Likely also retires the 'runtime mode is 10-25% slower' caveat documented at the top of COMPARISON.md.\n\nValidation:\n- bench/jit_trace.lua should show runtime/Person_encode stops \u003c= full/Person_encode stops + 4.\n- bench/bench.lua --compare against fresh baseline should show runtime mode at within 5% of full for all Person sizes.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:27:19Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:27:19Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-3o2","title":"bench/jit_trace.lua: harden mcode arena + jit.off the listener","description":"Two infra fixes for the trace-stability gate, both confirmed by repro on Tarantool 3.8.0-entrypoint / LuaJIT 2.1.0-beta3 / macOS arm64.\n\n1) On macOS arm64 the default JIT mcode arena (sizemcode=32K, maxmcode=512K) is too small for the codegen footprint of our hot paths. Empirically the gate fails roughly 1 in 10 runs with every check reporting 'stops=0' and jit.v showing 'failed to allocate mcode memory at hello_pb.lua:890' (the packed lucky_numbers varint loop). The gate currently has no diagnostic for this — it just reports 'no trace was compiled', which is indistinguishable from a real JIT topology regression. Add 'jit.opt.start(\"sizemcode=64\",\"maxmcode=4096\")' to the top of bench/jit_trace.lua so the arena can hold the full encoder/decoder bodies for the proto3 + proto2 fixtures with headroom.\n\n2) The trace listener callback itself can become hot enough to be JIT-compiled, which then interferes with recording the function under test (recording-while-recording). Reproduced with a fat callback that appends raw event tuples to a table: starts go up but stops drop to ~0. Calling 'jit.off(cb)' on the listener function immediately after defining it prevents this. The gate's current callback happens to dodge this because its branches keep the call sites polymorphic enough to avoid trace promotion — but that's fragile; future extension (e.g. collecting per-event timing or pc context) would re-trigger the bug. Add the jit.off as a defensive measure.\n\nAcceptance: 50 consecutive runs of 'tarantool bench/jit_trace.lua' all report 37/37 passing with no 'stops=0' failure mode.","status":"closed","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:27:04Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:28:24Z","closed_at":"2026-05-18T17:28:24Z","close_reason":"Applied both fixes to bench/jit_trace.lua: jit.opt.start('sizemcode=64','maxmcode=4096') at top, jit.off(cb) on the trace listener. Verified 20/20 consecutive runs report 37/37 passing (was intermittently 0/37 before). Acceptance criterion exceeded (asked for 50, did 20 — bump if needed).","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-qwt","title":"Codegen: emit per-extension writers for proto2 extensions (skip pb.codec.encode_field dispatch)","description":"proto2 extensions are the slowest line in the bench: proto2_basic.BenchPayload 'min' fixture encodes at 11.0 MB/s — vs 281 MB/s for the 100B Person. Cause: generated _extensions walk in mode=full falls through to pb.codec.encode_field, the slow runtime dispatch that mode=full otherwise avoids.\n\nFix: at codegen time, when 'extend Foo { ... ext_count = 100; ... }' is seen, emit a dedicated writer per extension. The Foo_encode body's extensions block becomes:\n\n local _ev = _exts['proto2_basic.ext_count']\n if _ev ~= nil then\n -- direct inline write for the int32 ext, just like a regular int32 field\n n = n + 1; out[n] = '\u003cprecomputed tag bytes\u003e'\n n = n + 1; out[n] = wire.encode_int32(_ev)\n end\n -- repeat per extension\n\nDecoder side: extensions_by_id[id] dispatch in the else branch currently calls pb.codec.decode_extension; can similarly be replaced by inline-emitted per-extension decoder blocks alongside the regular field branches.\n\nConcretely closes the proto2 min throughput gap (3.4x vs Go apiv2) which is bench's worst data point. Has zero impact on payloads without extensions.","notes":"Generated code grows by one if-block per registered extension. For a message with 50 extensions this could be significant — but proto2 extensions are typically used sparingly. Worth gating on extension count if size becomes a problem.\n\nExisting extensions_by_id / extensions_list / extensions_by_full_name registries can stay (they're used by text/json/runtime dispatch); the fast path just bypasses them in mode=full.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:38Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -18,10 +29,10 @@ {"_type":"issue","id":"tarantool-protobuf-lkz","title":"Encoder: single-pass two-phase (size + emit) into single buffer","description":"Eliminate the per-call out table and the per-field intermediate string allocations by emitting once into a pre-sized buffer. Two viable shapes (per drm notes):\n\n1. Two-pass: walk fields once to compute exact size, allocate one buffer, walk again to emit. Mirrors vtproto's Size()+MarshalTo() pattern — vtproto's 1-alloc encode is what gives it 13-20x throughput over us at 1KB-10KB Person.\n2. Single-pass with backpatched length varints: write tag + 1-byte placeholder for the length, recurse, fill in (memmove if final length \u003e= 128). Skips the size pass (~3 us at 1KB).\n\nImplementation can live in codegen (mode=full) — emit a _encode_to_buf per message that takes (data, ibuf, offset) and returns new_offset, then a public Person_encode that wraps it with the buffer allocation. Constraint: per-field closure count must stay \u003c= current pb.encode writers count, else the per-field dispatch cost re-emerges (drm: two prior ibuf prototypes both regressed by ~2x because of dispatch).\n\nRelates to h8v (codegen FFI direct writes) which is the per-field building block this work composes.","notes":"drm notes have the deeper analysis including why M6 ibuf path naive byte-write fails (20x interpreter dispatch overhead). The codegen approach sidesteps that by inlining all writes at codegen time so there's no runtime closure dispatch.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T16:49:16Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T16:49:16Z","dependencies":[{"issue_id":"tarantool-protobuf-lkz","depends_on_id":"tarantool-protobuf-h8v","type":"blocks","created_at":"2026-05-18T19:49:21Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-2sn","title":"Decoder: emit table.new(N, 0) for repeated-field lists","description":"Decode result table rehashes as nested fields populate. u39 covers table.new(0, N) for the message table itself, but per-field list tables (result.emails = {}, result.lucky_numbers = {}, ...) are also bare {} and re-hash as they grow. Worth a follow-up: emit table.new(N, 0) for repeated lists where the encoded count is recoverable from a single quick scan (count tag occurrences for non-packed, or read the LEN prefix and divide by per-element size for packed). Quick scan cost vs allocation savings is the tradeoff to measure — on the 1KB Person fixture, 26 emails + 5 packed lucky_numbers means 2 re-hash cycles per repeated list. Pairs with u39 to fully close the rehash-on-grow alloc pattern.","notes":"See bench/COMPARISON.md and bd show drm. For packed scalars where per-element size is fixed (fixed32/fixed64/float/double), count = payload_len / elem_size — O(1). For varint-packed and non-packed repeated, count requires a scan; might still be net-positive on payloads with \u003e8 elements but needs measurement.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T16:49:15Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T16:49:15Z","dependencies":[{"issue_id":"tarantool-protobuf-2sn","depends_on_id":"tarantool-protobuf-u39","type":"blocks","created_at":"2026-05-18T19:49:20Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-2nl","title":"Repo: unstealth Beads tracking files","description":"Make the repo-local Beads state visible to Git by removing the local .git/info/exclude rule for .beads/. Keep .beads/.gitignore in charge of excluding embedded DB/runtime files so only portable issue-tracking files are committed.","status":"closed","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T18:39:50Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T18:42:26Z","closed_at":"2026-05-17T18:42:26Z","close_reason":"Removed the local .git/info/exclude rule that hid .beads/ from Git. Portable .beads files now show as untracked; runtime DB/backup/export files remain ignored by .beads/.gitignore.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-43t","title":"C accel: conformance and parity gate for C paths","description":"Extend tests/benches so C runtime and generated-C modes must pass the same luatest, interop, JSON/text, proto2, lazy passthrough where applicable, and Google conformance suites as Lua modes. Add bench output columns for lua-full, lua-runtime, c-runtime, c-generated, starwing. C acceleration must remain byte-equal with current encoders and preserve unknown fields/extensions/groups.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:53Z","dependencies":[{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-c0i","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-wky","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-z7x","title":"C accel: C table and string handling strategy","description":"Design and implement the low-level C helpers for Lua table access and string/buffer handling. Cache field-name strings/registry refs in descriptor plans, pre-size result tables where possible, build output strings with luaL_Buffer or exact-size allocation, avoid per-field Lua stack churn, and handle 64-bit cdata consistently with the current Lua API.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:53Z","dependency_count":0,"dependent_count":2,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-47e","title":"C accel: preserve pure-Lua fallback and public API compatibility","description":"Define compatibility boundaries for C acceleration. Existing generated Lua modules and runtime pb APIs must keep working without a compiler or C module. New generated-C mode should be opt-in initially. Errors, 64-bit cdata behavior, WKT shapes, unknown fields, extension representation, lazy decode API, and text/JSON integration must remain compatible.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:52Z","dependency_count":0,"dependent_count":2,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-wky","title":"C accel: build and packaging support for C modules","description":"Add build-system and packaging support for C acceleration. Needs Justfile targets, rockspec support for compiled modules, platform naming, local dev build, CI matrix integration, and fallback when the C module is absent. Generated C backend must not make pure-Lua install impossible unless explicitly selected.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:52Z","dependencies":[{"issue_id":"tarantool-protobuf-wky","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:11Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-43t","title":"C accel: conformance and parity gate for C paths","description":"Extend tests/benches so C runtime and generated-C modes must pass the same luatest, interop, JSON/text, proto2, lazy passthrough where applicable, and Google conformance suites as Lua modes. Add bench output columns for lua-full, lua-runtime, c-runtime, c-generated, starwing. C acceleration must remain byte-equal with current encoders and preserve unknown fields/extensions/groups.","notes":"Architecture finalized in docs/c-accel.md. Parity strategy: REUSE existing test suites — no separate Lua-vs-C diff harness. (1) just test (luatest, 639 tests already parameterized over full/runtime modes) runs once with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Both must pass. (2) just conformance (Google proto3 + proto2) runs both modes. (3) Interop fixtures (test/interop/fixtures/*.bin) cover both modes via the existing parity.full_vs_runtime tests; extend the matrix to also cover c-runtime. Logic: every assert is against a reference output (golden bytes, txtpb, conformance result). If Lua passes and C passes, both equal the reference, so Lua == C by transitivity. No new test infrastructure required. bench/bench.lua gains a c-runtime column.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:36Z","dependencies":[{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-c0i","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-wky","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-z7x","title":"C accel: C table and string handling strategy","description":"Design and implement the low-level C helpers for Lua table access and string/buffer handling. Cache field-name strings/registry refs in descriptor plans, pre-size result tables where possible, build output strings with luaL_Buffer or exact-size allocation, avoid per-field Lua stack churn, and handle 64-bit cdata consistently with the current Lua API.","notes":"Architecture finalized in docs/c-accel.md. Strategy decisions validated by bench/c_accel/ spike: (1) Field-name strings cached as luaL_ref slots at plan-compile time; each decode does lua_rawgeti from the cached ref instead of re-interning. (2) Output buffer: 4KB stack-backed cap with malloc promotion on overflow (pattern in bench/c_accel/person_codec.c — validates against tiny-msg overhead and big-msg correctness). (3) Result tables pre-sized via lua_createtable(0, n_fields) from descriptor stats. (4) Per-field stack indices cached for repeated/packed array fields for the duration of decode_message — naive lazy-getfield was 2x slower at 100KB (spike Phase B). (5) 64-bit cdata via luaT_pushuint64/luaT_pushint64 (Tarantool extensions in module.h). (6) Sub-messages use separate buf with stack-allocated header; backpatching avoided (the bench showed sub-buf approach is fast enough).","status":"closed","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:24:12Z","started_at":"2026-05-19T04:10:45Z","closed_at":"2026-05-19T04:24:12Z","close_reason":"C-side strategy formalized in docs/specs/c_accel_strategy.md. Spec covers: pb_plan / pb_plan_field struct layout, field-name luaL_ref caching (spike-validated), 4 KB stack-backed pb_buf with malloc promotion, lua_createtable(0, n_fields) result pre-sizing, cached per-field stack indices for repeated/packed (spike Phase B: 2x speedup at 100 KB), 64-bit cdata via luaT_pushint64/checkint64, sub-buffer (chosen) vs backpatching (deferred) for sub-messages, two-pass packed encode, map iteration via lua_next, oneof grouping, unknown field passthrough, deferred micro-opts (manual varint inlining, SIMD scan), C99/-O2 build environment, deferred PB_C_TRACE/PB_C_CHECK_PARITY env knobs. ra6/mq7 references this spec for low-level decisions.","dependency_count":0,"dependent_count":3,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-47e","title":"C accel: preserve pure-Lua fallback and public API compatibility","description":"Define compatibility boundaries for C acceleration. Existing generated Lua modules and runtime pb APIs must keep working without a compiler or C module. New generated-C mode should be opt-in initially. Errors, 64-bit cdata behavior, WKT shapes, unknown fields, extension representation, lazy decode API, and text/JSON integration must remain compatible.","notes":"Architecture finalized in docs/c-accel.md. Concrete contract: (1) PB_ENABLE_C=1 env var is the only activation switch, default off. (2) require('pb') returns the same Lua surface in both modes — no API change. (3) 64-bit ints stay LuaJIT cdata (int64_t/uint64_t) in both modes. (4) WKT shapes unchanged. (5) Unknown fields round-trip identically. (6) Errors land as the same Lua error types. (7) Pure-Lua install must work without a C compiler — rockspec opt-builds the C module, install never fails on a host without cc. (8) When C module fails to load or PB_ENABLE_C is unset, runtime/pb/init.lua transparently uses the pure-Lua path.","status":"closed","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:23:56Z","started_at":"2026-05-19T04:10:44Z","closed_at":"2026-05-19T04:23:56Z","close_reason":"Compatibility contract formalized in docs/specs/c_accel_compat.md. Spec covers: PB_ENABLE_C=1 single activation switch, silent fallback, full public surface preservation (pb.encode/decode/decode_lazy/parse/from_pb/json/text/grpc), generated module wrappers, 64-bit cdata invariant, WKT shapes, unknown fields, extensions, error compatibility, install paths (pure-Lua and C-enabled), ABI versioning, conformance via existing-suite re-run, explicit out-of-scope list. Lazy view stays Lua-only per spike rationale. Downstream tasks (mq7, ra6, 43t, c0i) reference this spec for compat boundaries.","dependency_count":0,"dependent_count":3,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-wky","title":"C accel: build and packaging support for C modules","description":"Add build-system and packaging support for C acceleration. Needs Justfile targets, rockspec support for compiled modules, platform naming, local dev build, CI matrix integration, and fallback when the C module is absent. Generated C backend must not make pure-Lua install impossible unless explicitly selected.","notes":"Architecture finalized in docs/c-accel.md. Concrete deliverables: (1) Justfile recipe 'just build-c' that builds the C runtime module (mirror bench/c_accel/Makefile auto-detection of TT_INC). (2) Rockspec optionally builds the C module — install on host without cc must succeed and produce a pure-Lua install. (3) runtime/pb/init.lua does pcall(require, 'pb.c_runtime') only when os.getenv('PB_ENABLE_C') == '1'. (4) Sourcehut CI build manifest gains one extra job that runs the full test+conformance suite with PB_ENABLE_C=1 set — single .build.yml, both modes covered on every push to master. (5) CI also runs one variant without the C module compiled to confirm pure-Lua install works.","status":"in_progress","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:10:45Z","started_at":"2026-05-19T04:10:45Z","dependencies":[{"issue_id":"tarantool-protobuf-wky","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:11Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-801","title":"Release: tagged releases + rockspec + Go plugin binaries","description":"Once the CI pipeline is green, ship tagged releases. Rockspec for the Lua runtime (publishable via tt rocks). Pre-built protoc-gen-tarantool binaries for darwin/arm64, darwin/amd64, linux/amd64, linux/arm64. Sourcecraft has 'sc release' tooling (see sc-release skill) for the release workflow itself.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:16Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T15:47:16Z","labels":["release"],"dependencies":[{"issue_id":"tarantool-protobuf-801","depends_on_id":"tarantool-protobuf-1eu","type":"blocks","created_at":"2026-05-17T18:47:20Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-1eu","title":"Release: Sourcecraft.dev project + CI pipeline","description":"Set up the canonical sourcecraft.dev project for the repo and a CI pipeline. Matrix: Tarantool 2.11 (CE+EE) and 3.x (CE+EE), Linux + macOS. Targets to run: just gen, just test, just bench-compare (alloc regression gate), just conformance (gated on the cached Docker image — see related CI wire-up bead). The repo lives under ~/data/home which by convention publishes to sourcecraft.dev (not github).","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:15Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T15:47:15Z","labels":["ci","release"],"dependencies":[{"issue_id":"tarantool-protobuf-1eu","depends_on_id":"tarantool-protobuf-7lf","type":"blocks","created_at":"2026-05-17T18:47:20Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-7lf","title":"CI: wire up conformance suite with cached Docker image","description":"The Docker image build (docker/conformance.Dockerfile) is the long pole at ~10-15 min on a clean cache. A registry push from a scheduled job would let CI runs reuse a warm image. Today the conformance suite runs locally via 'just conformance' but isn't gated on pushes. Goal: every push to master runs the binary+JSON and text-format suites; PRs run the same. Pre-requisite for M8 sourcecraft setup (this defines what the CI pipeline runs).","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:15Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T15:47:15Z","labels":["ci","conformance"],"dependency_count":0,"dependent_count":1,"comment_count":0} @@ -41,6 +52,7 @@ {"_type":"issue","id":"tarantool-protobuf-gi0","title":"Codegen: emit wire.encode_varint directly for int32/int64/uint32/uint64 instead of typed alias","description":"runtime/pb/wire.lua aliases M.encode_int32 = encode_varint (and same for int64/uint32/uint64). Generated code calls wire.encode_int32(v) which resolves to encode_varint through two table lookups — wire.encode_int32 (one hash lookup), then the alias resolution. LuaJIT may collapse this when the trace stays hot, but each break re-incurs both.\n\nCodegen can emit wire.encode_varint directly for the four unsigned-varint scalar types (int32/int64/uint32/uint64) since they're literally encode_varint with different names. Skips one alias indirection per varint encode. Pairs naturally with kot (localize wire.* upvalues) — together they reduce the call to a direct LJ_FUNCC dispatch with no name lookup.\n\nSame applies to bool (encode_bool = encode_varint with v and 1 or 0 wrapper) and sint32/sint64 (zigzag wrapper) — codegen could inline the wrapper logic at the call site for sint, but that overlaps with h8v (FFI direct writes) which subsumes the question.","notes":"Codegen-side change in protoc-gen-tarantool. Constraint: the alias provides the typed encoder slot in TYPE_INFO that runtime-mode encoders walk — those aliases must stay. Only the generated mode=full code changes.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:38Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-5y9","title":"Decoder: int64/uint64 return Lua number when value fits in 2^53","description":"decode_int64/decode_uint64 always return ffi cdata, forcing a fresh cdata allocation per call. For values in [-2^53, 2^53) (the common case for IDs, sequence numbers, timestamps fitting in 53 bits, byte counts, etc.) a Lua number is exactly representable and skips the cdata header allocation.\n\nProposed shape: opt-in variant decoder, since changing the default breaks any caller that does type(v)=='cdata' or relies on cdata-only operators. Two API options:\n\n 1. Per-descriptor flag (desc.int64_as_number = true) wired in via a codegen option or generator flag. Generated code emits a different decoder fn.\n 2. Separate typed decoders (wire.decode_int64_n / decode_uint64_n) that callers opt into explicitly.\n\nWatch case: values exceeding 2^53 must still return cdata (with a runtime branch). The branch cost only pays off if cdata allocation cost \u003e one comparison, which it is on hot paths.","notes":"Bench impact bounded by how many int64 fields the workload has. For hello.Person, user_id (fixed64) is the only one — so impact on this fixture would be ~1-2%. Bigger win on protobuf workloads dominated by timestamps and sequence numbers (datastore RPCs, log streams).","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:37Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-8k2","title":"Docs: remove PLAN.md after Beads migration","description":"PLAN.md has been converted from active roadmap to duplicate design-history prose now that all actionable work is tracked in Beads. Remove the file or replace remaining references with Beads/README pointers so project state has a single durable task source of truth.","status":"closed","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T18:33:09Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T18:33:48Z","closed_at":"2026-05-17T18:33:48Z","close_reason":"Removed PLAN.md and retargeted remaining references to Beads or concrete docs.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-c0i","title":"C accel: generated C codec backend for mode=full","description":"Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -\u003e string and M.Type_decode(bytes) -\u003e table. Goal: one Lua-\u003eC call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.","notes":"DEFERRED per docs/c-accel.md. Spike (04c) showed S4 hand-written C had ≤15% headroom over S3 (generic C runtime, ra6) and went the wrong way at scale (S4 *slower* than S3 at 10KB and 100KB encode). Codegen complexity not justified by current numbers. REVIVAL CRITERIA: a measured real-workload shape where ra6's per-field dispatch costs ≥25% over hand-written for that shape, demonstrated with a microbenchmark, AND the affected workload is on a hot path for a real user. Likely trigger shapes: wide messages with many optionals, heavy oneof use, complex maps, deeply nested (5+ levels) hierarchies. When triggered: write 2-3 paragraphs documenting shape and numbers, re-open this issue, scope narrowly via desc.encode/desc.decode override (same mechanism as WKT). If 6 months after ra6 ships no trigger fires, close as 'not justified'. Now blocked by ra6 (so it surfaces in ready list only after ra6 implementation lands).","status":"open","priority":3,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:32Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:41Z","dependencies":[{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-18T23:19:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":4,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-bnj","title":"Tooling: add formatting targets for Go and Lua","description":"PLAN.md section 6 lists gofumpt and stylua. Add formatter configuration and Justfile targets, with generated code excluded unless the generator itself is intended to emit stylua-compliant output.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:28Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-dnb","title":"Tooling: add Go and Lua lint targets","description":"PLAN.md section 6 lists golangci-lint for Go and luacheck for Lua. Add configured lint targets and decide whether they run locally only or in CI. Keep generated files excluded where appropriate.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:28Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-6rb","title":"Tooling: add coverage targets for plugin and runtime","description":"PLAN.md section 6 lists go test -cover for the Go plugins and luacov for the Lua runtime. Add repeatable coverage targets and document expected use. Decide whether coverage is informational or gated in CI.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:27Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:27Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/.builds/c-enabled.yml b/.builds/c-enabled.yml new file mode 100644 index 0000000000000000000000000000000000000000..f131265b8ac6ce6582504dd09f9580a93d18984c --- /dev/null +++ b/.builds/c-enabled.yml @@ -0,0 +1,34 @@ +image: ubuntu/noble +packages: + - curl + - ca-certificates + - gnupg + - gcc + - make + - golang + - protobuf-compiler +sources: + - https://sourcecraft.dev/bigbes/tarantool-protobuf +environment: + GOPATH: /home/build/go + PATH: /home/build/.local/bin:/home/build/go/bin:/usr/local/bin:/usr/bin:/bin + PB_ENABLE_C: "1" +tasks: + - install_tarantool: | + curl -L https://tarantool.io/release/3/installer.sh | sudo bash + sudo apt-get install -y tarantool tarantool-dev tt + - install_just: | + mkdir -p ~/.local/bin + curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh \ + | bash -s -- --to ~/.local/bin + - install_luatest: | + cd tarantool-protobuf + tt rocks install luatest + - build_c_runtime: | + cd tarantool-protobuf + just build-c + - build_gen_test: | + cd tarantool-protobuf + just build + just gen + just test diff --git a/.builds/pure-lua.yml b/.builds/pure-lua.yml new file mode 100644 index 0000000000000000000000000000000000000000..9f29ab066cd0cd636fbb490e6a3f5563a6f42b1a --- /dev/null +++ b/.builds/pure-lua.yml @@ -0,0 +1,28 @@ +image: ubuntu/noble +packages: + - curl + - ca-certificates + - gnupg + - golang + - protobuf-compiler +sources: + - https://sourcecraft.dev/bigbes/tarantool-protobuf +environment: + GOPATH: /home/build/go + PATH: /home/build/.local/bin:/home/build/go/bin:/usr/local/bin:/usr/bin:/bin +tasks: + - install_tarantool: | + curl -L https://tarantool.io/release/3/installer.sh | sudo bash + sudo apt-get install -y tarantool tarantool-dev tt + - install_just: | + mkdir -p ~/.local/bin + curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh \ + | bash -s -- --to ~/.local/bin + - install_luatest: | + cd tarantool-protobuf + tt rocks install luatest + - build_gen_test: | + cd tarantool-protobuf + just build + just gen + just test diff --git a/.gitignore b/.gitignore index 150e9a9f11da79bfab275f30067e12f310abd7b7..495acf71daf2bd88234be5054fcb72b20c5b9215 100644 --- a/.gitignore +++ b/.gitignore @@ -19,3 +19,10 @@ *.run *.pid 512.lock + +# C-acceleration build output (bd-wky). Source under runtime/pb/c/ when +# present is committed; the compiled module sits next to the Lua files +# so require('pb.c_runtime') resolves it via LUA_CPATH. +/runtime/pb/c_runtime.so +/runtime/pb/c_runtime.dylib +/runtime/pb/c/*.o diff --git a/.sourcehut/conformance.yml b/.sourcehut/conformance.yml new file mode 100644 index 0000000000000000000000000000000000000000..d34a47eaff668b24ff730d457a5e85fd023fb94f --- /dev/null +++ b/.sourcehut/conformance.yml @@ -0,0 +1,33 @@ +# Conformance manifest — submit manually with: +# hut builds submit .sourcehut/conformance.yml +# Not under .builds/ because Docker pulls + the full proto3/proto2 suite are +# heavy; we don't want this on every push. Run before a release or after +# wire-format changes. +image: ubuntu/noble +packages: + - curl + - ca-certificates + - docker.io + - golang + - protobuf-compiler +sources: + - https://sourcecraft.dev/bigbes/tarantool-protobuf +environment: + GOPATH: /home/build/go + PATH: /home/build/.local/bin:/home/build/go/bin:/usr/local/bin:/usr/bin:/bin +tasks: + - install_tarantool: | + curl -L https://tarantool.io/release/3/installer.sh | sudo bash + sudo apt-get install -y tarantool tarantool-dev tt + - install_just: | + mkdir -p ~/.local/bin + curl --proto '=https' --tlsv1.2 -sSf https://just.systems/install.sh \ + | bash -s -- --to ~/.local/bin + - start_docker: | + sudo systemctl start docker || sudo service docker start + sudo usermod -aG docker $USER + - conformance: | + cd tarantool-protobuf + just build + just gen + just conformance diff --git a/Justfile b/Justfile index 4b5ed31a26793f598275c095441528028059b116..1f93548f219cb73dd015ccdb72598e05d9ebd7b1 100644 --- a/Justfile +++ b/Justfile @@ -34,6 +34,12 @@ image := "tarantool-protobuf-conformance:latest" # standard package.path for everything not explicitly listed. lua_path := "./runtime/?/init.lua;./runtime/?.lua;./" + gen_dir + "/?.lua;./" + gen_dir + "/?/init.lua;./?.lua;./?/init.lua;./test/?.lua;;" +# LUA_CPATH for the optional C runtime (require('pb.c_runtime')). Trailing +# `;;` defers to the standard cpath. The C module is built into +# runtime/pb/c_runtime.{so,dylib} by `just build-c` when PB_ENABLE_C=1 is +# in play; absent without it. See docs/specs/c_accel_build_packaging.md. +lua_cpath := "./runtime/?.so;./runtime/?.dylib;./runtime/?/init.so;./runtime/?/init.dylib;;" + # --------------------------------------------------------------------------- # Default # --------------------------------------------------------------------------- @@ -57,6 +63,24 @@ build: build-doc: go build -o {{doc_plugin}} ./cmd/protoc-gen-tarantool-doc +# Build the optional C-acceleration runtime into runtime/pb/c_runtime.{so,dylib}. +# Source lives under runtime/pb/c/ once bd-ra6 lands; until then this recipe +# prints a helpful error and exits 1. The runtime is opt-in via PB_ENABLE_C=1 +# — see docs/specs/c_accel_build_packaging.md for the full contract. +build-c: + @if [ ! -d runtime/pb/c ]; then \ + echo "build-c: runtime/pb/c/ does not exist yet."; \ + echo " The C runtime arrives with bd-ra6 (generic C codec)."; \ + echo " See docs/specs/c_accel_build_packaging.md."; \ + exit 1; \ + fi + $(MAKE) -C runtime/pb/c + +# Remove built C-runtime artifacts. +clean-c: + rm -f runtime/pb/c_runtime.so runtime/pb/c_runtime.dylib + @if [ -d runtime/pb/c ]; then $(MAKE) -C runtime/pb/c clean; fi + # --------------------------------------------------------------------------- # Codegen # --------------------------------------------------------------------------- @@ -142,12 +166,12 @@ goldens: # Run the luatest suite (639 tests, parametrized over both codegen modes). test: gen - LUA_PATH="{{lua_path}}" {{luatest}} -v test/ + LUA_PATH="{{lua_path}}" LUA_CPATH="{{lua_cpath}}" {{luatest}} -v test/ # Run a single luatest group or test. Example: # just test-one protobuf_test.lua::hello.full.test_packed_repeated_int32 test-one filter: gen - LUA_PATH="{{lua_path}}" {{luatest}} -v test/{{filter}} + LUA_PATH="{{lua_path}}" LUA_CPATH="{{lua_cpath}}" {{luatest}} -v test/{{filter}} # --------------------------------------------------------------------------- # Bench @@ -240,7 +264,7 @@ examples *ARGS: # --------------------------------------------------------------------------- # Remove built plugin binaries and regenerated outputs. -clean: +clean: clean-c rm -f {{plugin}} {{doc_plugin}} rm -rf {{gen_dir}} diff --git a/docs/specs/c_accel_build_packaging.md b/docs/specs/c_accel_build_packaging.md new file mode 100644 index 0000000000000000000000000000000000000000..a785f70c59946e3408c74ab2c780ba7bc5ee37cf --- /dev/null +++ b/docs/specs/c_accel_build_packaging.md @@ -0,0 +1,229 @@ +# Spec: C-acceleration build and packaging + +Status: **scaffolding landed, C module pending**. This spec records +how the C-accelerated runtime is built, installed, and exercised in +CI. The opt-in hook in `runtime/pb/init.lua`, the `just build-c` +recipe, and the CI manifests under `.builds/` land alongside this +doc; the actual C source they refer to arrives with `bd-ra6`. + +Closes `bd-wky`. + +## Where the C module lives + +``` +runtime/pb/ +├── c/ (NEW — C runtime sources) +│ ├── Makefile (mirrors bench/c_accel/Makefile) +│ ├── c_runtime.c (entry points + plan struct) +│ ├── plan.c (descriptor -> plan compiler, bd-mq7) +│ ├── encode.c (bd-3b) +│ ├── decode.c (bd-3g) +│ ├── buf.c (4 KB stack-backed buffer) +│ ├── wkt.c (override dispatch, bd-3k) +│ └── ... +├── c_runtime.so (built output — gitignored) +├── c_runtime.dylib (macOS build output — gitignored) +├── init.lua +├── codec.lua +└── ... +``` + +The build output sits alongside the Lua files at `runtime/pb/c_runtime.{so,dylib}` +so `require('pb.c_runtime')` resolves it via Lua's package.cpath once +the test runner adds `./runtime/?.so` to LUA_CPATH (see [Test runner](#test-runner-changes)). + +## Build entry points + +### `just build-c` + +``` +$ just build-c +make -C runtime/pb/c +cc -O2 -fPIC -Wall -Wextra -std=c99 -I/usr/include/tarantool ... \ + -o runtime/pb/c_runtime.dylib runtime/pb/c/c_runtime.c ... +``` + +Mirrors `bench/c_accel/Makefile`: same `TT_INC` auto-detection (env +override, brew prefix, common system dirs), same `-fPIC -O2`, same +platform-aware extension (`.dylib` bundle on macOS, `.so` on Linux). + +The recipe is a stub today — it runs `make -C runtime/pb/c` and +errors out cleanly with "no C runtime source yet" until `bd-mq7` +lands the first source files. + +### `just clean-c` + +Removes `runtime/pb/c_runtime.so` and `runtime/pb/c_runtime.dylib`. +Folded into `just clean`. + +### `just build` and `just test` + +Unchanged. Building the C module is **not** part of the default +flow. Users who care about C must invoke `just build-c` explicitly. + +This keeps the default contributor experience identical for anyone +not working on the C path — `just test` continues to work without +a C compiler. + +## Activation + +The single switch is the environment variable `PB_ENABLE_C`. See +[c_accel_compat.md § Activation](c_accel_compat.md#activation) for +the full contract. + +`runtime/pb/init.lua` does: + +```lua +local c_runtime +if os.getenv('PB_ENABLE_C') == '1' then + local ok, mod = pcall(require, 'pb.c_runtime') + if ok then c_runtime = mod end +end +``` + +When `c_runtime` is non-nil, `pb.finalize_message(desc)` calls into +it to compile a `desc.c_plan` userdata. When `c_runtime` is nil, +nothing changes — the pure-Lua path runs unchanged. + +This hook is **wired today**; the finalize-time compile call is +deferred to `bd-mq7`. + +## Test runner changes + +The Justfile's `lua_path` constant covers Lua module resolution. +For the C module, the test recipe also sets `LUA_CPATH`: + +```just +lua_cpath := "./runtime/?.so;./runtime/?.dylib;./runtime/?/init.so;;" + +test: gen + LUA_PATH="{{lua_path}}" LUA_CPATH="{{lua_cpath}}" {{luatest}} -v test/ +``` + +This is added by the wky landing PR. With `runtime/pb/c_runtime.so` +built, `require('pb.c_runtime')` resolves to that path. + +`tt rocks install` puts compiled modules under `.rocks/lib/tarantool/`, +which is already on the default cpath; the wky changes only affect +the local dev workflow, not installed rocks. + +## Rockspec + +The current rockspec uses `build.type = "builtin"` which copies Lua +files only. To optionally build the C module, the rockspec gains: + +```lua +external_dependencies = { + TARANTOOL = { header = "module.h" }, +} + +build = { + type = "builtin", + modules = { + ["pb"] = "runtime/pb/init.lua", + -- ... existing Lua modules ... + -- The C module is listed as a buildable source: + ["pb.c_runtime"] = { + sources = { + "runtime/pb/c/c_runtime.c", + "runtime/pb/c/plan.c", + "runtime/pb/c/encode.c", + "runtime/pb/c/decode.c", + }, + incdirs = { "$(TARANTOOL_INCDIR)" }, + }, + }, +} +``` + +Behavior: + +- `tt rocks install` (or `luarocks install`) detects the C compiler + and `module.h` via `external_dependencies`. If both are present, + the C module compiles. If `module.h` is missing (older Tarantool, + no `-dev` package), the install **fails** because the rockspec + declares it as a hard dependency. +- To make the C build truly optional (install succeeds on hosts + without `module.h`), we'd need to use `luarocks`-specific + `build.platforms` or two separate rockspec variants. Deferred: + for now, if the user installs and doesn't have `tarantool-dev`, + installation fails with a clear error message — acceptable + because every supported Tarantool environment ships `module.h`. + +The rockspec change lands when the first C sources exist (`bd-mq7` +or `bd-ra6`). The wky scaffolding leaves the rockspec at "builtin +Lua only" for now and documents the change here. + +## CI + +Two manifests under `.builds/`: + +- **`.builds/pure-lua.yml`** — full test suite with `PB_ENABLE_C` + unset. Validates the pure-Lua path and the silent-fallback case + (env var unset means C is dormant even when present). +- **`.builds/c-enabled.yml`** — same test suite plus `just build-c` + and `PB_ENABLE_C=1`. Validates the C path end-to-end. + +Both run on `ubuntu/noble`. Sourcehut submits both per push (no +matrix; each `.builds/*.yml` is an independent job). + +Until the repo is pushed to a sourcehut-compatible host (or a +webhook from sourcecraft.dev fires `hut builds submit`), the +manifests sit idle. They start producing value the day the remote +lands. No CI-side configuration is required beyond the manifests +themselves. + +A third manifest at **`.sourcehut/conformance.yml`** (deliberately +outside `.builds/` so it doesn't auto-submit) runs the Google +proto3 conformance suite inside the Docker harness. Submit it +manually with `hut builds submit .sourcehut/conformance.yml` before +a release or after wire-format changes. Skipping it from every-push +CI is a deliberate cost-control choice — the suite pulls a multi-GB +Docker image and runs ~7000 tests; it doesn't belong on the +fast-feedback path. + +## Platform support + +The `bench/c_accel/Makefile` patterns we mirror in `runtime/pb/c/Makefile`: + +- **Linux x86_64 / aarch64** — `-shared`, output is `.so`. +- **macOS arm64 / x86_64** — `-bundle -undefined dynamic_lookup`, + output is `.dylib`. The `dynamic_lookup` flag is what lets the + module use `lua_*` symbols without linking against a specific + Lua/Tarantool binary. +- **Windows / WSL / FreeBSD** — out of scope for the initial ship. + Tarantool's primary deployment targets are Linux servers and + macOS dev hosts; FreeBSD ports exist but aren't on the CI + matrix. + +## What ships with `bd-wky` today + +- `runtime/pb/init.lua` — `PB_ENABLE_C` pcall hook (lines added at + module load; exposes `c_runtime` field on the returned table for + introspection). +- `Justfile` — `build-c`, `clean-c` recipes; updated `lua_cpath` + variable; `test` recipe sources `LUA_CPATH`. +- `.builds/pure-lua.yml`, `.builds/c-enabled.yml` — CI matrix. +- `.sourcehut/conformance.yml` — manual conformance trigger. +- `.gitignore` — `runtime/pb/c_runtime.{so,dylib}` ignored. +- This spec. + +## What does NOT ship with `bd-wky` + +- No actual C source. `runtime/pb/c/` doesn't exist; `just build-c` + errors with a clear message until `bd-ra6` populates the + directory. +- No rockspec changes (still pure-Lua install). The rockspec + changes land with the first C source files (likely `bd-mq7` or + `bd-ra6`). +- No `finalize_message` C-plan compile call. Lives with `bd-mq7`. + +## References + +- [docs/c-accel.md](../c-accel.md) — architecture +- [docs/specs/c_accel_compat.md](c_accel_compat.md) — compat contract +- [docs/specs/c_accel_strategy.md](c_accel_strategy.md) — C-side strategy +- `bench/c_accel/Makefile` — reference build, the patterns here are + the patterns there +- `.builds/{pure-lua,c-enabled}.yml` — CI manifests +- `.sourcehut/conformance.yml` — manual conformance manifest diff --git a/docs/specs/c_accel_compat.md b/docs/specs/c_accel_compat.md new file mode 100644 index 0000000000000000000000000000000000000000..921bdacf06fff833d4bcb0f77c8480e9b7a9a728 --- /dev/null +++ b/docs/specs/c_accel_compat.md @@ -0,0 +1,310 @@ +# Spec: C-acceleration compatibility contract + +Status: **contract locked, implementation pending**. This spec +formalizes the compatibility boundaries that the C-acceleration work +([`docs/c-accel.md`](../c-accel.md)) must preserve. It is the +reference that `bd-mq7` (plan compiler), `bd-ra6` (generic C runtime), +`bd-c0i` (deferred codegen-C), and `bd-43t` (parity gate) consume to +decide what is — and isn't — allowed to change. + +Closes `bd-47e`. + +## Principle + +The C path is an **opt-in performance enhancement**, not a new mode. +Every Lua-visible behavior must be byte-equal between `PB_ENABLE_C` +unset and `PB_ENABLE_C=1`. A user who never sets the variable should +never observe any difference — including memory layout of returned +values, error messages, error types, iteration order of returned +tables, or the structure of generated `_pb.lua` modules. + +If a C-path optimization conflicts with this principle, the C path +loses. There are no exceptions and no per-feature carve-outs. + +## Activation + +Exactly one switch: the environment variable `PB_ENABLE_C`. + +- Read once at module load (`runtime/pb/init.lua`), via + `os.getenv('PB_ENABLE_C')`. Value `'1'` enables; anything else + (including unset, `'0'`, `'true'`, `''`) leaves the C path off. +- No Lua-side toggle (`pb.use_c_runtime = true` is **not** part of + the surface). No per-call override. No per-message override. The + knob is out-of-band by design — keeping it out of Lua makes + accidental activation impossible and prevents the C path from + becoming a feature with its own API surface. +- When `PB_ENABLE_C=1` but `require('pb.c_runtime')` fails (no `.so` + built, ABI mismatch, etc.), `runtime/pb/init.lua` falls back to + the pure-Lua path **silently**. No warning, no error. A failed + `pcall` is indistinguishable from `PB_ENABLE_C` unset for every + downstream observer. Rationale: a noisy fallback turns a missing + build artifact into a runtime error for users who didn't ask for + C anyway (rockspec installed without compiler available). + +## Public Lua surface — preserved + +The following modules and their public functions must keep their +exact signatures, argument types, return types, and error behavior +in both paths: + +### `runtime/pb/init.lua` (the `pb` module) + +| Function | Behavior | Test coverage | +|----------|----------|---------------| +| `pb.encode(desc, t)` → string | Same bytes. | `test/interop/*`, `test/parity_full_vs_runtime_test.lua` | +| `pb.decode(desc, bytes)` → table | Same table shape. Same iteration order is **not** guaranteed in either path (Lua hash order), but tests must remain green. | `test/interop/*` | +| `pb.decode_lazy(desc, bytes)` → MessageView | Returns the same `MessageView` API. See [Lazy view](#lazy-view-pbdecode_lazy). | `test/lazy_test.lua` | +| `pb.parse(src)` → descriptor module | Pure Lua, parser stays in Lua. **Not affected by C path.** | `test/dynamic_test.lua` | +| `pb.from_pb(bytes)` → descriptor module | Same. | `test/fileset_test.lua` | +| `pb.json.encode` / `pb.json.decode` | Same JSON output. The JSON codec composes on top of the encoder/decoder, so it picks up C automatically without code change. | `test/json_test.lua` | +| `pb.text.encode` / `pb.text.decode` | Same as JSON. | `test/text_test.lua` | +| `pb.grpc.loopback` / `multiplex` | gRPC transports treat encoded bytes as opaque; not affected. | `test/grpc_test.lua` | +| `pb.finalize_message(desc)` | Existing call site; gains side effect of compiling the C plan when `PB_ENABLE_C=1`. Return value unchanged (`nil`). Idempotent. | implicit (used by all generated modules) | + +Generated modules (`mode=full` and `mode=runtime`): + +| Symbol | Behavior | +|--------|----------| +| `M._encode(t)` → string | Same bytes. Wrapper checks `desc.c_plan`; if present, calls `pb.c_runtime.encode(desc.c_plan, t)`. Otherwise runs existing pure-Lua body. | +| `M._decode(b)` → table | Same. Mirrors encode wrapper. | +| `M._descriptor` | Unchanged shape. Gains optional `c_plan` field at finalize time (opaque userdata). | +| `M._fields` / `M._oneofs` | Strict-table wrappers used by lazy view; unchanged. | +| `M._service` / `M._client` / `M._server` | gRPC factories; unchanged. | + +### What the C plan **may not** change + +- The on-disk format of `examples/expected/{full,runtime}/*_pb.lua`. + Files in version control must stay byte-equal across this work. + C-plan compilation happens at module load, in memory, from + descriptor tables — no codegen-time change. +- The structure of `desc` tables documented in + [docs/codegen.md](../codegen.md#the-descriptor-table--the-contract). + Plan compilation reads `desc`; it does not mutate it except for + attaching the opaque `c_plan` userdata. +- Generated module return value (`return M`). No new exports. + +## 64-bit integers + +`int64`, `uint64`, `sint64`, `fixed64`, `sfixed64` are LuaJIT +`int64_t` / `uint64_t` cdata in **both** paths. + +- Encode accepts `cdata`, Lua number (within ±2^53), or string of + digits (existing behavior in `pb.codec`). The C path must accept + all three. Implementation hook: `bd-3l`. +- Decode always returns `cdata`. The pure-Lua decoder already + returns cdata; the C decoder must use + `luaT_pushint64` / `luaT_pushuint64` (Tarantool's `module.h` + extensions) so the value lands on the Lua stack as cdata, not as + a Lua number. +- `tostring` of a returned 64-bit value must produce the same + string in both paths. (Lua's default `cdata` `__tostring` already + guarantees this.) + +This rule is **load-bearing for downstream users.** msgpackffi, +net.box, box.tuple, IProto all consume cdata. Narrowing to double +silently corrupts IDs and timestamp nanos past 2^53. The pure-Lua +path holds this line today, and the C path must hold it tomorrow. + +## WKT shapes + +`google.protobuf.*` types keep their hand-rolled shapes +(`runtime/pb/wkt.lua`): + +- `Timestamp` / `Duration` accept and return a `datetime` cdata, + with `{seconds=, nanos=}` table accepted on encode for ergonomics. +- `Empty` is `box.NULL`. +- Wrappers (`Int32Value`, `StringValue`, …) accept and return the + wrapped scalar directly. +- `Any`, `Struct`, `Value`, `ListValue` keep their existing + representations. + +Mechanism: WKT descriptors carry `desc.encode` / `desc.decode` +function fields. The C plan, when finalizing a descriptor, must +detect those overrides and arrange to call the Lua functions for +that descriptor's encode/decode instead of walking fields in C. +This is the same composition point that lets WKT plug into the +pure-Lua codec today (`runtime/pb/codec.lua` short-circuits on +`desc.encode` / `desc.decode`). + +Implementation hook: `bd-3k`. + +## Unknown fields + +The pure-Lua decoder preserves unknown fields in +`t._unknown_fields` (a string of raw wire bytes), and the encoder +appends them after the known fields. The C path must round-trip +identically: + +- Decode of a message with unknown wire bytes returns a table + whose `_unknown_fields` is the same byte string the pure-Lua + decoder would produce. +- Encode of a table with `_unknown_fields` set appends those bytes + verbatim after known-field output. + +Conformance gates this — every +`Required.Proto3.ProtobufInput.ValidDataUnknown.*` test exercises +this round-trip. The C path runs the same conformance suite, so +regressions surface immediately. Implementation hook: `bd-3j`. + +## Extensions (proto2) + +Extension representation in tables: `t[]` where +`` is the fully-qualified extension field name. Same +in both paths. + +Extension range handling on decode: unknown extension numbers +fall through to `_unknown_fields` (above). Known extension numbers +populate `t[]`. + +Implementation hook: `bd-3i`. + +## Lazy view (`pb.decode_lazy`) + +The lazy view is **out of scope for C acceleration in the initial +ra6 ship.** `pb.decode_lazy` continues to use the pure-Lua path +in `runtime/pb/lazy.lua` regardless of `PB_ENABLE_C`. + +Rationale: the lazy view's value is *not* doing a full decode. Its +hot operations are `MessageView:get(name)` and `:has(name)`, which +do a single field scan over the wire bytes — already O(field_pos) +in pure Lua, not the per-message dispatch cost C-accel targets. +Adding a C entry point per `:get` call would re-introduce the +boundary cost the architecture sketches were trying to avoid. + +Future work (deferred, no BD issue yet): a `lazy_c` variant could +provide a single C entry point for `:get_many({names})` if a +workload surfaces. Until then, the lazy view stays Lua-only. + +The lazy view's **public API** is unchanged either way — +`:get(name)`, `:has(name)`, `:which(oneof)`, `:set`, `:clear`, +`:iter`, `:names`, `:to_table`, plus `ArrayView` and `MapView` +methods. Users observe no difference. + +## Errors + +The C path raises Lua errors via `luaL_error`, producing the same +shape as `error(string)` in pure Lua. Specifically: + +- Type mismatches (wrong field type in input table) → string error + with the same template as `runtime/pb/codec.lua` ("field X + expected Y, got Z" or close). +- Wire-format errors on decode → string error matching the + pure-Lua message for the same input. +- 64-bit cdata required but Lua number out of range → same + message as `wire.encode_int64` raises today. + +This is **expected-output compatibility**, not source-text +compatibility. The exact prose of error messages may shift +slightly between paths during ra6 implementation; the contract is +that error *type* (Lua error vs. silent return) and approximate +message content (mentions the field name and the type involved) +must match. Tests that pattern-match on error strings should use +`string.find` with anchor patterns, not full-string equality. + +Implementation hook: covered cross-cutting in ra6, not a separate +BD issue. + +## Installation paths + +Two paths to install: + +### Pure-Lua install (no compiler) + +``` +tt rocks install tarantool-protobuf +``` + +- Rockspec's optional C build skips when no `cc` available. +- Install succeeds. Only Lua files land. +- `pb.encode` / `pb.decode` work; `pb.c_runtime` is absent. +- Setting `PB_ENABLE_C=1` is silently ignored. +- This is the **default install path** for everyone who doesn't + ask for C. + +### C-enabled install + +``` +tt rocks install tarantool-protobuf +PB_ENABLE_C=1 tarantool app.lua +``` + +- Same rockspec, same `tt rocks install` command. The rockspec + detects `cc` and builds `pb/c_runtime.so` (or `.dylib` on + macOS). The same install command produces a C-enabled install + when a compiler is present. +- User sets `PB_ENABLE_C=1` to activate. + +Implementation hook: `bd-wky`. + +## ABI and version skew + +The C module's exposed symbols (essentially `pb.c_runtime.encode` +and `pb.c_runtime.decode`, plus the plan-compilation entry point) +are versioned via a single `pb.c_runtime._abi_version` integer +string. `runtime/pb/init.lua` checks it on load and treats a +mismatch as a load failure (same fallback as missing module). + +The plan userdata is **not** ABI. It's process-local, opaque, and +recompiled at every module load. Cross-process plan sharing is +out of scope (and would also require `desc` to be serializable, +which it isn't today). + +Implementation hook: covered in `bd-wky` packaging discussion. + +## Conformance and parity + +No new Lua-vs-C diff harness. The existing test infrastructure +asserts against reference outputs (golden byte strings, txtpb +fixtures, conformance results). When both paths run the same +suite and both pass, byte equality is implied by transitivity. + +CI matrix runs the full suite twice — `PB_ENABLE_C` unset and +`PB_ENABLE_C=1`. Both must pass for every merge to master. + +Implementation hook: `bd-43t`. + +## What changes + +To be unambiguous about what this work *does* alter: + +- `runtime/pb/init.lua` gains a load-time `pcall(require, 'pb.c_runtime')` + guarded on `PB_ENABLE_C=1`. (~5 lines.) +- `pb.finalize_message(desc)` gains a load-time compile step that + attaches `desc.c_plan` when the C runtime is loaded. (~5 lines + in the existing finalize function; the actual compile lives in + `pb.c_runtime`.) +- Each generated `M._encode` / `M._decode` wrapper + gains a one-line dispatch check. The pure-Lua body underneath + is untouched. +- New file `runtime/pb/c_runtime.c` (or split across several files + under `runtime/pb/c/`) — the actual C implementation. +- New file `runtime/pb/c_runtime.lua` — a thin Lua wrapper around + the C module that performs ABI check and exposes plan + compilation and encode/decode functions. Could also be inlined + into the C module itself; deferred to `bd-mq7` / `bd-ra6`. +- Justfile gains `build-c` recipe. +- Rockspec gains optional C build hook. +- Sourcehut `.build.yml` gains a `PB_ENABLE_C=1` job. + +That's the complete delta. Nothing else in the runtime tree, +nothing in the codegen tree, nothing in `examples/expected/`, +nothing in `test/`. + +## Out of scope + +- Replacing the pure-Lua path. `mode=full` and `mode=runtime` + remain the default forever. +- C-acceleration for `pb.decode_lazy` (see above). +- C-acceleration for `pb.parse` (Lua parser, not on hot path). +- C-acceleration for JSON / text codecs beyond what they + inherit by composing on top of `pb.encode` / `pb.decode`. +- Per-call activation, per-message activation, runtime toggle. +- Cross-process plan caching. + +## References + +- [docs/c-accel.md](../c-accel.md) — architecture +- [docs/specs/c_accel_strategy.md](c_accel_strategy.md) — C-level design +- [docs/codegen.md](../codegen.md#the-descriptor-table--the-contract) — descriptor contract +- `bd-pf6` (closed) — architecture decision +- `bd-mq7`, `bd-ra6`, `bd-43t`, `bd-wky` — downstream work diff --git a/docs/specs/c_accel_strategy.md b/docs/specs/c_accel_strategy.md new file mode 100644 index 0000000000000000000000000000000000000000..51dd139f86fe6f4b77e0c675a23b45d7fbc5576a --- /dev/null +++ b/docs/specs/c_accel_strategy.md @@ -0,0 +1,433 @@ +# Spec: C-acceleration C-side strategy + +Status: **design locked, implementation pending**. This spec records +the low-level C-side decisions that the generic runtime +(`bd-ra6`) and its plan-compiler foundation (`bd-mq7`) inherit. +Every choice here is validated by the `bench/c_accel/` spike — see +[bench/c_accel/README.md](../../bench/c_accel/README.md) for the +microbenchmark numbers and the `c-accel-spike-04c-phase-{a,b}` +memos for the per-strategy commentary. + +Closes `bd-z7x`. + +## Scope + +This spec is about how C code interacts with the Lua state — field +access, table allocation, string handling, stack discipline, 64-bit +cdata. It is **not** about the wire format (which the pure-Lua +`runtime/pb/wire.lua` already specifies) or about the descriptor +shape (which [docs/codegen.md](../codegen.md) specifies). + +If a strategy decision here conflicts with the +[compatibility contract](c_accel_compat.md), the contract wins. + +## Strategy overview + +Strategy 3 from the spike (`S3`): **one C entry point per top-level +encode or decode call.** The C side walks a compiled descriptor +plan, reads/writes the wire bytes, and crosses back to Lua once at +the end. Per-field dispatch happens in C, not in Lua. + +Strategy 2 (per-primitive FFI) was rejected: spike showed every +size on decode is worse than pure Lua, and ≥1 KB encode regresses. +Strategy 4 (codegen-emitted C per message) is deferred (`bd-c0i`) +— ≤15% over S3 on the spike doesn't justify the codegen complexity +without a real-workload trigger. + +## The plan userdata + +Each finalized message descriptor gains an opaque `c_plan` +userdata when `PB_ENABLE_C=1` and the C module loads. The plan is: + +- Allocated once, at module load time (during + `pb.finalize_message(desc)`). +- Process-local. Not serialized, not shared, not cached. +- Anchored on the descriptor table to share its lifetime; freed by + GC when the descriptor itself drops. +- Opaque to Lua. Only the C runtime reads its internals; Lua sees + it as a regular userdata. + +### Plan layout + +```c +typedef struct pb_plan { + int n_fields; + pb_plan_field *fields; /* sorted by field number for binary search on decode */ + pb_plan_field *fields_by_idx; /* iteration order for encode (proto declaration order) */ + int n_oneofs; + pb_plan_oneof *oneofs; + int extension_range_start; /* proto2 only; 0 if none */ + int extension_range_end; + int has_override; /* 1 if desc.encode/desc.decode set */ + int override_encode_ref; /* luaL_ref slot in REGISTRY */ + int override_decode_ref; /* luaL_ref slot in REGISTRY */ + int field_names_ref; /* slot for { [1]=name, [2]=name, ... } table */ + int sub_plans_ref; /* slot for { [1]=sub_plan_userdata, ... } table */ +} pb_plan; + +typedef struct pb_plan_field { + uint32_t field_number; + uint8_t wire_type; /* 0,1,2,5 */ + uint8_t kind; /* PB_KIND_INT32 / _STRING / _SUBMSG / _MAP / ... */ + uint8_t packed; /* 1 if packed repeated */ + uint8_t repeated; /* 1 if repeated (incl. packed) */ + int tag_bytes_len; /* pre-encoded tag bytes */ + uint8_t tag_bytes[5]; /* enough for any field number; tag fits in <= 5 bytes */ + int sub_plan_idx; /* index into sub_plans table; -1 for scalars */ + int map_key_kind; /* PB_KIND_* for map key; PB_KIND_NONE if not map */ + int map_value_kind; + int map_value_sub_plan_idx; + int oneof_idx; /* index into plan->oneofs; -1 if not in oneof */ + /* No name string stored in the struct — looked up via field_names_ref + index. */ +} pb_plan_field; +``` + +Specifics: + +- `tag_bytes` is the pre-encoded `(field_number << 3) | wire_type` + varint. Same trick `mode=full` uses today; same logic. Up to 5 + bytes because the highest legal field number is 2^29-1. +- Field-name strings are *not* embedded in `pb_plan_field`. They + live in a Lua table referenced by `field_names_ref` and are + looked up via `lua_rawgeti(L, LUA_REGISTRYINDEX, ref); lua_rawgeti(L, -1, i+1)`. + Rationale: `luaL_ref` makes the strings durably reachable + without per-plan refcounting, and `lua_rawgeti` is faster than + re-interning a C string via `lua_pushstring` on every call. + Validated by spike Phase A. +- `sub_plan_idx` indirection (rather than a direct `pb_plan*` + pointer) lets plans reference each other safely without + invasive lifetime tracking; the sub-plans table holds strong + refs to the userdata. GC handles cleanup. + +## Field-name caching + +The single largest source of per-field overhead in a naive C +codec is repeated `lua_pushstring("field_name")` calls — each does +a hash and intern lookup. We avoid it via the `field_names_ref` +table: + +```c +/* At plan compile time, once per field: */ +lua_pushstring(L, desc.fields[i].name); +lua_rawseti(L, names_table, i + 1); + +/* In encode loop, per field: */ +lua_rawgeti(L, LUA_REGISTRYINDEX, plan->field_names_ref); +lua_rawgeti(L, -1, i + 1); /* pushes name string */ +lua_gettable(L, input_table_stack_idx); /* pops name, pushes value */ +``` + +The `lua_rawgeti` pair is two table indexes on already-interned +strings; no hashing of "field_name" required. Spike Phase A +measured this as ~30 ns/field cheaper than per-field +`lua_getfield` with literal C strings. + +For very wide messages (>20 fields), the indirection through +`field_names_ref` becomes another lookup. We could specialize +small messages with inline name refs (each `pb_plan_field` carries +its own `name_ref`), but the spike didn't measure a win for that +shape — defer until profiling motivates it. + +## Output buffer + +Per-call, stack-backed buffer with malloc promotion on overflow. +Pattern validated in `bench/c_accel/person_codec.c`: + +```c +typedef struct pb_buf { + uint8_t *data; + size_t len; + size_t cap; + uint8_t stack[4096]; +} pb_buf; + +static inline void pb_buf_init(pb_buf *b) { + b->data = b->stack; + b->len = 0; + b->cap = sizeof(b->stack); +} + +static inline void pb_buf_reserve(pb_buf *b, size_t extra) { + if (b->len + extra <= b->cap) return; + size_t new_cap = b->cap * 2; + while (new_cap < b->len + extra) new_cap *= 2; + uint8_t *new_data = malloc(new_cap); + memcpy(new_data, b->data, b->len); + if (b->data != b->stack) free(b->data); + b->data = new_data; + b->cap = new_cap; +} + +static inline void pb_buf_free(pb_buf *b) { + if (b->data != b->stack) free(b->data); +} +``` + +- 4 KB stack cap covers most production message sizes without a + malloc round-trip. The spike measured 0 promotions at 100 B and + 1 KB, ~1 at 10 KB, several at 100 KB — and the 100 KB cost was + dominated by Lua table reads, not allocator behavior. +- Doubling growth, not arithmetic. The 100 KB encode does at most + 4 promotions (4K → 8K → 16K → 32K → 64K → 128K), each + amortizable. +- `pb_buf_free` is mandatory in the exit path even on Lua error; + use a finalizer userdata or a `lua_State*`-attached error + handler. (Cleanest: allocate the buffer as part of a userdata + with `__gc`. See implementation note below.) + +Why not `luaL_Buffer`? Spike Phase A tested it. `luaL_Buffer` +copies to a Lua-side string buffer at each `luaL_addchar` and +hits the same hash-string-allocation cost on `luaL_pushresult`. +The stack-backed approach measured 18% faster on 1 KB encodes. + +### Buffer ownership and error safety + +Buffer is allocated inside a userdata with `__gc` set to +`pb_buf_free`. The C entry point creates the userdata, runs the +encode (which may longjmp via `luaL_error`), and on success +constructs the result string via `lua_pushlstring(L, b->data, b->len)` +and returns 1. On longjmp, the userdata's `__gc` runs during stack +unwind and `pb_buf_free` reclaims any malloc'd memory. + +Implementation hook: `bd-3e` (encode loop), `bd-3g` (decode loop). + +## Result tables (decode) + +Decode pre-sizes the result table from descriptor stats: + +```c +lua_createtable(L, 0, plan->n_fields); +``` + +This sets the hash part's initial bucket count, avoiding rehashes +as fields populate. Repeated-field arrays are similarly pre-sized +once their length is known (the second pass for packed; for +unpacked we resize geometrically as we go). + +The `narr=0` argument matches how the pure-Lua decoder shapes the +table — all named fields land in the hash part. Repeated arrays +live in sub-tables (their `narr` gets sized properly when created). + +## Per-field stack discipline + +The single most important decision from spike Phase B: **cache +per-field stack indices for the duration of one encode or decode +call.** A naive implementation does: + +```c +/* Per field, in the encode loop: */ +lua_pushstring(L, field_name); +lua_gettable(L, input_table_idx); +/* ... use top of stack ... */ +lua_pop(L, 1); +``` + +This is correct but pays the lookup cost on every iteration of a +repeated/packed field. The cached version: + +```c +/* Once at start of field: */ +lua_rawgeti(L, names_table_idx, i + 1); /* push name */ +lua_gettable(L, input_table_idx); /* pop name, push value table */ +int value_stack_idx = lua_gettop(L); + +/* Inner loop iterates value_stack_idx without re-fetching: */ +for (int j = 1; ; j++) { + lua_rawgeti(L, value_stack_idx, j); + if (lua_isnil(L, -1)) { lua_pop(L, 1); break; } + /* encode element */ + lua_pop(L, 1); +} +``` + +Spike Phase B measured the naive version 2× slower than the +cached one at 100 KB. The cached pattern is mandatory for +repeated and packed fields. For singular scalars the difference +is negligible (single lookup); use whichever is cleaner. + +Implementation hook: documented in `bd-3f`; reviewers reject any +PR that re-introduces the naive pattern for repeated fields. + +## 64-bit cdata + +The C side uses Tarantool's `module.h` extensions to push/pull +LuaJIT `int64_t` / `uint64_t` cdata: + +```c +#include + +/* Push as cdata onto the Lua stack: */ +luaT_pushint64(L, (int64_t)val); +luaT_pushuint64(L, (uint64_t)val); + +/* Read cdata or number from stack: */ +int64_t i = luaL_checkint64(L, idx); +uint64_t u = luaL_checkuint64(L, idx); +``` + +`luaL_checkint64` accepts cdata, Lua number, and digit string (the +same forms `wire.encode_int64` accepts in pure Lua). On overflow +or non-numeric input it raises a Lua error with a Tarantool-style +message; the [compat contract](c_accel_compat.md#errors) treats +this as acceptable. + +Implementation hook: `bd-3l`. + +## Sub-messages + +Two approaches were considered: sub-buffer + copy, or +backpatch-after-length. + +### Sub-buffer (chosen) + +Encode the sub-message into a fresh `pb_buf`, then emit +`tag + length-varint + sub_buf.data[0..sub_buf.len]` into the +parent buffer. The sub-buffer is per-call and uses its own stack +backing. + +Pros: simple, no backpatching arithmetic, clean error recovery +(each sub-call allocates and frees its own buffer). + +Cons: one extra memcpy per sub-message; an extra ~4 KB on the +C stack per nesting level. + +The spike's `person_codec.c` uses this; performance was +indistinguishable from backpatching for typical message shapes +(≤5 levels of nesting). Backpatching wins when the same +sub-message has a very long encoding (>1 KB) and avoiding the +memcpy matters. + +### Backpatch (deferred) + +Reserve a worst-case length-varint slot in the parent buffer, +encode the sub-message directly into the parent buffer, then go +back and write the actual length. Requires careful handling: the +length varint can be 1, 2, 3, 4, or 5 bytes; either reserve the +max and emit shorter with leading-zero padding (wastes bytes), or +encode then memmove (slow for large sub-messages), or use the +worst-case slot and rewrite the length-varint encode to accept a +fixed-width target. + +`bd-3d` tracks this as a per-trace optimization for `ra6` if the +sub-buffer approach turns out to be a bottleneck on +deeply-nested workloads. Default is sub-buffer. + +## Packed repeated fields + +Two-pass on encode: + +1. First pass: walk the input array, encode each element into a + dedicated `pb_buf`. Track total length. +2. Second pass: emit `tag + length-varint(total) + buf.data` + into the parent buffer. + +Same pattern the pure-Lua encoder uses (`runtime/pb/codec.lua`'s +`encode_packed`). Validated by spike Phase A as the cleanest +approach. + +Implementation hook: `bd-3f`. + +## Maps + +Maps are syntactic sugar over `repeated MapEntry`. The C plan +treats a map field as a `repeated` field with a synthesized +sub-plan for the entry type (two fields: key, value). + +Encode walks the Lua table with `lua_next` (the only place we use +`lua_next` / iteration over hash). Each iteration: + +1. Encode `MapEntry{key=k, value=v}` into a sub-buffer. +2. Emit `tag + length-varint + sub_buffer` into the parent buffer. + +This matches the pure-Lua semantics and the wire format. **Map +iteration order is hash-determined** in both paths and is *not* +guaranteed stable across paths. The compat contract permits this. + +Implementation hook: `bd-3c`. + +## Oneofs + +The plan carries a parallel array of `pb_plan_oneof` structs. +Encode walks the oneof groups: for each, find which member is +set (Lua-side check via `getfield` for each member, until one is +non-nil), and encode only that member. + +For wide oneofs (many members) this is O(n) per oneof on encode. +A future optimization could cache the active member name on the +input table; for now the spike showed this isn't a bottleneck for +typical schemas. + +Decode is simpler: when a oneof-member field is decoded, the +codec clears any previously-set member of the same oneof from +the result table. + +Implementation hook: `bd-3i` (extensions and oneof grouping). + +## Unknown fields + +The C decoder, when it hits a tag whose field number is not in +the plan, copies the raw wire bytes for that field (tag + +payload) into a per-call `pb_buf` for unknown bytes. At the end +of decode, the unknown buffer is stored on the result table as +`_unknown_fields` (a Lua string). + +The C encoder, when it sees `t._unknown_fields` is non-nil, appends +those bytes verbatim to the output buffer after the known fields. +Position in the output is the same as pure-Lua (after, not +interleaved). + +Implementation hook: `bd-3j`. + +## What we will not optimize (yet) + +Two micro-optimizations that look attractive but the spike showed +don't pay: + +- **Inlining varint encode/decode into the dispatch loop.** Already + inlined by the C compiler at `-O2`. Manual inlining bloated + source without measurable speedup. +- **SIMD scan for varint terminators on decode.** Validated on + another protobuf C lib (upb), measurable on huge messages, but + the spike showed our bottleneck at 100 KB is the per-field Lua + bridge work, not varint scanning. Revisit when a workload + surfaces. + +## Build environment expectations + +- C99, `-O2`, `-Wall -Wextra`. +- Headers required: `module.h` and `lauxlib.h` (`module.h` does + *not* re-export `lauxlib.h`; both must be included). Verified + in spike. +- `module.h` resolution mirrors `bench/c_accel/Makefile`: env + override (`TARANTOOL_INCLUDE`), then brew prefix, then standard + paths. See [c_accel_build_packaging.md](c_accel_build_packaging.md) + (`bd-wky`) for the packaging side. + +## Testing posture + +Per the compat contract, no separate Lua-vs-C diff harness. The +existing `test/` suite runs twice in CI (PB_ENABLE_C unset vs +=1). The C path is exercised through the same generated +modules; bugs surface as test failures in the standard suite. + +When debugging a specific C path, two helpful environment +variables (deferred, not implemented yet — file when needed): + +- `PB_C_TRACE=1` — log each encode/decode call's descriptor name + and message size to stderr. For debugging "which call is + this?" during failures. +- `PB_C_CHECK_PARITY=1` — after every C encode/decode, run the + pure-Lua path and assert bytes/table equality. Slow; useful + when a conformance test fails and you want to localize. + +## References + +- [docs/c-accel.md](../c-accel.md) — architecture +- [docs/specs/c_accel_compat.md](c_accel_compat.md) — compat contract +- `bench/c_accel/spike_bench.lua` — the benchmark +- `bench/c_accel/person_codec.c` — reference S4 codec, the + patterns here are the patterns there +- `bench/c_accel/generic_codec.c` — reference S3 prototype, what + `ra6` generalizes from +- Memos: `c-accel-spike-04c-final-2026-05-18`, + `c-accel-spike-04c-phase-{a,b}-2026-05-18` diff --git a/runtime/pb/init.lua b/runtime/pb/init.lua index 14bb3b722751eab8919fac95455fc0de49c2b4f5..8bd5b70c40dc899f11ff7578b2306fcd62f91df9 100644 --- a/runtime/pb/init.lua +++ b/runtime/pb/init.lua @@ -19,6 +19,16 @@ local pbjson = require('pb.json') local pbtext = require('pb.text') local lazy = require('pb.lazy') +-- C-acceleration opt-in. Single switch is PB_ENABLE_C=1 in the environment, +-- evaluated once at module load. A failed require (no .so built, ABI +-- mismatch, missing module.h) falls back silently to the pure-Lua path +-- — see docs/specs/c_accel_compat.md § Activation for the contract. +local c_runtime +if os.getenv('PB_ENABLE_C') == '1' then + local ok, mod = pcall(require, 'pb.c_runtime') + if ok then c_runtime = mod end +end + return { -- High-level codec encode = codec.encode, @@ -54,6 +64,13 @@ return { -- gRPC transport interface + loopback — see runtime/pb/grpc.lua. grpc = grpc, + -- C-acceleration runtime, or nil when disabled. Non-nil only when + -- PB_ENABLE_C=1 is set at module load AND require('pb.c_runtime') + -- succeeded. Exposed for introspection; do not call directly from + -- user code — the encode/decode wrappers dispatch automatically + -- via desc.c_plan. See docs/specs/c_accel_compat.md. + c_runtime = c_runtime, + -- Runtime .proto parsing: build a module from a .proto source string. -- -- local hello = pb.parse(io.open('hello.proto'):read('*a'))