From d7b0364a82bd5beb8c2fb9d643a191a1aa42e5ab Mon Sep 17 00:00:00 2001 From: Eugene Blikh Date: Sat, 23 May 2026 23:04:54 +0300 Subject: [PATCH] beads: close ra6, 43t + drop deferred c0i dep from 43t MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ra6 (generic C runtime codec): all 16 sub-tasks (3a-3l) plus rc8 strict- decode parity bug closed. Umbrella issue closed as scope-complete. 43t (parity gate): closed after Justfile + bench.lua wiring. The c0i dependency was removed first — c0i is deferred indefinitely per the 04c spike conclusion, so keeping it as a blocker would prevent 43t from ever closing despite the parity work being complete. Also added the c-runtime-parity-gate-43t persistent memory recording the parity strategy and initial perf snapshot vs Lua-runtime. bd-ra6, bd-43t --- .beads/interactions.jsonl | 2 ++ .beads/issues.jsonl | 8 ++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.beads/interactions.jsonl b/.beads/interactions.jsonl index b79e52e6905fd7cb3b19a0e591eed4aae77bca52..5ce16e88fd46dc4e7997091efb1f46d0d23e04fc 100644 --- a/.beads/interactions.jsonl +++ b/.beads/interactions.jsonl @@ -31,3 +31,5 @@ {"id":"int-b39fa74a","kind":"field_change","created_at":"2026-05-23T18:22:01.107078Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-rmf","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} {"id":"int-8dbcb031","kind":"field_change","created_at":"2026-05-23T18:38:56.610443Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-m7u","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} {"id":"int-500abbb3","kind":"field_change","created_at":"2026-05-23T19:50:37.871661Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-rc8","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}} +{"id":"int-0be8d1bc","kind":"field_change","created_at":"2026-05-23T19:56:04.251964Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-ra6","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"All 16 sub-tasks (3a-3l) plus rc8 strict-decode parity bug closed. Generic C runtime codec landed: descriptor plan compiler, encode/decode scalars + repeated/packed + sub-messages + maps + oneofs + 64-bit cdata + WKT hooks + unknown fields + proto2 + dispatch + strict-decode parity. Ready for 43t parity gate."}} +{"id":"int-2a7d9cba","kind":"field_change","created_at":"2026-05-23T20:04:23.62044Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-43t","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Parity gate landed. Added test-c / test-all / bench-c Justfile recipes; bench.lua relabels runtime→c-runtime under PB_ENABLE_C=1 and refuses --baseline/--compare in that mode. Reuses existing 1043-test luatest matrix, Google conformance suite, and interop fixtures via the transitivity-through-reference strategy. c-generated column dropped from scope (c0i deferred per docs/c-accel.md); starwing column lives in bench/starwing_bench.lua separately."}} diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 44a217628d282235530f0a96485a09f99cfe8b0b..dbe804fef51d6e8d3b196b5784598dad7b59e131 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,4 +1,4 @@ -{"_type":"issue","id":"tarantool-protobuf-rc8","title":"C decoder: strict-decode parity with pure-Lua codec (wire-type 6/7, UTF-8, field-num, merge)","description":"After bd-43t wiring (init.lua dispatch + codegen lazy-compile prologue), the C codec is actually exercised by `PB_ENABLE_C=1 just test` and `just conformance-c`. 77 conformance failures and 13 luatest failures cluster into 7 categories that the pure-Lua decoder already rejects:\n\n1. UnknownWireType6 / UnknownWireType7 — C decoder accepts illegal wire types instead of erroring (Required.Proto[23].ProtobufInput.UnknownWireType{6,7}_Field{1,2,3}_Version{0..3}, 32 cases).\n2. IllegalZeroFieldNum_Case_{0,1,3} — field number 0 must be rejected.\n3. BadTag_FieldNumberSlightlyTooHigh / FieldNumberTooHigh — field numbers \u003e 2^29-1 must be rejected.\n4. BadTag_OverlongVarint — tag varints over the spec-mandated length must be rejected.\n5. RejectInvalidUtf8.String.{Singular,Repeated,Oneof,MapKey,MapValue} — invalid UTF-8 in string/map-key fields must be rejected.\n6. UnmatchedStartGroup / UnmatchedStartGroupNested (proto2) — group balancing.\n7. ValidDataOneof.MESSAGE.Merge / RepeatedScalarMessageMerge — oneof message-merge semantics and repeated-scalar merge accumulate vs replace.\n\nReference impl: runtime/pb/codec.lua + wire.lua (decode_tag rejects wt 6/7 since commit 7442ea2; utf8 validation via utf8.len; merge_message recurses; etc.). C decoder lives in runtime/pb/c/c_runtime.c.\n\nStatus: pb.encode/pb.decode lazy-dispatch to pb.c_runtime.encode/decode whenever PB_ENABLE_C=1 and a c_plan exists. Wrappers in mode=full prologue + runtime mode via pb.encode central dispatch. Dispatch is proven (in-container probe + 1026/1039 luatest pass after wiring vs 748 with C inert / skip).\n\nRepro: PB_ENABLE_C=1 just test → 13 fails in conformance.core.* group. just conformance-c → 77 unexpected failures in binary suite.\n\nBlocks: tarantool-protobuf-43t conformance gate.","notes":"Fixed in c_runtime.c. Strict-decode parity items implemented:\n- decode_body: reject wire types 6/7, field number 0, field number \u003e 2^29-1, overlong tag varint\n- dec_push_kind PB_KIND_STRING: RFC 3629 UTF-8 validator (port of utf8.len)\n- decode_body: track egroup_seen, error on loop exit if stop_group_id != 0\n- decode_body singular message dispatch + decode_extension_into singular: merge into prev table via merge_subresult_into, recurses for nested message, concat for repeated, last-wins per key for map, skip for WKT custom-decode\n- merge_subresult_into ports codec.lua's merge_message C-side\n\nRegression coverage (test/conformance_test.lua conformance.core group):\n- pre-existing tests covered field-number/wire-type/overlong/UTF-8 singular-repeated-oneof/merge — now exercise C path via PB_ENABLE_C=1\n- added test_invalid_utf8_map_key_rejected, test_invalid_utf8_map_value_rejected\n- added test_unmatched_start_group_rejected, test_unmatched_start_group_nested_rejected (proto2 via PROTO2_NAME)\n\nResults:\n- PB_ENABLE_C=1 just test: 1043/1043 pass (was 1039 baseline; +4 from new tests)\n- PB_ENABLE_C=1 just conformance-c: 2806 + 434 pass, 0 unexpected failures (was 77)\n- just conformance (Lua path): 2806 + 434 pass — no regression","status":"in_progress","priority":1,"issue_type":"bug","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-23T19:29:06Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T19:50:29Z","started_at":"2026-05-23T19:34:31Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-rc8","title":"C decoder: strict-decode parity with pure-Lua codec (wire-type 6/7, UTF-8, field-num, merge)","description":"After bd-43t wiring (init.lua dispatch + codegen lazy-compile prologue), the C codec is actually exercised by `PB_ENABLE_C=1 just test` and `just conformance-c`. 77 conformance failures and 13 luatest failures cluster into 7 categories that the pure-Lua decoder already rejects:\n\n1. UnknownWireType6 / UnknownWireType7 — C decoder accepts illegal wire types instead of erroring (Required.Proto[23].ProtobufInput.UnknownWireType{6,7}_Field{1,2,3}_Version{0..3}, 32 cases).\n2. IllegalZeroFieldNum_Case_{0,1,3} — field number 0 must be rejected.\n3. BadTag_FieldNumberSlightlyTooHigh / FieldNumberTooHigh — field numbers \u003e 2^29-1 must be rejected.\n4. BadTag_OverlongVarint — tag varints over the spec-mandated length must be rejected.\n5. RejectInvalidUtf8.String.{Singular,Repeated,Oneof,MapKey,MapValue} — invalid UTF-8 in string/map-key fields must be rejected.\n6. UnmatchedStartGroup / UnmatchedStartGroupNested (proto2) — group balancing.\n7. ValidDataOneof.MESSAGE.Merge / RepeatedScalarMessageMerge — oneof message-merge semantics and repeated-scalar merge accumulate vs replace.\n\nReference impl: runtime/pb/codec.lua + wire.lua (decode_tag rejects wt 6/7 since commit 7442ea2; utf8 validation via utf8.len; merge_message recurses; etc.). C decoder lives in runtime/pb/c/c_runtime.c.\n\nStatus: pb.encode/pb.decode lazy-dispatch to pb.c_runtime.encode/decode whenever PB_ENABLE_C=1 and a c_plan exists. Wrappers in mode=full prologue + runtime mode via pb.encode central dispatch. Dispatch is proven (in-container probe + 1026/1039 luatest pass after wiring vs 748 with C inert / skip).\n\nRepro: PB_ENABLE_C=1 just test → 13 fails in conformance.core.* group. just conformance-c → 77 unexpected failures in binary suite.\n\nBlocks: tarantool-protobuf-43t conformance gate.","notes":"Fixed in c_runtime.c. Strict-decode parity items implemented:\n- decode_body: reject wire types 6/7, field number 0, field number \u003e 2^29-1, overlong tag varint\n- dec_push_kind PB_KIND_STRING: RFC 3629 UTF-8 validator (port of utf8.len)\n- decode_body: track egroup_seen, error on loop exit if stop_group_id != 0\n- decode_body singular message dispatch + decode_extension_into singular: merge into prev table via merge_subresult_into, recurses for nested message, concat for repeated, last-wins per key for map, skip for WKT custom-decode\n- merge_subresult_into ports codec.lua's merge_message C-side\n\nRegression coverage (test/conformance_test.lua conformance.core group):\n- pre-existing tests covered field-number/wire-type/overlong/UTF-8 singular-repeated-oneof/merge — now exercise C path via PB_ENABLE_C=1\n- added test_invalid_utf8_map_key_rejected, test_invalid_utf8_map_value_rejected\n- added test_unmatched_start_group_rejected, test_unmatched_start_group_nested_rejected (proto2 via PROTO2_NAME)\n\nResults:\n- PB_ENABLE_C=1 just test: 1043/1043 pass (was 1039 baseline; +4 from new tests)\n- PB_ENABLE_C=1 just conformance-c: 2806 + 434 pass, 0 unexpected failures (was 77)\n- just conformance (Lua path): 2806 + 434 pass — no regression","status":"closed","priority":1,"issue_type":"bug","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-23T19:29:06Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T19:50:38Z","started_at":"2026-05-23T19:34:31Z","closed_at":"2026-05-23T19:50:38Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-exy","title":"ra6 3f: repeated strings + repeated messages (cached stack-idx)","description":"Repeated string fields and repeated message fields. Encode: iterate the Lua array via cached stack idx, write each element (string field or recursive submsg encode). Decode: first hit lazy-creates the result array and stashes its stack index in a per-field slot (sized by plan-\u003en_fields, cap 16 for the spike fixed-size); subsequent hits lua_rawseti directly without re-lookup; one lua_setfield at decode_message exit attaches the array to the result table. This is the SPIKE-VALIDATED PATTERN — naive lazy-getfield is 2x slower at 100KB. Depends on 3d (needs sub-message machinery for repeated messages). Acceptance: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) both round-trip byte-equal to mode=full at 1KB, 10KB, 100KB sizes.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T11:07:38Z","started_at":"2026-05-23T11:05:14Z","closed_at":"2026-05-23T11:07:38Z","close_reason":"Acceptance tests landed: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) round-trip byte-equal to mode=full at 1KB/10KB/100KB. Repeated string + message dispatch was already implemented as part of 3e (encode_repeated_field handles MESSAGE + string/bytes branches alongside scalars, decode_body uses cached list_stack_idx[] per-field); this ticket adds the formal size-scaled acceptance coverage.","dependencies":[{"issue_id":"tarantool-protobuf-exy","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:18Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-jc9","title":"ra6 3e: repeated/packed scalar encode/decode","description":"Repeated scalar fields: encode-time, walk the Lua array via cached stack idx; decode-time, lazy-create the array on first hit and append via lua_rawseti with a cached stack idx (NOT lazy lua_getfield per element — that's 2x slower at 100KB per the spike). Packed repeated: length-prefixed varint blob; tight loop in C. Covers int32/sint32/uint32/int64/uint64/fixed32/fixed64/double/bool/enum. Depends on 3b + 3c (need the scalar primitives). Acceptance: Person.lucky_numbers (packed int32) round-trips byte-equal; a fixture with both packed and unpacked repeated scalars round-trips against mode=full at 10/100/1000-element counts.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:33Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T10:39:55Z","started_at":"2026-05-23T10:29:02Z","closed_at":"2026-05-23T10:39:55Z","close_reason":"Closed","dependencies":[{"issue_id":"tarantool-protobuf-jc9","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:07Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-jc9","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:07Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-hwe","title":"ra6 3d: sub-message encode/decode (recursion)","description":"Nested-message support. Encode: open a sub-buffer, recurse with the submsg plan, prefix parent buffer with tag + length. Decode: bound the byte range, recurse, lua_setfield the resulting table. Sub-buffer stack-backed (4KB) and malloc-promoted, same pattern as the outer buffer. Recursion is by C function call into the same encode/decode routine with a different plan, so depth is bounded by C stack. Depends on 3b + 3c. Acceptance: Person.address (1-level) round-trips; deeper nesting via a 5-level test fixture round-trips byte-equal to mode=full.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-18T20:20:24Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T10:24:19Z","started_at":"2026-05-23T10:11:41Z","closed_at":"2026-05-23T10:24:19Z","close_reason":"ra6 3d: sub-message encode/decode (recursion) — refactored encode_lua/decode_lua into reusable encode_body/decode_body, added singular sub-message encode (force-promoted parent heap_idx before recursion to make lua_settop safe) and decode (temporarily shrunk c-\u003elen for bounded inner read). 22 new tests cover round-trip, empty sub-message presence, proto3-optional fields, 5-level depth (new test/proto/c_nested.proto), parent-buffer regrow path, truncated-input errors. Full suite 854/854 with PB_ENABLE_C=1.","dependencies":[{"issue_id":"tarantool-protobuf-hwe","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:06Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-hwe","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:05Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":4,"comment_count":0} @@ -10,7 +10,7 @@ {"_type":"issue","id":"tarantool-protobuf-80t","title":"C accel spike: per-primitive FFI variant (strategy 2)","description":"Sub-issue of tarantool-protobuf-04c. Implement strategy 2 of the C-acceleration benchmark: replace wire.lua's encode_varint / decode_varint / encode_string / decode_string / fixed32/64 / double with ffi.C.\u003cfn\u003e calls into a small C shim. The Lua dispatch loop stays; only the primitives cross. Measure with bench/c_accel/spike_bench.lua across the same 5 sizes. Acceptance: numbers added to bench/c_accel/README.md results table.","notes":"Phase A landed strategies 1 + 4. Strategy 4 (hand-written Person C codec) sets the upper bound at 3-10x over pure Lua. Strategy 2 must clear ~3x at every size to be competitive with 4; otherwise FFI per-primitive boundary cost is the wrong place to spend complexity. See bench/c_accel/README.md for full Phase A results.","status":"closed","priority":1,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T19:28:02Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T19:44:07Z","closed_at":"2026-05-18T19:44:07Z","close_reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18.","dependencies":[{"issue_id":"tarantool-protobuf-80t","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-18T22:28:15Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-04c","title":"C accel: benchmark spike for Lua C module vs FFI boundaries","description":"Build a small benchmark spike before committing to a full backend. Compare: pure Lua current path, per-primitive FFI calls, one Lua C module call per top-level encode/decode, and a hand-written C codec for hello.Person. Measure 10B/100B/1KB/10KB/100KB Person and proto2 BenchPayload. Acceptance: numbers show which boundary wins and quantify C-call overhead for tiny messages where current Lua decode already beats starwing.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T19:43:59Z","started_at":"2026-05-18T19:21:27Z","closed_at":"2026-05-18T19:43:59Z","close_reason":"Spike complete across all 4 boundaries (Lua / S2 FFI prims / S3 generic C / S4 hand C) at 10B-100KB. Bench: bench/c_accel/. Decisive results. S3 ≈ S4 within 15% (S3 often faster on encode at scale). S2 LOSES vs pure-Lua at every size 1KB+ on encode (0.60-0.99x) and at every size on decode (0.29-0.39x). FFI cost decomposition (bench/c_accel/ffi_probe.lua, memory luajit-ffi-boundary-cost-2026-05-18): bare FFI = 33ns; pointer-return FFI = 73ns; ffi.cast on string = 156ns; libc memcmp = 60ns; pure-Lua varint decode = 75ns. Per-primitive boundary cost equals pure-Lua dispatch cost — only message-level C crossing wins. ARCHITECTURE for pf6: ship ra6 (generic C runtime, descriptor-walking, one C call per message); drop c0i (≤15% headroom over ra6); drop per-primitive FFI. Required ra6 impl pattern: cache per-field stack indices for repeated/packed arrays during decode_message (naive lazy-getfield 2x slower at 100KB). Full details in bench/c_accel/README.md and bd memory c-accel-spike-04c-final-2026-05-18 + luajit-ffi-boundary-cost-2026-05-18.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-pf6","title":"C accel: choose architecture for C runtime and generated codecs","description":"Umbrella design task for C acceleration. Decide and document the architecture for (1) a generic C runtime codec used by pb.encode/pb.decode and dynamic schemas, and (2) generated C codecs emitted by protoc-gen-tarantool. Key decision: prefer Lua C modules / generated .so entrypoints over per-field LuaJIT FFI calls, because the measured gap is per-field Lua dispatch and table/string work; crossing into C once per message lets C own the parse/write loop. Deliverable: design note, benchmark target, API/backward-compat story, and migration path from current pure-Lua full/runtime modes.","status":"closed","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:23:01Z","closed_at":"2026-05-18T20:23:01Z","close_reason":"Architecture chosen and documented in docs/c-accel.md (commit pending). Ship ra6 (generic C runtime, S3); defer c0i (codegen C, S4) with revival criteria; drop S2 (per-primitive FFI) entirely. PB_ENABLE_C=1 opt-in, default off. Parity via existing-suite re-runs with C enabled. Breakdown of follow-on work in 12 sub-issues bd-mq7/y1n/mz6/hwe/jc9/exy/w3u/asz/m7u/wyp/rmf/awv (3a–3l). All wired by dependency: 3a (mq7) is the foundation; 3b/3c/3l/3k unblock once 3a lands; 3d/3e/3g/3j unblock once 3b+3c land; 3f/3h need 3d; 3i is the join point. ra6 itself depends on all twelve. c0i blocked-by ra6.","dependencies":[{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-04c","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-1eq","type":"blocks","created_at":"2026-05-18T22:28:17Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-pf6","depends_on_id":"tarantool-protobuf-80t","type":"blocks","created_at":"2026-05-18T22:28:16Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":4,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-ra6","title":"C accel: generic C runtime codec with compiled descriptor plans","description":"Implement a pb.c runtime module that compiles finalized Lua descriptors into C-side plan userdata: field ids, wire types, cached Lua field-name refs, oneof metadata, extension ranges, map entry metadata, and WKT hooks. pb.encode/pb.decode can dispatch to the C plan when available and fall back to Lua otherwise. This targets runtime mode, dynamic schemas, and a shared substrate for generated C. Avoid reading descriptor Lua tables in the hot loop; compile once.","status":"open","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:32:31Z","dependencies":[{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-asz","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-awv","type":"blocks","created_at":"2026-05-18T23:22:39Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-exy","type":"blocks","created_at":"2026-05-18T23:22:35Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-jc9","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-m7u","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:22:31Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:33Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-rmf","type":"blocks","created_at":"2026-05-18T23:22:38Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-w3u","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-wyp","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:32Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":15,"dependent_count":2,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-ra6","title":"C accel: generic C runtime codec with compiled descriptor plans","description":"Implement a pb.c runtime module that compiles finalized Lua descriptors into C-side plan userdata: field ids, wire types, cached Lua field-name refs, oneof metadata, extension ranges, map entry metadata, and WKT hooks. pb.encode/pb.decode can dispatch to the C plan when available and fall back to Lua otherwise. This targets runtime mode, dynamic schemas, and a shared substrate for generated C. Avoid reading descriptor Lua tables in the hot loop; compile once.","status":"closed","priority":1,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:31Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T19:56:04Z","closed_at":"2026-05-23T19:56:04Z","close_reason":"All 16 sub-tasks (3a-3l) plus rc8 strict-decode parity bug closed. Generic C runtime codec landed: descriptor plan compiler, encode/decode scalars + repeated/packed + sub-messages + maps + oneofs + 64-bit cdata + WKT hooks + unknown fields + proto2 + dispatch + strict-decode parity. Ready for 43t parity gate.","dependencies":[{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-asz","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-awv","type":"blocks","created_at":"2026-05-18T23:22:39Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-exy","type":"blocks","created_at":"2026-05-18T23:22:35Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-hwe","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-jc9","type":"blocks","created_at":"2026-05-18T23:22:34Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-m7u","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mq7","type":"blocks","created_at":"2026-05-18T23:22:31Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-mz6","type":"blocks","created_at":"2026-05-18T23:22:33Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-rmf","type":"blocks","created_at":"2026-05-18T23:22:38Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-w3u","type":"blocks","created_at":"2026-05-18T23:22:36Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-wyp","type":"blocks","created_at":"2026-05-18T23:22:37Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-y1n","type":"blocks","created_at":"2026-05-18T23:22:32Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-ra6","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:28Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":15,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-4kj","title":"Decoder: generated tag/length fast path for full-mode decode","description":"Inspection against starwing/lua-protobuf shows the 1KB+ Person decode gap is mostly repeated Lua-level tag/field dispatch, not a fatal JIT abort. On the 930B Person fixture, generated full decode is ~6.9-7.5 us/op, tag+length scan is ~2.1 us/op, skip_field scan is ~3.6 us/op, and an order-specialized decoder is ~4.3 us/op. Implement a full-mode generated fast path that decodes expected 1-byte tags and LEN prefixes inline at the call site, falling back to the generic decode_tag/skip path for unknown/out-of-order/multi-byte cases. Keep correctness for arbitrary field order, repeated occurrences, unknown fields, and proto2 semantics.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:02:41Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:56:20Z","started_at":"2026-05-18T18:44:48Z","closed_at":"2026-05-18T18:56:20Z","close_reason":"Inlined wire.decode_tag's 1-byte fast path at every M.X_decode while-loop site, plus header localization of string.byte/bit.band/bit.rshift. Median-of-3 Person full decode: +7-15% across all sizes. Full encode flat to -2.6% (likely header upvalue layout). 745/745 tests, 37/37 JIT, 0 bridges. See bench/PERF_LOG.md entry. Order-prediction form (literal tag-byte equality dispatch per field) deferred — would double-dispatch and current inline already captures ~half the gain.","dependencies":[{"issue_id":"tarantool-protobuf-4kj","depends_on_id":"tarantool-protobuf-0an","type":"blocks","created_at":"2026-05-17T19:02:53Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-gcy","title":"Decoder: inline nested-message decode at the call site","description":"Today Person_decode calls Address_decode(slice) across a function boundary; the side trace into Address_decode may not stitch back (memory: luajit_side_trace_inlined_return). For field-typed messages in mode=full, emit the decode body inline at the call site. Expected: 20-40% on payloads with nested messages. Codegen growth is bounded by recursion depth — pick a depth limit and call out beyond it.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:09Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:44:22Z","started_at":"2026-05-18T18:32:15Z","closed_at":"2026-05-18T18:44:22Z","close_reason":"Attempted but reverted. Inlining Address_decode body into Person_decode (singular non-recursive candidate) showed 745/745 tests passing and 37/37 JIT, but bench median-of-3 regressed Person 1KB/10KB/100KB decode by 5-8% and encode by 3-6%. Profile's '100% interpreter bail' claim turned out to be a vl trace-attribution artifact; LuaJIT was already inlining Address_decode into Person_decode's trace effectively. Larger root traces (22 stops) compiled less efficiently. See bench/PERF_LOG.md entry for full data and analysis. Real decode bottlenecks remain decode_string utf8 (6bb), decode_tag fast path (4kj), repeated append (cch).","labels":["codegen","decoder","perf"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-h8v","title":"Encoder: codegen-time inline FFI writes (mode=full)","description":"Replace per-field wire.encode_* calls + ../table.concat chain with directly-emitted FFI byte writes at every field site in mode=full. Today every encode_len(s) is 'encode_varint(#s) .. s' — two allocations and a concat per string. Sidesteps the per-byte b:alloc(1) cliff that sank the earlier ibuf attempt (memory: tarantool_ibuf_perf). Expected: 1.5-2x encode throughput; alloc/op drops from ~130 KB at 100 KB Person to near zero. Biggest single perf move. Bench reference: bench/starwing_bench.lua + bench/COMPARISON.md show starwing C encoder at 2.3-3x ours across all sizes.","status":"closed","priority":1,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T15:47:04Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T18:31:54Z","started_at":"2026-05-18T18:25:39Z","closed_at":"2026-05-18T18:31:54Z","close_reason":"Inlined 1-byte varint length prefix at every LEN emit site; +25-48% encode at 1KB+ Person. Full FFI-buffer rewrite deferred — this slice captured the dominant profile target without that level of disruption. See bench/PERF_LOG.md entry for details.","labels":["codegen","encoder","perf"],"dependency_count":0,"dependent_count":2,"comment_count":0} @@ -32,7 +32,7 @@ {"_type":"issue","id":"tarantool-protobuf-lkz","title":"Encoder: single-pass two-phase (size + emit) into single buffer","description":"Eliminate the per-call out table and the per-field intermediate string allocations by emitting once into a pre-sized buffer. Two viable shapes (per drm notes):\n\n1. Two-pass: walk fields once to compute exact size, allocate one buffer, walk again to emit. Mirrors vtproto's Size()+MarshalTo() pattern — vtproto's 1-alloc encode is what gives it 13-20x throughput over us at 1KB-10KB Person.\n2. Single-pass with backpatched length varints: write tag + 1-byte placeholder for the length, recurse, fill in (memmove if final length \u003e= 128). Skips the size pass (~3 us at 1KB).\n\nImplementation can live in codegen (mode=full) — emit a _encode_to_buf per message that takes (data, ibuf, offset) and returns new_offset, then a public Person_encode that wraps it with the buffer allocation. Constraint: per-field closure count must stay \u003c= current pb.encode writers count, else the per-field dispatch cost re-emerges (drm: two prior ibuf prototypes both regressed by ~2x because of dispatch).\n\nRelates to h8v (codegen FFI direct writes) which is the per-field building block this work composes.","notes":"drm notes have the deeper analysis including why M6 ibuf path naive byte-write fails (20x interpreter dispatch overhead). The codegen approach sidesteps that by inlining all writes at codegen time so there's no runtime closure dispatch.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T16:49:16Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T16:49:16Z","dependencies":[{"issue_id":"tarantool-protobuf-lkz","depends_on_id":"tarantool-protobuf-h8v","type":"blocks","created_at":"2026-05-18T19:49:21Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-2sn","title":"Decoder: emit table.new(N, 0) for repeated-field lists","description":"Decode result table rehashes as nested fields populate. u39 covers table.new(0, N) for the message table itself, but per-field list tables (result.emails = {}, result.lucky_numbers = {}, ...) are also bare {} and re-hash as they grow. Worth a follow-up: emit table.new(N, 0) for repeated lists where the encoded count is recoverable from a single quick scan (count tag occurrences for non-packed, or read the LEN prefix and divide by per-element size for packed). Quick scan cost vs allocation savings is the tradeoff to measure — on the 1KB Person fixture, 26 emails + 5 packed lucky_numbers means 2 re-hash cycles per repeated list. Pairs with u39 to fully close the rehash-on-grow alloc pattern.","notes":"See bench/COMPARISON.md and bd show drm. For packed scalars where per-element size is fixed (fixed32/fixed64/float/double), count = payload_len / elem_size — O(1). For varint-packed and non-packed repeated, count requires a scan; might still be net-positive on payloads with \u003e8 elements but needs measurement.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T16:49:15Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T16:49:15Z","dependencies":[{"issue_id":"tarantool-protobuf-2sn","depends_on_id":"tarantool-protobuf-u39","type":"blocks","created_at":"2026-05-18T19:49:20Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-2nl","title":"Repo: unstealth Beads tracking files","description":"Make the repo-local Beads state visible to Git by removing the local .git/info/exclude rule for .beads/. Keep .beads/.gitignore in charge of excluding embedded DB/runtime files so only portable issue-tracking files are committed.","status":"closed","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T18:39:50Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T18:42:26Z","closed_at":"2026-05-17T18:42:26Z","close_reason":"Removed the local .git/info/exclude rule that hid .beads/ from Git. Portable .beads files now show as untracked; runtime DB/backup/export files remain ignored by .beads/.gitignore.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-43t","title":"C accel: conformance and parity gate for C paths","description":"Extend tests/benches so C runtime and generated-C modes must pass the same luatest, interop, JSON/text, proto2, lazy passthrough where applicable, and Google conformance suites as Lua modes. Add bench output columns for lua-full, lua-runtime, c-runtime, c-generated, starwing. C acceleration must remain byte-equal with current encoders and preserve unknown fields/extensions/groups.","notes":"Architecture finalized in docs/c-accel.md. Parity strategy: REUSE existing test suites — no separate Lua-vs-C diff harness. (1) just test (luatest, 639 tests already parameterized over full/runtime modes) runs once with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Both must pass. (2) just conformance (Google proto3 + proto2) runs both modes. (3) Interop fixtures (test/interop/fixtures/*.bin) cover both modes via the existing parity.full_vs_runtime tests; extend the matrix to also cover c-runtime. Logic: every assert is against a reference output (golden bytes, txtpb, conformance result). If Lua passes and C passes, both equal the reference, so Lua == C by transitivity. No new test infrastructure required. bench/bench.lua gains a c-runtime column.","status":"open","priority":2,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:36Z","dependencies":[{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-c0i","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-wky","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":3,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-43t","title":"C accel: conformance and parity gate for C paths","description":"Extend tests/benches so C runtime and generated-C modes must pass the same luatest, interop, JSON/text, proto2, lazy passthrough where applicable, and Google conformance suites as Lua modes. Add bench output columns for lua-full, lua-runtime, c-runtime, c-generated, starwing. C acceleration must remain byte-equal with current encoders and preserve unknown fields/extensions/groups.","notes":"Architecture finalized in docs/c-accel.md. Parity strategy: REUSE existing test suites — no separate Lua-vs-C diff harness. (1) just test (luatest, 639 tests already parameterized over full/runtime modes) runs once with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Both must pass. (2) just conformance (Google proto3 + proto2) runs both modes. (3) Interop fixtures (test/interop/fixtures/*.bin) cover both modes via the existing parity.full_vs_runtime tests; extend the matrix to also cover c-runtime. Logic: every assert is against a reference output (golden bytes, txtpb, conformance result). If Lua passes and C passes, both equal the reference, so Lua == C by transitivity. No new test infrastructure required. bench/bench.lua gains a c-runtime column.","status":"in_progress","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-23T19:56:10Z","started_at":"2026-05-23T19:56:10Z","dependencies":[{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-43t","depends_on_id":"tarantool-protobuf-wky","type":"blocks","created_at":"2026-05-17T19:33:44Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":2,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-z7x","title":"C accel: C table and string handling strategy","description":"Design and implement the low-level C helpers for Lua table access and string/buffer handling. Cache field-name strings/registry refs in descriptor plans, pre-size result tables where possible, build output strings with luaL_Buffer or exact-size allocation, avoid per-field Lua stack churn, and handle 64-bit cdata consistently with the current Lua API.","notes":"Architecture finalized in docs/c-accel.md. Strategy decisions validated by bench/c_accel/ spike: (1) Field-name strings cached as luaL_ref slots at plan-compile time; each decode does lua_rawgeti from the cached ref instead of re-interning. (2) Output buffer: 4KB stack-backed cap with malloc promotion on overflow (pattern in bench/c_accel/person_codec.c — validates against tiny-msg overhead and big-msg correctness). (3) Result tables pre-sized via lua_createtable(0, n_fields) from descriptor stats. (4) Per-field stack indices cached for repeated/packed array fields for the duration of decode_message — naive lazy-getfield was 2x slower at 100KB (spike Phase B). (5) 64-bit cdata via luaT_pushuint64/luaT_pushint64 (Tarantool extensions in module.h). (6) Sub-messages use separate buf with stack-allocated header; backpatching avoided (the bench showed sub-buf approach is fast enough).","status":"closed","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:53Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:24:12Z","started_at":"2026-05-19T04:10:45Z","closed_at":"2026-05-19T04:24:12Z","close_reason":"C-side strategy formalized in docs/specs/c_accel_strategy.md. Spec covers: pb_plan / pb_plan_field struct layout, field-name luaL_ref caching (spike-validated), 4 KB stack-backed pb_buf with malloc promotion, lua_createtable(0, n_fields) result pre-sizing, cached per-field stack indices for repeated/packed (spike Phase B: 2x speedup at 100 KB), 64-bit cdata via luaT_pushint64/checkint64, sub-buffer (chosen) vs backpatching (deferred) for sub-messages, two-pass packed encode, map iteration via lua_next, oneof grouping, unknown field passthrough, deferred micro-opts (manual varint inlining, SIMD scan), C99/-O2 build environment, deferred PB_C_TRACE/PB_C_CHECK_PARITY env knobs. ra6/mq7 references this spec for low-level decisions.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-47e","title":"C accel: preserve pure-Lua fallback and public API compatibility","description":"Define compatibility boundaries for C acceleration. Existing generated Lua modules and runtime pb APIs must keep working without a compiler or C module. New generated-C mode should be opt-in initially. Errors, 64-bit cdata behavior, WKT shapes, unknown fields, extension representation, lazy decode API, and text/JSON integration must remain compatible.","notes":"Architecture finalized in docs/c-accel.md. Concrete contract: (1) PB_ENABLE_C=1 env var is the only activation switch, default off. (2) require('pb') returns the same Lua surface in both modes — no API change. (3) 64-bit ints stay LuaJIT cdata (int64_t/uint64_t) in both modes. (4) WKT shapes unchanged. (5) Unknown fields round-trip identically. (6) Errors land as the same Lua error types. (7) Pure-Lua install must work without a C compiler — rockspec opt-builds the C module, install never fails on a host without cc. (8) When C module fails to load or PB_ENABLE_C is unset, runtime/pb/init.lua transparently uses the pure-Lua path.","status":"closed","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:23:56Z","started_at":"2026-05-19T04:10:44Z","closed_at":"2026-05-19T04:23:56Z","close_reason":"Compatibility contract formalized in docs/specs/c_accel_compat.md. Spec covers: PB_ENABLE_C=1 single activation switch, silent fallback, full public surface preservation (pb.encode/decode/decode_lazy/parse/from_pb/json/text/grpc), generated module wrappers, 64-bit cdata invariant, WKT shapes, unknown fields, extensions, error compatibility, install paths (pure-Lua and C-enabled), ABI versioning, conformance via existing-suite re-run, explicit out-of-scope list. Lazy view stays Lua-only per spike rationale. Downstream tasks (mq7, ra6, 43t, c0i) reference this spec for compat boundaries.","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-wky","title":"C accel: build and packaging support for C modules","description":"Add build-system and packaging support for C acceleration. Needs Justfile targets, rockspec support for compiled modules, platform naming, local dev build, CI matrix integration, and fallback when the C module is absent. Generated C backend must not make pure-Lua install impossible unless explicitly selected.","notes":"Architecture finalized in docs/c-accel.md. Concrete deliverables: (1) Justfile recipe 'just build-c' that builds the C runtime module (mirror bench/c_accel/Makefile auto-detection of TT_INC). (2) Rockspec optionally builds the C module — install on host without cc must succeed and produce a pure-Lua install. (3) runtime/pb/init.lua does pcall(require, 'pb.c_runtime') only when os.getenv('PB_ENABLE_C') == '1'. (4) Sourcehut CI build manifest gains one extra job that runs the full test+conformance suite with PB_ENABLE_C=1 set — single .build.yml, both modes covered on every push to master. (5) CI also runs one variant without the C module compiled to confirm pure-Lua install works.","status":"closed","priority":2,"issue_type":"task","assignee":"Eugene Blikh","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:52Z","created_by":"Eugene Blikh","updated_at":"2026-05-19T04:24:13Z","started_at":"2026-05-19T04:10:45Z","closed_at":"2026-05-19T04:24:13Z","close_reason":"Build and packaging scaffolding landed: docs/specs/c_accel_build_packaging.md spec, runtime/pb/init.lua PB_ENABLE_C=1 opt-in pcall hook (exposed as pb.c_runtime field), Justfile build-c/clean-c recipes (stub erroring cleanly until bd-ra6 lands runtime/pb/c/), Justfile lua_cpath constant + LUA_CPATH on test/test-one, .gitignore for compiled .so/.dylib, sourcehut .builds/{pure-lua,c-enabled}.yml manifests (ubuntu/noble, parallel jobs via separate files since sourcehut has no matrix), .sourcehut/conformance.yml for manual conformance triggers. Rockspec changes deferred to first C-source landing (described in spec). All 745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1 (silent fallback verified).","dependencies":[{"issue_id":"tarantool-protobuf-wky","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:11Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} @@ -55,7 +55,7 @@ {"_type":"issue","id":"tarantool-protobuf-gi0","title":"Codegen: emit wire.encode_varint directly for int32/int64/uint32/uint64 instead of typed alias","description":"runtime/pb/wire.lua aliases M.encode_int32 = encode_varint (and same for int64/uint32/uint64). Generated code calls wire.encode_int32(v) which resolves to encode_varint through two table lookups — wire.encode_int32 (one hash lookup), then the alias resolution. LuaJIT may collapse this when the trace stays hot, but each break re-incurs both.\n\nCodegen can emit wire.encode_varint directly for the four unsigned-varint scalar types (int32/int64/uint32/uint64) since they're literally encode_varint with different names. Skips one alias indirection per varint encode. Pairs naturally with kot (localize wire.* upvalues) — together they reduce the call to a direct LJ_FUNCC dispatch with no name lookup.\n\nSame applies to bool (encode_bool = encode_varint with v and 1 or 0 wrapper) and sint32/sint64 (zigzag wrapper) — codegen could inline the wrapper logic at the call site for sint, but that overlaps with h8v (FFI direct writes) which subsumes the question.","notes":"Codegen-side change in protoc-gen-tarantool. Constraint: the alias provides the typed encoder slot in TYPE_INFO that runtime-mode encoders walk — those aliases must stay. Only the generated mode=full code changes.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:38Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:38Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-5y9","title":"Decoder: int64/uint64 return Lua number when value fits in 2^53","description":"decode_int64/decode_uint64 always return ffi cdata, forcing a fresh cdata allocation per call. For values in [-2^53, 2^53) (the common case for IDs, sequence numbers, timestamps fitting in 53 bits, byte counts, etc.) a Lua number is exactly representable and skips the cdata header allocation.\n\nProposed shape: opt-in variant decoder, since changing the default breaks any caller that does type(v)=='cdata' or relies on cdata-only operators. Two API options:\n\n 1. Per-descriptor flag (desc.int64_as_number = true) wired in via a codegen option or generator flag. Generated code emits a different decoder fn.\n 2. Separate typed decoders (wire.decode_int64_n / decode_uint64_n) that callers opt into explicitly.\n\nWatch case: values exceeding 2^53 must still return cdata (with a runtime branch). The branch cost only pays off if cdata allocation cost \u003e one comparison, which it is on hot paths.","notes":"Bench impact bounded by how many int64 fields the workload has. For hello.Person, user_id (fixed64) is the only one — so impact on this fixture would be ~1-2%. Bigger win on protobuf workloads dominated by timestamps and sequence numbers (datastore RPCs, log streams).","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-18T17:14:37Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T17:14:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-8k2","title":"Docs: remove PLAN.md after Beads migration","description":"PLAN.md has been converted from active roadmap to duplicate design-history prose now that all actionable work is tracked in Beads. Remove the file or replace remaining references with Beads/README pointers so project state has a single durable task source of truth.","status":"closed","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T18:33:09Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T18:33:48Z","closed_at":"2026-05-17T18:33:48Z","close_reason":"Removed PLAN.md and retargeted remaining references to Beads or concrete docs.","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"tarantool-protobuf-c0i","title":"C accel: generated C codec backend for mode=full","description":"Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -\u003e string and M.Type_decode(bytes) -\u003e table. Goal: one Lua-\u003eC call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.","notes":"DEFERRED per docs/c-accel.md. Spike (04c) showed S4 hand-written C had ≤15% headroom over S3 (generic C runtime, ra6) and went the wrong way at scale (S4 *slower* than S3 at 10KB and 100KB encode). Codegen complexity not justified by current numbers. REVIVAL CRITERIA: a measured real-workload shape where ra6's per-field dispatch costs ≥25% over hand-written for that shape, demonstrated with a microbenchmark, AND the affected workload is on a hot path for a real user. Likely trigger shapes: wide messages with many optionals, heavy oneof use, complex maps, deeply nested (5+ levels) hierarchies. When triggered: write 2-3 paragraphs documenting shape and numbers, re-open this issue, scope narrowly via desc.encode/desc.decode override (same mechanism as WKT). If 6 months after ra6 ships no trigger fires, close as 'not justified'. Now blocked by ra6 (so it surfaces in ready list only after ra6 implementation lands).","status":"open","priority":3,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:32Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:41Z","dependencies":[{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-18T23:19:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":4,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"tarantool-protobuf-c0i","title":"C accel: generated C codec backend for mode=full","description":"Extend protoc-gen-tarantool with an optional generated-C backend. Emit C source plus Lua wrappers for each .proto package; generated functions know tag bytes, field names, defaults, oneofs, maps, proto2 required/defaults/groups/extensions, and nested message calls. The Lua API remains M.Type_encode(t) -\u003e string and M.Type_decode(bytes) -\u003e table. Goal: one Lua-\u003eC call per top-level message, no Lua table.concat, no Lua decode_tag ladder, no per-field FFI boundary.","notes":"DEFERRED per docs/c-accel.md. Spike (04c) showed S4 hand-written C had ≤15% headroom over S3 (generic C runtime, ra6) and went the wrong way at scale (S4 *slower* than S3 at 10KB and 100KB encode). Codegen complexity not justified by current numbers. REVIVAL CRITERIA: a measured real-workload shape where ra6's per-field dispatch costs ≥25% over hand-written for that shape, demonstrated with a microbenchmark, AND the affected workload is on a hot path for a real user. Likely trigger shapes: wide messages with many optionals, heavy oneof use, complex maps, deeply nested (5+ levels) hierarchies. When triggered: write 2-3 paragraphs documenting shape and numbers, re-open this issue, scope narrowly via desc.encode/desc.decode override (same mechanism as WKT). If 6 months after ra6 ships no trigger fires, close as 'not justified'. Now blocked by ra6 (so it surfaces in ready list only after ra6 implementation lands).","status":"open","priority":3,"issue_type":"feature","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:32:32Z","created_by":"Eugene Blikh","updated_at":"2026-05-18T20:19:41Z","dependencies":[{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-47e","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-pf6","type":"blocks","created_at":"2026-05-17T19:33:12Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-ra6","type":"blocks","created_at":"2026-05-18T23:19:44Z","created_by":"Eugene Blikh","metadata":"{}"},{"issue_id":"tarantool-protobuf-c0i","depends_on_id":"tarantool-protobuf-z7x","type":"blocks","created_at":"2026-05-17T19:33:29Z","created_by":"Eugene Blikh","metadata":"{}"}],"dependency_count":4,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-bnj","title":"Tooling: add formatting targets for Go and Lua","description":"PLAN.md section 6 lists gofumpt and stylua. Add formatter configuration and Justfile targets, with generated code excluded unless the generator itself is intended to emit stylua-compliant output.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:28Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-dnb","title":"Tooling: add Go and Lua lint targets","description":"PLAN.md section 6 lists golangci-lint for Go and luacheck for Lua. Add configured lint targets and decide whether they run locally only or in CI. Keep generated files excluded where appropriate.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:28Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"tarantool-protobuf-6rb","title":"Tooling: add coverage targets for plugin and runtime","description":"PLAN.md section 6 lists go test -cover for the Go plugins and luacov for the Lua runtime. Add repeatable coverage targets and document expected use. Decide whether coverage is informational or gated in CI.","status":"open","priority":3,"issue_type":"task","owner":"bigbes@gmail.com","created_at":"2026-05-17T16:28:27Z","created_by":"Eugene Blikh","updated_at":"2026-05-17T16:28:27Z","dependency_count":0,"dependent_count":0,"comment_count":0}