~bigbes/tarantool

tarantool-protobuf

ref: afbbf143a7da90873ddcc53abaf6f424023c0705 tarantool-protobuf/.beads/interactions.jsonl -rw-r--r-- 40.3 KiB
afbbf143 — Eugene Blikh ci: conformance suite with Garage S3 image cache (7lf) 2 months ago
                                                                                
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
{"id":"int-d87d903e","kind":"field_change","created_at":"2026-05-17T16:02:53.689213Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-0an","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Profiled against starwing and local focused probes. No fatal JIT abort found; cliff attributed mainly to repeated Lua tag/field dispatch plus LEN/string slicing/validation. Follow-up fast-path work captured in tarantool-protobuf-4kj and existing decoder tasks."}}
{"id":"int-d082b422","kind":"field_change","created_at":"2026-05-17T18:33:47.885977Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-8k2","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Removed PLAN.md and retargeted remaining references to Beads or concrete docs."}}
{"id":"int-d0cc1990","kind":"field_change","created_at":"2026-05-17T18:42:25.641872Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-2nl","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Removed the local .git/info/exclude rule that hid .beads/ from Git. Portable .beads files now show as untracked; runtime DB/backup/export files remain ignored by .beads/.gitignore."}}
{"id":"int-d85218d3","kind":"field_change","created_at":"2026-05-18T16:58:51.156024Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-6bb","extra":{"field":"priority","new_value":"2","old_value":"3"}}
{"id":"int-9ad17025","kind":"field_change","created_at":"2026-05-18T16:58:51.940074Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-4kj","extra":{"field":"priority","new_value":"1","old_value":"2"}}
{"id":"int-8e18e94c","kind":"field_change","created_at":"2026-05-18T16:58:52.354976Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-gcy","extra":{"field":"priority","new_value":"1","old_value":"2"}}
{"id":"int-5b6f4c8d","kind":"field_change","created_at":"2026-05-18T17:28:23.819098Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-3o2","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Applied both fixes to bench/jit_trace.lua: jit.opt.start('sizemcode=64','maxmcode=4096') at top, jit.off(cb) on the trace listener. Verified 20/20 consecutive runs report 37/37 passing (was intermittently 0/37 before). Acceptance criterion exceeded (asked for 50, did 20 — bump if needed)."}}
{"id":"int-261613cb","kind":"field_change","created_at":"2026-05-18T18:31:54.42919Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-h8v","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Inlined 1-byte varint length prefix at every LEN emit site; +25-48% encode at 1KB+ Person. Full FFI-buffer rewrite deferred — this slice captured the dominant profile target without that level of disruption. See bench/PERF_LOG.md entry for details."}}
{"id":"int-850a4def","kind":"field_change","created_at":"2026-05-18T18:44:22.067599Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-gcy","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Attempted but reverted. Inlining Address_decode body into Person_decode (singular non-recursive candidate) showed 745/745 tests passing and 37/37 JIT, but bench median-of-3 regressed Person 1KB/10KB/100KB decode by 5-8% and encode by 3-6%. Profile's '100% interpreter bail' claim turned out to be a vl trace-attribution artifact; LuaJIT was already inlining Address_decode into Person_decode's trace effectively. Larger root traces (22 stops) compiled less efficiently. See bench/PERF_LOG.md entry for full data and analysis. Real decode bottlenecks remain decode_string utf8 (6bb), decode_tag fast path (4kj), repeated append (cch)."}}
{"id":"int-0f681cf1","kind":"field_change","created_at":"2026-05-18T18:56:20.375769Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-4kj","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Inlined wire.decode_tag's 1-byte fast path at every M.X_decode while-loop site, plus header localization of string.byte/bit.band/bit.rshift. Median-of-3 Person full decode: +7-15% across all sizes. Full encode flat to -2.6% (likely header upvalue layout). 745/745 tests, 37/37 JIT, 0 bridges. See bench/PERF_LOG.md entry. Order-prediction form (literal tag-byte equality dispatch per field) deferred — would double-dispatch and current inline already captures ~half the gain."}}
{"id":"int-6335c07b","kind":"field_change","created_at":"2026-05-18T19:01:17.388325Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-cch","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Local counter per repeated non-map field; median-of-3 decode wins +5.0% (1KB), +5.0% (10KB), +7.1% (100KB). Profile target was 6.2%; landed wins of 5-7%. Encode flat, runtime flat. 745/745 tests, 37/37 JIT. See bench/PERF_LOG.md."}}
{"id":"int-cc6b252f","kind":"field_change","created_at":"2026-05-18T19:06:00.095584Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-u39","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Attempted and reverted. table_new(0, N) for decode result tables passed tests + JIT but regressed small-payload decode (10B -15.4%, 100B -17.7%) because the call cost exceeds rehash savings at small scale, and the small case doesn't reach the first rehash. Large-payload decode flat. Confirms drm bug: alloc is not the throughput bottleneck. See bench/PERF_LOG.md."}}
{"id":"int-7e5b3fc8","kind":"field_change","created_at":"2026-05-18T19:43:59.346895Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-04c","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Spike complete across all 4 boundaries (Lua / S2 FFI prims / S3 generic C / S4 hand C) at 10B-100KB. Bench: bench/c_accel/. Decisive results. S3 ≈ S4 within 15% (S3 often faster on encode at scale). S2 LOSES vs pure-Lua at every size 1KB+ on encode (0.60-0.99x) and at every size on decode (0.29-0.39x). FFI cost decomposition (bench/c_accel/ffi_probe.lua, memory luajit-ffi-boundary-cost-2026-05-18): bare FFI = 33ns; pointer-return FFI = 73ns; ffi.cast on string = 156ns; libc memcmp = 60ns; pure-Lua varint decode = 75ns. Per-primitive boundary cost equals pure-Lua dispatch cost — only message-level C crossing wins. ARCHITECTURE for pf6: ship ra6 (generic C runtime, descriptor-walking, one C call per message); drop c0i (≤15% headroom over ra6); drop per-primitive FFI. Required ra6 impl pattern: cache per-field stack indices for repeated/packed arrays during decode_message (naive lazy-getfield 2x slower at 100KB). Full details in bench/c_accel/README.md and bd memory c-accel-spike-04c-final-2026-05-18 + luajit-ffi-boundary-cost-2026-05-18."}}
{"id":"int-09acf873","kind":"field_change","created_at":"2026-05-18T19:44:07.276353Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-80t","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18."}}
{"id":"int-4869e8ec","kind":"field_change","created_at":"2026-05-18T19:44:07.459433Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-1eq","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Measured as part of the 04c spike; numbers and architecture conclusion captured in 04c close, bench/c_accel/README.md, and bd memory c-accel-spike-04c-final-2026-05-18."}}
{"id":"int-af4c1c4d","kind":"field_change","created_at":"2026-05-18T20:19:41.337001Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-c0i","extra":{"field":"priority","new_value":"3","old_value":"1"}}
{"id":"int-a4bcd105","kind":"field_change","created_at":"2026-05-18T20:23:01.540904Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-pf6","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Architecture chosen and documented in docs/c-accel.md (commit pending). Ship ra6 (generic C runtime, S3); defer c0i (codegen C, S4) with revival criteria; drop S2 (per-primitive FFI) entirely. PB_ENABLE_C=1 opt-in, default off. Parity via existing-suite re-runs with C enabled. Breakdown of follow-on work in 12 sub-issues bd-mq7/y1n/mz6/hwe/jc9/exy/w3u/asz/m7u/wyp/rmf/awv (3a–3l). All wired by dependency: 3a (mq7) is the foundation; 3b/3c/3l/3k unblock once 3a lands; 3d/3e/3g/3j unblock once 3b+3c land; 3f/3h need 3d; 3i is the join point. ra6 itself depends on all twelve. c0i blocked-by ra6."}}
{"id":"int-ba4ac242","kind":"field_change","created_at":"2026-05-19T04:23:56.245983Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-47e","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Compatibility contract formalized in docs/specs/c_accel_compat.md. Spec covers: PB_ENABLE_C=1 single activation switch, silent fallback, full public surface preservation (pb.encode/decode/decode_lazy/parse/from_pb/json/text/grpc), generated module wrappers, 64-bit cdata invariant, WKT shapes, unknown fields, extensions, error compatibility, install paths (pure-Lua and C-enabled), ABI versioning, conformance via existing-suite re-run, explicit out-of-scope list. Lazy view stays Lua-only per spike rationale. Downstream tasks (mq7, ra6, 43t, c0i) reference this spec for compat boundaries."}}
{"id":"int-951d03f7","kind":"field_change","created_at":"2026-05-19T04:24:12.141983Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-z7x","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"C-side strategy formalized in docs/specs/c_accel_strategy.md. Spec covers: pb_plan / pb_plan_field struct layout, field-name luaL_ref caching (spike-validated), 4 KB stack-backed pb_buf with malloc promotion, lua_createtable(0, n_fields) result pre-sizing, cached per-field stack indices for repeated/packed (spike Phase B: 2x speedup at 100 KB), 64-bit cdata via luaT_pushint64/checkint64, sub-buffer (chosen) vs backpatching (deferred) for sub-messages, two-pass packed encode, map iteration via lua_next, oneof grouping, unknown field passthrough, deferred micro-opts (manual varint inlining, SIMD scan), C99/-O2 build environment, deferred PB_C_TRACE/PB_C_CHECK_PARITY env knobs. ra6/mq7 references this spec for low-level decisions."}}
{"id":"int-7ce7c344","kind":"field_change","created_at":"2026-05-19T04:24:13.236096Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-wky","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Build and packaging scaffolding landed: docs/specs/c_accel_build_packaging.md spec, runtime/pb/init.lua PB_ENABLE_C=1 opt-in pcall hook (exposed as pb.c_runtime field), Justfile build-c/clean-c recipes (stub erroring cleanly until bd-ra6 lands runtime/pb/c/), Justfile lua_cpath constant + LUA_CPATH on test/test-one, .gitignore for compiled .so/.dylib, sourcehut .builds/{pure-lua,c-enabled}.yml manifests (ubuntu/noble, parallel jobs via separate files since sourcehut has no matrix), .sourcehut/conformance.yml for manual conformance triggers. Rockspec changes deferred to first C-source landing (described in spec). All 745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1 (silent fallback verified)."}}
{"id":"int-b6e91276","kind":"field_change","created_at":"2026-05-19T04:46:39.175032Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-mq7","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Descriptor -> C plan compiler landed in runtime/pb/c/c_runtime.c. Compiles a finalized Lua descriptor into an opaque pb_plan userdata with: per-field records (field_number, wire_type, kind, repeated/packed/optional, pre-encoded tag bytes, sub_plan_idx, oneof_idx, enum_ref), oneof grouping (parallel array with member indices + field oneof_idx back-pointers), WKT override pointers (has_override + override_encode_ref/decode_ref skipping field-walk), extension range hooks (proto2 scaffolding), field-name luaL_ref cache, sub-plan table for cycle-safe recursion. Self-references (Person.friends -> Person) handled by stashing c_plan on desc BEFORE recursing, so resolve_sub_plan returns the same userdata. Idempotent: re-compiling returns cached plan. Build via 'just build-c'. Smoke test at test/c_runtime_plan_test.lua: 26 assertions across both codegen modes (full + runtime) verify module surface, scalar/enum/message/map/repeated/packed shapes, sub-plan resolution, self-reference cycle break, oneof grouping (Result.outcome), idempotency, WKT override detection (Timestamp). Full suite: 771/771 with PB_ENABLE_C=1, 745+26 skipped without (silent fallback verified). Unblocks 3b/3c/3l/3k (y1n, mz6, awv, rmf)."}}
{"id":"int-59f687bd","kind":"field_change","created_at":"2026-05-23T09:41:30.983997Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-y1n","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"C-side singular-scalar encode landed in runtime/pb/c/c_runtime.c. New entry pb.c_runtime.encode(plan, msg) -> string. Buffer strategy: 4KB stack scratch promoted to lua_newuserdata on overflow — mid-encode luaL_error doesn't leak (userdata GC'd at unwind). Covered kinds: int32/64, uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double, enum (number/string lookup via field->enum_ref's by_name), string, bytes. Repeated/map/message silently skipped (3d/3e scope). Proto3 zero-suppression mirrors mode=full exactly: empty strings, zero ints/fixed, +0.0 double skipped; -0.0 emitted via type-pun byte equality (matches Lua's 1/v == -math.huge guard); proto3-optional fields bypass suppression. WKT override plans rejected (3k/bd-rmf scope). New test test/c_runtime_encode_test.lua: 16 cases per mode (full + runtime) = 32 tests covering acceptance (Person {name='x', age=42, balance=-7, user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full), per-kind sweep, edge cases (proto3-optional empty string, -0.0 double, enum-string lookup + unknown-name error, cdata uint64 fixed64, heap-buffer grow at 8KB string, repeated/message-skip silent). Full suite: 806/806 with PB_ENABLE_C=1, 748/748 + 58 skipped without (silent fallback intact). Unblocks 3d/3e/3g/3i/3j."}}
{"id":"int-32f1e096","kind":"field_change","created_at":"2026-05-23T10:00:25.652311Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-mz6","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"C-side singular scalar/enum/string/bytes decode. pb.c_runtime.decode(plan, bytes) -> table mirrors mode=full output: int64 family pushes cdata via luaL_pushint64/pushuint64, others Lua number/string/boolean. Linear field_number scan over plan->fields per tag; repeated/map/message and unknown tags skipped by wire type — 3d/3e/3i extend later. 26 tests pass (full + runtime modes); full suite 832/832 with PB_ENABLE_C=1, 748/748+84 skipped without."}}
{"id":"int-634d0065","kind":"field_change","created_at":"2026-05-23T10:24:19.417709Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-hwe","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"ra6 3d: sub-message encode/decode (recursion) — refactored encode_lua/decode_lua into reusable encode_body/decode_body, added singular sub-message encode (force-promoted parent heap_idx before recursion to make lua_settop safe) and decode (temporarily shrunk c->len for bounded inner read). 22 new tests cover round-trip, empty sub-message presence, proto3-optional fields, 5-level depth (new test/proto/c_nested.proto), parent-buffer regrow path, truncated-input errors. Full suite 854/854 with PB_ENABLE_C=1."}}
{"id":"int-869e5402","kind":"field_change","created_at":"2026-05-23T10:39:55.349195Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-jc9","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-92bcd95a","kind":"field_change","created_at":"2026-05-23T11:07:38.503706Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-exy","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Acceptance tests landed: Person.emails (repeated string) and Person.friends (repeated Person, self-reference) round-trip byte-equal to mode=full at 1KB/10KB/100KB. Repeated string + message dispatch was already implemented as part of 3e (encode_repeated_field handles MESSAGE + string/bytes branches alongside scalars, decode_body uses cached list_stack_idx[] per-field); this ticket adds the formal size-scaled acceptance coverage."}}
{"id":"int-e3eda42e","kind":"field_change","created_at":"2026-05-23T11:17:06.762081Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-awv","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-a512d09e","kind":"field_change","created_at":"2026-05-23T13:17:50.84577Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-w3u","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-1b6998d4","kind":"field_change","created_at":"2026-05-23T17:07:06.194133Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-asz","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"ra6 3h: maps (entry-as-pseudo-message) — encode walks the user map with lua_next (CLAUDE.md JIT exception), builds each entry payload in a stack-backed sub-buffer with synthetic tag(1,key) + tag(2,value), proto3-elides defaults independently for key and value, and emits outer tag + len + payload into the parent buffer. Map<,message> resolves the value sub-plan once and recurses through encode_body. Decode reads the entry payload bounded, dispatches inner id=1/id=2 (skipping anything else per spec), and lua_rawsets into a lazy-created result map table; missing key or value falls back to the proto3 zero. Reuses the existing list_stack_idx[] slot for the lazy map cache (a field is either repeated or map, never both). 12 new tests cover round-trip for ages_by_nickname (string→int32), nickname_by_age (int32→string), and addresses_by_label (string→message), plus default-elision, multi-key correctness, empty maps, and unknown-inner-id tolerance. Full suite 964/964 with PB_ENABLE_C=1."}}
{"id":"int-bc164b8a","kind":"field_change","created_at":"2026-05-23T18:04:37.771425Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-wyp","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-b39fa74a","kind":"field_change","created_at":"2026-05-23T18:22:01.107078Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-rmf","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-8dbcb031","kind":"field_change","created_at":"2026-05-23T18:38:56.610443Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-m7u","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-500abbb3","kind":"field_change","created_at":"2026-05-23T19:50:37.871661Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-rc8","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-0be8d1bc","kind":"field_change","created_at":"2026-05-23T19:56:04.251964Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-ra6","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"All 16 sub-tasks (3a-3l) plus rc8 strict-decode parity bug closed. Generic C runtime codec landed: descriptor plan compiler, encode/decode scalars + repeated/packed + sub-messages + maps + oneofs + 64-bit cdata + WKT hooks + unknown fields + proto2 + dispatch + strict-decode parity. Ready for 43t parity gate."}}
{"id":"int-2a7d9cba","kind":"field_change","created_at":"2026-05-23T20:04:23.62044Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-43t","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Parity gate landed. Added test-c / test-all / bench-c Justfile recipes; bench.lua relabels runtime→c-runtime under PB_ENABLE_C=1 and refuses --baseline/--compare in that mode. Reuses existing 1043-test luatest matrix, Google conformance suite, and interop fixtures via the transitivity-through-reference strategy. c-generated column dropped from scope (c0i deferred per docs/c-accel.md); starwing column lives in bench/starwing_bench.lua separately."}}
{"id":"int-a631a4f9","kind":"field_change","created_at":"2026-05-23T20:13:43.682922Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-74c","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Phase-2 EmmyLua annotations landed. Added ---@param/---@return on public surface of codec.lua (encode_message/decode_message/encode_field/compile_writers/compile_readers/merge_message), grpc.lua (loopback/multiplex/new_stream_pair/wrap_call/wrap_server_stream/wrap_server_view), json.lua (M.encode/M.decode with new pb.JsonEncodeOpts/pb.JsonDecodeOpts shapes), and wkt.lua (register/lookup/any_pack/any_unpack). Also corrected two pre-existing _types.lua signature lies: pb.register is (desc), not (full_name, desc); pb.any.pack is (desc, t, prefix?), not (t, type_url). just test 752/752, just test-c 1043/1043."}}
{"id":"int-37f11812","kind":"field_change","created_at":"2026-05-23T20:48:32.931008Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-a6n","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Inlined 1-byte LEN fast path for string/bytes scalar + repeated fields in generated full-mode _decode; skips wire.decode_string/_bytes function call frame and inlines utf8_len. Dropped the U8CP cast-at-top idea — 24B/decode cdata allocation that LuaJIT couldn't sink, regressed small messages. Person_decode full: 10B -6.9%, 100B -7.2%, 1KB -10.7%, 10KB -12.2%, 100KB -14.0%. Zero alloc impact. Tests: 1043 pass, 37 JIT trace checks pass."}}
{"id":"int-1ae608c3","kind":"field_change","created_at":"2026-05-24T04:20:10.628185Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-bgu","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Spike (bench/c_accel/bgu_probe.lua, removed) measured ffi.string vs buf:sub for the 1-byte LEN string fast path on Tarantool 3.8.0/LuaJIT 2.1.0-beta3 / Apple M-series. Results (ns/call):\n\n  size  buf:sub  per-call-cast  amortized/26  amortized/4\n  10B   89.7     280.5 (3.1x)   224.9 (2.5x)  246.2 (2.7x)\n  32B   107.6    299.4 (2.8x)   243.8 (2.3x)  260.9 (2.4x)\n  80B   154.3    362.4 (2.3x)   300.1 (1.9x)  314.9 (2.0x)\n\nV1 (per-call ffi.cast(U8CP, buf) inside decode_string): 2.3-3.1x slower.\nV2 (cast hoisted to caller, threaded as ptr arg, amortized over 26 emails — the Person 1KB peak shape): still 1.9-2.5x slower. Cast-once-amortize-many fails because 'ptr + np - 1' mints a fresh cdata wrapper per call that the JIT cannot sink across the decode_string frame — same root cause as a6n's failed top-of-_decode cast. The amortization helps a little (V2 < V1) but does not close the gap.\n\nConclusion: bgu's premise (ffi.string lets the JIT fuse the string read with surrounding code) does not hold on the current LuaJIT fork — the cdata allocation cost dominates any reduction in dispatch. Closing as superseded by a6n-decode-inline-len-fast-path-2026-05-23. Revival criteria: a measurable cdata-sink win on a future LuaJIT (likely needs upstream allocation-sink improvements for pointer-arith cdata), OR an API redesign where the decoder works on a cdata buffer end-to-end (not a Lua string)."}}
{"id":"int-9713dfd3","kind":"field_change","created_at":"2026-05-24T04:30:26.987394Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-4ql","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Spike (bench/c_accel/ibuf_probe.lua, removed) implemented a hand-coded Person_encode_ibuf mirroring what protoc-gen-tarantool would emit: stable pre-allocated cdata scratch buffer, direct p[i] byte writes, single-pass with backpatched length for nested Address + packed lucky_numbers, ffi.copy(p+i, lua_str, n) for strings. Same-byte correctness verified across 10B/100B/1KB/10KB.\n\nBench (ns/op, Tarantool 3.8.0 / Apple M-series), Person_encode vs ibuf scratch-only:\n\n  size    Person_encode  ibuf scratch  ibuf+ffi.string  speedup\n  10B     462.8          239.1         273.7            1.69x (win)\n  100B    470.1          243.1         286.3            1.64x (win)\n  1KB     3693.8         4872.6        5141.6           0.72x (loss)\n  10KB    17146          42807         43023            0.40x (loss)\n  100KB   159081         422510        432701           0.37x (loss)\n\nCrossover ~26 emails. Root cause: each email pays an ffi.copy(p+i, lua_str, n) boundary (~50 ns/call). At 26 emails = ~1.3 us pure boundary; at 2800 emails (100KB) = ~140 us pure boundary. Meanwhile Person_encode appends Lua-string refs to an out table (no FFI boundary) and pays ONE bulk table.concat memcpy at the end regardless of count. Per-field boundary work beats per-message bulk work only when field count is very small.\n\nFor 4ql's stated use case (net.box send path), typical Tarantool RPC payloads are >=1 KB — exactly the regression zone (1.4-2.7x slower). Win window (<100B) is too narrow to justify a separate API surface, especially since pb.encode is already 462-470ns at that size — saving 200ns on a sub-microsecond operation is not a meaningful net.box gain.\n\nThree abandoned attempts now (per-byte b:alloc cliff, v2 two-pass bulk reserve, this single-pass backpatch). All hit the same per-field FFI boundary tax. Closing as 'not viable on current LuaJIT'. REVIVAL CRITERIA: a future LuaJIT that can sink FFI calls into traces (so per-field ffi.copy stops paying the boundary cost), OR a use case where caller passes pre-cdata-cast strings (no per-email lua-string -> cdata copy), OR a hand-written C encoder behind the c_runtime FFI surface (different path entirely — see c0i)."}}
{"id":"int-a5091fea","kind":"field_change","created_at":"2026-05-24T05:40:24.477555Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-kot","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-555861dc","kind":"field_change","created_at":"2026-05-24T10:00:51.655012Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-21d","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Landed via compile_encode_body in pb.codec. Trace topology met target (runtime/Person_encode 25 stops → 10; full is 8 — within +4). Encode throughput: small-Person +42% (10B 10.9→15.5, 100B 101→144 MB/s); large sizes +2-9%. Within-5%-of-full encode parity NOT met at all sizes (residual is per-closure call overhead — would need loadstring-inlined writer bodies, separate scope). Decode untouched. Filed h8x for pre-existing decode_group bimodal flake surfaced during validation."}}
{"id":"int-d04dbb29","kind":"field_change","created_at":"2026-05-24T10:55:07.008361Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-qwt","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-07a03cbf","kind":"field_change","created_at":"2026-05-24T10:55:07.983595Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-2sn","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Closed"}}
{"id":"int-13e4df9a","kind":"field_change","created_at":"2026-05-24T14:18:34.172401Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-h8x","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"wont_fix"}}
{"id":"int-d1b61dc9","kind":"field_change","created_at":"2026-05-24T15:24:55.485839Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-drm","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Closed"}}
{"id":"int-762f7441","kind":"field_change","created_at":"2026-05-24T16:53:14.99548Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-2ri","extra":{"field":"status","new_value":"closed","old_value":"open"}}
{"id":"int-a3f84d37","kind":"field_change","created_at":"2026-05-24T16:53:33.608711Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-lkz","extra":{"field":"status","new_value":"closed","old_value":"open"}}
{"id":"int-643d6ba3","kind":"field_change","created_at":"2026-05-24T16:53:43.706654Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-86g","extra":{"field":"status","new_value":"closed","old_value":"open"}}
{"id":"int-7db2afd6","kind":"field_change","created_at":"2026-05-24T17:05:10.538707Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-ch2","extra":{"field":"status","new_value":"closed","old_value":"in_progress"}}
{"id":"int-b981997e","kind":"field_change","created_at":"2026-05-24T17:20:53.622551Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-aah","extra":{"field":"status","new_value":"closed","old_value":"in_progress"}}
{"id":"int-2a1c9c90","kind":"field_change","created_at":"2026-05-24T18:20:11.709034Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-6bb","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Full-mode codegen emits <Msg>_decode_unsafe alongside <Msg>_decode that skips utf8_len at every string site (singular/repeated/map-kv/extensions/>=128-byte fallback) and recurses into sub-messages' _decode_unsafe. Skips pb.c_runtime dispatch (C runtime validates today; tracked in kyt). Runtime mode deferred to 58u (compile parallel _reader_unsafe closures). 6 tests in test/decode_unsafe_test.lua; full suite + conformance pass; ~20% perf win on string-heavy 1KB Person microbench. Generated code ~35% larger as expected. Docs in docs/api-modes.md."}}
{"id":"int-27d2602e","kind":"field_change","created_at":"2026-05-24T19:00:34.358869Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-b12","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Already fixed in commit 2656c97 (2026-05-17, same day issue filed). register_extension appends to extensions_list (parallel array); codec.encode_message, text.emit_message, json.encode_message all iterate via 'for i=1,#elist'. Inline-mode codegen also walks the list via 'for _i = N+1, #_elist' for runtime-registered extensions past the statically-known set. No pairs() over extensions remains anywhere on the hot path. The issue was never closed in bd."}}
{"id":"int-a28aba03","kind":"field_change","created_at":"2026-05-24T19:11:19.58095Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-58u","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Runtime mode now exposes pb.decode_unsafe and M.<Name>_decode_unsafe. codec.lua adds scalar_unsafe table, parameterizes build_reader/build_repeated_reader/decode_one to accept (scalar_tbl, decode_msg_fn, decode_group_fn), and adds compile_readers_unsafe + decode_message_unsafe + decode_group_unsafe + decode_extension_unsafe as literal clones with the three substitutions documented in codec.lua. Tests parameterized over both modes (14 cases), perf microbench shows ~8% gain in runtime mode (~20% in full mode). Conformance + JIT trace gates still pass. kyt remains open for unifying with C accel."}}
{"id":"int-4c37d283","kind":"field_change","created_at":"2026-05-24T19:48:24.175219Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-kyt","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"C runtime: decode_unsafe(plan, buf) entrypoint added; full-mode codegen _decode_unsafe dispatches via pb.c_runtime.decode_unsafe; pb.decode_unsafe in init.lua routes through C when c_runtime loaded. dec_ctx.skip_utf8 gates is_valid_utf8. Suites: test 766/766, test-c 1057/1057. Perf on string-heavy Person (418B, 16 emails + 16 nicknames): C-unsafe 455 MB/s vs C-safe 364 MB/s (+25%) vs Lua-unsafe 142 MB/s (3.2x)."}}
{"id":"int-f3ed4f0e","kind":"field_change","created_at":"2026-05-24T20:08:54.261959Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-auj","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Codegen: extended inline tag fast path (1-byte → 1+2-byte) and inline LEN fast path (1-byte → 1+2-byte) in emitInlineDecode / emitInlineStringBytesScalar / emitInlineStringBytesRepeated. Covers field IDs 1..4095 and string/bytes lengths 0..16383 without leaving the parent trace. Acceptance: criterion #2 met — multi-byte-varint Person decode 31.6→34.7 MB/s (+10%) on the big_person fixture (200B name + multi-byte packed lucky_numbers), small Person unchanged at 14 MB/s. Bridge rate now 4/100 (down from 10-20% at issue filing). Residual bridges have migrated from the tag site to the packed-varint inner loop (wire.decode_int32 in a packed payload); fully eliminating them would require per-scalar-type inline 2-byte paths in the packed-loop emitter — diminishing-returns territory, skipped. Suites: test 766/766, test-c 1057/1057."}}
{"id":"int-495792c2","kind":"field_change","created_at":"2026-05-24T20:14:48.721199Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-gi0","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Premise invalid. wire.encode_int32 / encode_int64 / encode_uint32 / encode_uint64 ARE literally encode_varint at module load (one assignment, same function value). kot's localize already captures wire.encode_int32 once per function, so the in-body call is a direct local read with no alias resolution. Verified: wire.encode_int32 == wire.encode_varint -> true. No extra TGETS at call time. The only residual is kot creating separate locals when a function uses both encode_int32 and encode_varint (could dedupe to one closure upvalue) — marginal closure-size win, not the 'two table lookups' premise. Closing as won't-fix; the work was already done by Lua alias semantics + kot localization."}}
{"id":"int-ec44fb9c","kind":"field_change","created_at":"2026-05-24T20:14:49.551175Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-a7l","extra":{"field":"status","new_value":"open","old_value":"in_progress"}}
{"id":"int-2308143b","kind":"field_change","created_at":"2026-05-24T20:22:34.980613Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-ozn","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Codegen: inlined 1-byte + 2-byte varint fast paths in the packed-scalar decode inner loop for int32/uint32/bool (and enum via a sister helper). Same pattern as auj's tag/LEN inline. Bridge rate dropped to ~1-2% (3/200), did not consistently hit 50-clean-in-a-row (max streak 31/50), but throughput criterion met with significant headroom:\n  small Person  (1-byte everything)         14.0 -> 23.4 MB/s   (+67%)\n  big   Person  (2-byte LEN + multi-byte packed) 34.7 -> 39.6 MB/s (+14% vs auj, +25% vs pre-auj)\nSkipped int64/uint64/sint*/sint64 inline — those return cdata or need zigzag wrappers; per-type complexity bigger than the residual gain on common workloads. Suites: test 766/766, test-c 1057/1057, bench/Person no regression at any size."}}
{"id":"int-c001d014","kind":"field_change","created_at":"2026-05-24T20:41:36.159459Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-3qu","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Added jit.opt.start('sizemcode=64', 'maxmcode=4096') to all 9 bench scripts (bench.lua, lazy_bench.lua, profile.lua, shapes_bench.lua, starwing_bench.lua, wire_bench.lua, alloc_probe.lua, map_bench.lua, packed_bench.lua — last two new since the issue was filed). All 7 non-interactive bench scripts run rc=0; profile.lua and starwing_bench.lua loadfile-check clean.\n\nRefreshed bench/baseline.json per step 2. Surprise win: full-mode decode allocations dropped 0.5-50% as a side-effect of auj/ozn that was only visible after the snapshot. proto2_basic.BenchPayload mid decode: 2.313 -> 1.156 KB/op (-50%); 1KB Person decode: 0.977 -> 0.953 KB/op. Runtime-mode unchanged — confirms the win is full-mode codegen specific.\n\nSteps 3-4 (update COMPARISON.md throughput tables, verify variance band shrinks) are deferred to a work.lab.local run per the bench-on-work-lab-local memory — laptop variance is 50%+ on identical state, can't trust throughput A/Bs locally."}}
{"id":"int-b63a66a3","kind":"field_change","created_at":"2026-05-24T21:02:23.021984Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-a7l","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Codegen emits typed fast variants for sint64/fixed64/sfixed64 — wire.encode_sint64_i(INT64(v)), wire.encode_fixed64_u(UINT64(v)), wire.encode_sfixed64_u(UINT64(v)) — skipping the to_int64/to_uint64 runtime type dispatch at every call site where the field type is statically known. Wire-level: added encode_sint64_i / encode_fixed64_u / encode_sfixed64_u alongside the existing encoders; encode_fixed64 is now a thin wrapper over encode_fixed64_u. INT64/UINT64 added as file-header upvalues alongside ffi.\n\nCodegen applied at: singular non-string scalar, singular required scalar, repeated non-packed per-element, extension singular, extension repeated per-element, packed scalar slow path, proto2 extension repeated. int32/int64/uint32/uint64 unchanged since they alias to encode_varint which doesn't call to_uint64 except on the slow-slow path.\n\nMeasured on c_int64.Wide encode (full mode, no PB_ENABLE_C, 4-run median):\n  lua-num inputs   2769 -> 2520 ns/op   (+9.9% throughput)\n  cdata inputs    12993 -> 12935 ns/op  (unchanged; to_int64 already takes the cdata fast path there)\n\nJust below the issue's 10-15% claim but real and measurable, and exactly the case where it matters (Lua-number-input is the common case for user code on sint64/fixed64 fields). Suites: test 766/766, test-c 1057/1057."}}
{"id":"int-e62c573d","kind":"field_change","created_at":"2026-05-24T21:22:51.675074Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-5y9","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Implemented as opt-in via plugin flag --tarantool_opt=int64_as_number=true (mode=full only; default off). Adds wire.decode_int64_n / decode_uint64_n / decode_sint64_n / decode_fixed64_n / decode_sfixed64_n that return Lua number when the decoded value fits [-2^53, 2^53] (inclusive — both endpoints are powers of two and exact as doubles), cdata otherwise. LL/ULL literals are int64_t/uint64_t cdata; comparisons compile to plain 64-bit integer compares on trace.\n\nCodegen plumbs cfg.Int64AsNumber through writer.int64AsNumber (avoids threading through 10+ function signatures). decodeFnSuffix() returns \"_n\" for the affected scalar types only when the flag is set. All wire.decode_<st> emit sites updated.\n\nPlugin guards: --int64_as_number=true with mode=runtime errors out (would require descriptor flag wired through pb.codec — out of scope). Under PB_ENABLE_C=1 the option is a no-op since the C runtime makes its own number-vs-cdata decision via luaL_pushint64; test gates accordingly with t.skip.\n\nMeasured tradeoff on c_int64.Wide decode (full mode, no PB_ENABLE_C, 5 fields):\n  tiny   (all 1-byte vars)   2050 -> 1700 ns/op   (-17%, cdata avoided entirely)\n  medium (3-byte vars)       3220 -> 3600 ns/op   (+11%, extra cmp+tonumber)\n  huge   (past 2^53)         7575 -> 7750 ns/op   (+2%, noise)\n\nWorkload-specific: enable for fields dominated by small IDs/counters/small timestamps (where 1-byte varint hits), leave off for large values. Documented in the plugin flag help.\n\nTest fixture: examples/expected/full_n/c_int64/c_int64_pb.lua regenerated by 'just gen-int64-as-number'. test/int64_as_number_test.lua covers byte-identical encoding, small-value Lua number returns, default cdata returns, 2^53 boundary inclusive, past-2^53 cdata fallback, Lua-number-input round-trip. Suites: test 771/771, test-c 1057/1057 (with 5 skipped under PB_ENABLE_C=1)."}}
{"id":"int-031d6264","kind":"field_change","created_at":"2026-05-25T04:29:44.952643Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-0u1","extra":{"field":"status","new_value":"closed","old_value":"open","reason":"Premise outdated — every typed varint decoder (decode_int32/int64/uint32/uint64/sint32/sint64/bool) already inlines the 1-byte fast path at the function head, mirroring decode_len's pattern. Verified at wire.lua:513-557. Closing as already-done."}}
{"id":"int-ef38f455","kind":"field_change","created_at":"2026-05-25T05:58:06.695205Z","actor":"Eugene Blikh","issue_id":"tarantool-protobuf-e4t","extra":{"field":"status","new_value":"closed","old_value":"in_progress","reason":"Already implemented in commit b75b879 (codegen: EmmyLua type annotations for messages, enums, wrappers). Generated _pb.lua files carry ---@alias for enums, ---@class+---@field for messages (with ? on proto3 explicit-optional and oneof branches), ---@param/---@return on _new/_encode/_decode/_decode_lazy/_text/_has_/_clear_ wrappers. Cross-file refs use proto full names (hello.Address, google.protobuf.Timestamp), so WKT and imports resolve to the same declared class. Verified just gen produces no diff and just test passes 771/771."}}