~bigbes/tarantool

tarantool-protobuf

ref: ad5d73898eeaf88d40e97612141ebcedd6d8ed02 tarantool-protobuf/runtime/pb/lazy.lua -rw-r--r-- 23.0 KiB
ad5d7389 — Eugene Blikh 3 months ago
lazy: SoA index layout (sparse-read 0.66× → 0.90×, passthrough 1.5× → 1.7×)

Replace per-segment Lua tables with four parallel integer arrays
(id, tag_start, val_start, next_start). For an emails-heavy Person at
100KB that's 4 tables of 2800 ints instead of 2800 tables of 4 keys —
~5× fewer table allocations on decode_lazy.

ArrayView and MapView are similarly flattened: each holds a single
int array of val_starts instead of one mini-table per element.
Packed-payload expansion produces the same shape so :at(i) is one
array index lookup + decode call.

Before / after on bench/lazy_bench.lua (1KB / 10KB / 100KB):

  passthrough (decode + reencode)
    full:    1.43× → 1.73×   1.13× → 1.62×   1.04× → 1.69×
    runtime: 1.43× → 1.94×   1.15× → 1.73×   1.16× → 1.84×

  sparse read (:get name + :get age)
    full:    0.70× → 0.90×   0.65× → 0.94×   0.60× → 1.03×
    runtime: 0.76× → 0.99×   0.66× → 1.05×   0.68× → 1.16×

  rewrite name (decode + set + reencode)
    full:    0.98× → 1.11×   0.82× → 1.09×   0.81× → 1.14×
    runtime: 1.06× → 1.26×   0.95× → 1.15×   0.85× → 1.25×

Sparse read goes from a loss to break-even or better; passthrough
gain widens; mutate-then-reencode flips from regression to consistent
win. JIT-trace gate still 19/19 — the index loop's single side-trace
bridge (decode_tag at lazy.lua:49) was at decode_tag in the old code
too; same pattern, different line number.

Drops the unused wt field from the SoA: consumers never re-read the
tag wire type after the index pass. Unknown-field splice and lazy
:encode emit byte slices directly from tag_start..next_start.
77ccfc15 — Eugene Blikh 3 months ago
lazy: zero-copy decode view (decode_lazy) with passthrough re-encode

Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.

Surface (see runtime/pb/lazy.lua):
  - MessageView: :get / :has / :which / :iter / :names / :set / :encode
  - ArrayView:   :len / :at / :iter / :tolist
  - MapView:     :get / :has / :keys / :iter / :totable

:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).

Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.

Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.