~bigbes/tarantool

tarantool-protobuf

ref: ff460b9b7aa9700c1648300000750e62e9108e45 tarantool-protobuf/runtime/pb/lazy.lua -rw-r--r-- 24.0 KiB
0c13fd73 — Eugene Blikh 3 months ago
wire: truncate varints to 32 bits in int32/uint32/sint32/enum decode

decode_int32/uint32/sint32 returned full uint64 values when the wire input
carried bits above bit 31, corrupting re-encode on 51 conformance tests that
exercise overlong or over-range varints. Per the proto3 spec the decoder
must keep only the low 32 bits (and sign-extend for signed types).

Adds wire.varint_to_int32 / varint_to_uint32 and routes every enum-varint
decode site through them: wire.lua typed decoders, codec.lua (5 enum
sites), lazy.lua (3 sites), and the generated code via inline.go (3 sites).

Drops 51 entries from test/conformance/known_failures.txt.
b75b8791 — Eugene Blikh 3 months ago
codegen: EmmyLua type annotations for messages, enums, wrappers

Generated Lua modules now carry lua-language-server type annotations:
  ---@alias <full.Enum> integer            per enum
  ---@class <full.Message>                  per message
  ---@field <name> <type>                   per field
  ---@param / ---@return                    per wrapper

Mappings:
  bool                       -> boolean
  string / bytes             -> string
  float / double             -> number
  all int kinds              -> integer  (64-bit cdata typed as integer;
                                          LSP has no cdata model)
  enum / message             -> <full.Name>  (resolves to declared alias/class)
  repeated T                 -> T[]
  map<K,V>                   -> table<K, V>

Presence markers (trailing `?` on field name):
  proto3 explicit optional
  oneof branches             (only one is set at a time)

Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').

Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.

Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.
ad5d7389 — Eugene Blikh 3 months ago
lazy: SoA index layout (sparse-read 0.66× → 0.90×, passthrough 1.5× → 1.7×)

Replace per-segment Lua tables with four parallel integer arrays
(id, tag_start, val_start, next_start). For an emails-heavy Person at
100KB that's 4 tables of 2800 ints instead of 2800 tables of 4 keys —
~5× fewer table allocations on decode_lazy.

ArrayView and MapView are similarly flattened: each holds a single
int array of val_starts instead of one mini-table per element.
Packed-payload expansion produces the same shape so :at(i) is one
array index lookup + decode call.

Before / after on bench/lazy_bench.lua (1KB / 10KB / 100KB):

  passthrough (decode + reencode)
    full:    1.43× → 1.73×   1.13× → 1.62×   1.04× → 1.69×
    runtime: 1.43× → 1.94×   1.15× → 1.73×   1.16× → 1.84×

  sparse read (:get name + :get age)
    full:    0.70× → 0.90×   0.65× → 0.94×   0.60× → 1.03×
    runtime: 0.76× → 0.99×   0.66× → 1.05×   0.68× → 1.16×

  rewrite name (decode + set + reencode)
    full:    0.98× → 1.11×   0.82× → 1.09×   0.81× → 1.14×
    runtime: 1.06× → 1.26×   0.95× → 1.15×   0.85× → 1.25×

Sparse read goes from a loss to break-even or better; passthrough
gain widens; mutate-then-reencode flips from regression to consistent
win. JIT-trace gate still 19/19 — the index loop's single side-trace
bridge (decode_tag at lazy.lua:49) was at decode_tag in the old code
too; same pattern, different line number.

Drops the unused wt field from the SoA: consumers never re-read the
tag wire type after the index pass. Unknown-field splice and lazy
:encode emit byte slices directly from tag_start..next_start.
77ccfc15 — Eugene Blikh 3 months ago
lazy: zero-copy decode view (decode_lazy) with passthrough re-encode

Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.

Surface (see runtime/pb/lazy.lua):
  - MessageView: :get / :has / :which / :iter / :names / :set / :encode
  - ArrayView:   :len / :at / :iter / :tolist
  - MapView:     :get / :has / :keys / :iter / :totable

:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).

Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.

Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.