~bigbes/tarantool

tarantool-protobuf

ref: b7e97d6a1ed1ba37d46a9cfc85c32b8ced751cf6 tarantool-protobuf/runtime/pb/lazy.lua -rw-r--r-- 24.1 KiB
8721548d — Eugene Blikh 2 months ago
lsp: foundational types + annotate pb runtime entry + lazy views

Add lua-language-server scaffolding so editors and LLM assistants get
parameter / return types for the public pb runtime. Pure metadata --
no runtime behavior change. 748 tests still pass.

- .luarc.json at repo root: declare LuaJIT runtime, project require
  paths (so require('full.hello.hello_pb') resolves), ignored
  directories (.rocks, generated dylibs, bench/starwing third-party).

- runtime/pb/_types.lua (@meta): foundational @class declarations
  for the descriptor contract (pb.Descriptor, pb.Field, pb.Enum-
  Descriptor, pb.OneofDescriptor), the public pb module surface
  (pb.Module), lazy view classes (pb.MessageView / pb.ArrayView /
  pb.MapView), gRPC transport contract (pb.GrpcTransport plus
  Service / Method descriptors), and the json / text / wire
  sub-modules. Mirrors docs/codegen.md's descriptor shape.

- runtime/pb/init.lua: @type pb.Module annotation on the returned
  table so hover on pb.encode / pb.decode_lazy / pb.parse picks up
  the signatures from _types.lua.

- runtime/pb/lazy.lua: @class annotations on the three view locals
  so lua-language-server merges them with the pb.MessageView /
  pb.ArrayView / pb.MapView declarations in _types.lua.

Next phases: annotate codec/grpc/json (phase 2), emit per-message
@class blocks from the plugin codegen (phase 3).
343e4ba4 — Eugene Blikh 3 months ago
text: render captured unknown fields, tolerate SGROUP in skip

Two changes close the proto3 text-format conformance suite:

  1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
     inner tags until a matching EGROUP, with field_id checked against
     the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
     full-mode `_pb.lua`) now pass the tag's field_id so groups inside
     unknown-field skips don't error.

  2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
     `opts.print_unknown_fields=true` and emits each entry in
     TextFormat numeric-field form:
       VARINT  -> "<id>: <uint64>"
       I64/I32 -> "<id>: 0x<hex>"
       LEN     -> speculative "<id> { <recurse> }"; rolls back to
                  byte-string form if the inner bytes don't parse as a
                  sub-message
       SGROUP  -> "<id> { <recurse> }" through matching EGROUP

`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.

Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
0c13fd73 — Eugene Blikh 3 months ago
wire: truncate varints to 32 bits in int32/uint32/sint32/enum decode

decode_int32/uint32/sint32 returned full uint64 values when the wire input
carried bits above bit 31, corrupting re-encode on 51 conformance tests that
exercise overlong or over-range varints. Per the proto3 spec the decoder
must keep only the low 32 bits (and sign-extend for signed types).

Adds wire.varint_to_int32 / varint_to_uint32 and routes every enum-varint
decode site through them: wire.lua typed decoders, codec.lua (5 enum
sites), lazy.lua (3 sites), and the generated code via inline.go (3 sites).

Drops 51 entries from test/conformance/known_failures.txt.
b75b8791 — Eugene Blikh 3 months ago
codegen: EmmyLua type annotations for messages, enums, wrappers

Generated Lua modules now carry lua-language-server type annotations:
  ---@alias <full.Enum> integer            per enum
  ---@class <full.Message>                  per message
  ---@field <name> <type>                   per field
  ---@param / ---@return                    per wrapper

Mappings:
  bool                       -> boolean
  string / bytes             -> string
  float / double             -> number
  all int kinds              -> integer  (64-bit cdata typed as integer;
                                          LSP has no cdata model)
  enum / message             -> <full.Name>  (resolves to declared alias/class)
  repeated T                 -> T[]
  map<K,V>                   -> table<K, V>

Presence markers (trailing `?` on field name):
  proto3 explicit optional
  oneof branches             (only one is set at a time)

Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').

Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.

Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.
ad5d7389 — Eugene Blikh 3 months ago
lazy: SoA index layout (sparse-read 0.66× → 0.90×, passthrough 1.5× → 1.7×)

Replace per-segment Lua tables with four parallel integer arrays
(id, tag_start, val_start, next_start). For an emails-heavy Person at
100KB that's 4 tables of 2800 ints instead of 2800 tables of 4 keys —
~5× fewer table allocations on decode_lazy.

ArrayView and MapView are similarly flattened: each holds a single
int array of val_starts instead of one mini-table per element.
Packed-payload expansion produces the same shape so :at(i) is one
array index lookup + decode call.

Before / after on bench/lazy_bench.lua (1KB / 10KB / 100KB):

  passthrough (decode + reencode)
    full:    1.43× → 1.73×   1.13× → 1.62×   1.04× → 1.69×
    runtime: 1.43× → 1.94×   1.15× → 1.73×   1.16× → 1.84×

  sparse read (:get name + :get age)
    full:    0.70× → 0.90×   0.65× → 0.94×   0.60× → 1.03×
    runtime: 0.76× → 0.99×   0.66× → 1.05×   0.68× → 1.16×

  rewrite name (decode + set + reencode)
    full:    0.98× → 1.11×   0.82× → 1.09×   0.81× → 1.14×
    runtime: 1.06× → 1.26×   0.95× → 1.15×   0.85× → 1.25×

Sparse read goes from a loss to break-even or better; passthrough
gain widens; mutate-then-reencode flips from regression to consistent
win. JIT-trace gate still 19/19 — the index loop's single side-trace
bridge (decode_tag at lazy.lua:49) was at decode_tag in the old code
too; same pattern, different line number.

Drops the unused wt field from the SoA: consumers never re-read the
tag wire type after the index pass. Unknown-field splice and lazy
:encode emit byte slices directly from tag_start..next_start.
77ccfc15 — Eugene Blikh 3 months ago
lazy: zero-copy decode view (decode_lazy) with passthrough re-encode

Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.

Surface (see runtime/pb/lazy.lua):
  - MessageView: :get / :has / :which / :iter / :names / :set / :encode
  - ArrayView:   :len / :at / :iter / :tolist
  - MapView:     :get / :has / :keys / :iter / :totable

:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).

Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.

Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.