~bigbes/tarantool

tarantool-protobuf

ref: 1bbd8f313c8f0721802b427ce86c40b4f470e867 tarantool-protobuf/cmd/protoc-gen-tarantool/internal/gen/gen.go -rw-r--r-- 17.1 KiB
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
428ee1f6 — Eugene Blikh 3 months ago
codegen: propagate proto comments into generated _pb.lua

Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:

- Message / enum descriptions become `---` blocks above `---@class` /
  `---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
  matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
  lines inside the table literals so readers without an LSP still see
  the proto-side context.

Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
7ec8f2b4 — Eugene Blikh 3 months ago
text: add pb.text.decode and wire it into conformance dispatch

Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.

Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.

cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.

Closes the text-conformance-output branch.
0044b163 — Eugene Blikh 3 months ago
codegen: text-format printer (pb.text.encode + <Msg>_text wrappers)

Adds a descriptor-driven text-format encoder mirroring `protoc --decode`
output: one field per line, 2-space indent, octal byte escapes,
`nan`/`inf` floats, `opts.single_line=true` for compact one-liners.
WKT-aware — Timestamp/Duration accept datetime cdata or {seconds,nanos},
wrappers print their unwrapped scalar, Struct/Value/ListValue walk the
tagged-table form, FieldMask flattens to `paths:` lines, Any stays
opaque.

Both codegen modes emit `M.<Type>_text(t, opts)`, surfaced as `pb.text`
on the public table. Encode-only; matching parser deferred.

Tests cover scalars, repeated, maps, oneof, optional presence, all WKT
types, single-line mode, and the generated wrapper across both modes
(74 cases). 403/403 luatest green, 19/19 jit-trace gate green.
b75b8791 — Eugene Blikh 3 months ago
codegen: EmmyLua type annotations for messages, enums, wrappers

Generated Lua modules now carry lua-language-server type annotations:
  ---@alias <full.Enum> integer            per enum
  ---@class <full.Message>                  per message
  ---@field <name> <type>                   per field
  ---@param / ---@return                    per wrapper

Mappings:
  bool                       -> boolean
  string / bytes             -> string
  float / double             -> number
  all int kinds              -> integer  (64-bit cdata typed as integer;
                                          LSP has no cdata model)
  enum / message             -> <full.Name>  (resolves to declared alias/class)
  repeated T                 -> T[]
  map<K,V>                   -> table<K, V>

Presence markers (trailing `?` on field name):
  proto3 explicit optional
  oneof branches             (only one is set at a time)

Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').

Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.

Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.
77ccfc15 — Eugene Blikh 3 months ago
lazy: zero-copy decode view (decode_lazy) with passthrough re-encode

Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.

Surface (see runtime/pb/lazy.lua):
  - MessageView: :get / :has / :which / :iter / :names / :set / :encode
  - ArrayView:   :len / :at / :iter / :tolist
  - MapView:     :get / :has / :keys / :iter / :totable

:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).

Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.

Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.
784dea4d — Eugene Blikh 3 months ago
Initial commit: protoc-gen-tarantool plugin + pb runtime

A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.

Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.