~bigbes/tarantool

tarantool-protobuf

ref: ed383c9288055385f17c848de41e0aa7a2089dc4 tarantool-protobuf/cmd/protoc-gen-tarantool d---------
ed383c92 — Eugene Blikh 2 months ago
codegen: inline proto2 extension writers/readers (qwt) + table.new(N,0) for packed lists (2sn)

qwt closes the proto2_basic.BenchPayload `min` bench's worst data point:
encode 1.89M → 3.30M msgs/s (+75%), decode 1.02M → 1.26M msgs/s (+23%).
Mechanism: collectExtsByExtendee groups all `extend Foo { ... }`
declarations by extendee FullName across input files; emitInlineEncode
emits one dedicated writer per extension instead of dispatching through
pb.codec.encode_field, and emitInlineDecode adds an elseif arm per
extension id straight into result._extensions[full_name]. Dynamic
extensions_list walk preserved for forward compat, gated on
#_elist > N so it pays one int compare when no runtime extension was
registered.

2sn pre-sizes packed-scalar repeated lists via table.new(N, 0) where
the count is recoverable from the LEN payload — exact (lim >> 2 or
lim >> 3) for fixed-width, upper bound (lim) for varint-packed. The
allocation is deferred into the `if wt == 2 then` branch so the
per-element fallback and non-packable types keep the bare-`{}`
alloc. Avoids u39's regression mode because the call cost is paid
once per repeated-field-first-occurrence, not per message decode.

Bench summary for hello.Person full mode: encode +3-5% across sizes,
decode +1-3% across sizes. Tests: 752/752 pure-Lua, 1043/1043 with
PB_ENABLE_C=1, JIT 37/37. PERF_LOG entry covers the rationale and
caveats.
0fb62135 — Eugene Blikh 2 months ago
codegen: localize wire.* upvalues per generated message function

Capture each _encode/_decode body, scan for wire.<name> refs, and
rewrite refs that appear >=2 times to bare locals with a
"local X = wire.X" prelude. Single-use refs stay as wire.X — without
the threshold the prelude TGETS outweighed the in-body saving on
sparse small-message decode.

Measured (median-of-3, shapes bench full mode):
- scalar-heavy: enc +38.5%, dec +39.2%
- packed-int32x100: enc +51.1%, dec +38.2%
- map-strxi32-*: enc +5.5%, dec +6-7%
- nested/oneof/wkt: +1-5% (within ~5% variance band)
- bench.lua Person small sizes: neutral

closes kot
b09c7213 — Eugene Blikh 2 months ago
codegen: inline 1-byte LEN fast path for string/bytes decode

Singular and repeated string/bytes fields in generated full-mode _decode
now read the length byte and dispatch in-line instead of calling
wire.decode_string / wire.decode_bytes. For length < 128 (the common
short-string RPC case), the path stays inside the parent JIT trace:
no child-trace stitch, no per-element function frame, utf8_len lookup
hoisted to a generated-file upvalue.

The original a6n design — one ffi.cast(U8CP, buf) at the top of each
_decode — was abandoned: the cdata wrapper is 24 bytes per call and
LuaJIT can't sink the allocation because the pointer local lives
across wire.decode_* call frames. Net regression at small sizes
(+11-16%) overwhelmed the single-byte-read savings.

Bench (hello.Person full-mode decode):

      size    ns/op before   ns/op after   delta
      10B          419            390     -6.9%
      100B         515            478     -7.2%
      1KB         9004           8042    -10.7%
      10KB       64644          56745    -12.2%
      100KB     625638         537750    -14.0%

Zero allocation impact across all sizes. Tests: 1043 pass, 37 JIT
trace gates pass. (a6n)
1eb062c1 — Eugene Blikh 2 months ago
c_runtime: wire encode/decode dispatch + conformance-c harness

pb.encode / pb.decode lazy-compile desc.c_plan on first call and route
to pb.c_runtime.encode/decode when PB_ENABLE_C=1 loaded the module.
Eager compile at finalize_message time fails on codegen's
forward-declared descriptors — sub-messages don't have .fields yet —
so compilation is deferred until first encode/decode, by which time
the whole module table is populated and sub-plan chase resolves.

Full-mode codegen wrappers (M.<Type>_{encode,decode}) gain the same
lazy-compile prologue, bypassing the inline body when the C runtime
is loaded. Runtime-mode wrappers already call pb.encode and pick up
dispatch centrally.

Harness side: the conformance Docker image now installs tarantool-dev
+ build-essential so the C runtime can be built in-container; a new
`just conformance-c` recipe builds runtime/pb/c_runtime.so inside the
container, runs the suite with PB_ENABLE_C=1, then cleans the .so to
keep the bind mount free of foreign-platform binaries. The runner
script also pre-populates package.loaded.pb (avoids
.rocks/lib/tarantool/pb.so from starwing lua-protobuf masking ours)
and orders package.cpath by jit.os so mixed .dylib/.so trees from
host/container interleavings don't cross-load.

Verified:
- just test                 → 748 pass, 291 C-conditional skipped
- PB_ENABLE_C=1 just test   → 1026/1039 pass; 13 fails are strict-decode
                              gaps in the C decoder (bd-rc8)
- just conformance-c        → 2729/2806 binary suite pass; 77 unexpected
                              failures match the same gap categories
                              (illegal wire-type 6/7, field-num 0/over,
                              overlong tag varint, UTF-8 rejection,
                              message merge for oneof/repeated)

Strict-decode parity tracked in bd-rc8; this commit closes the wiring
half of bd-43t (conformance gate for C paths).
a6eebdb3 — Eugene Blikh 3 months ago
codegen: local counter per repeated field on decode

Replaces `list[#list + 1] = val` with `_n_<fname> = _n_<fname> + 1;
list[_n_<fname>] = val` in every generated M.X_decode repeated-field
append site. One counter local per repeated non-map field, declared at
function entry. Counters survive across loop iterations so out-of-order
wire entries for the same field continue past the existing length
without re-scanning.

Profile attributed 6.2% of hello.Person 1KB decode to the `#list + 1`
re-traversal (26-email Person paid 26 list scans per decode).

Bench (Person full decode, msgs/s, median-of-3 vs post-4kj):
1KB +5.0%, 10KB +5.0%, 100KB +7.1%. Encode flat. Tests 745/745.
JIT 37/37, 0 bridges.

beads-tarantool-protobuf-cch
feecf8e0 — Eugene Blikh 3 months ago
codegen: inline 1-byte tag fast path at decode call sites

Hoists wire.decode_tag's 1-byte fast path into every generated
M.X_decode while-loop, falling back to the helper for multi-byte
tags (field ids > 15). The 1-byte case covers every protobuf field
with id 1..15 and is the dominant decode dispatch in real payloads.

Header now localizes string.byte, bit.band, and bit.rshift so the
inlined ops compile to straight local calls.

Bench (Person full decode, msgs/s, median-of-3 vs proper post-h8v
3-run baseline): 10B +14.7%, 100B +8.9%, 1KB +7.5%, 10KB +7.0%,
100KB +8.5%. Full encode is flat to small (-0.1% to -2.6%) at large
sizes, plausibly from header-upvalue layout. JIT trace gate: 37/37,
all bridges still 0. Tests: 745/745.

Also documented in bench/PERF_LOG.md, including the methodology
note that h8v's earlier numbers used single-run baselines and are
therefore ~3-5% optimistic; medians-of-3 are the standard now.

beads-tarantool-protobuf-4kj
9ee21c09 — Eugene Blikh 3 months ago
codegen: inline 1-byte varint length prefix at every LEN emit site

Eliminates the wire.encode_varint(#body) call + dispatch for every
length-delimited field in mode=full codegen. Profile flagged the
out[n] = wire.encode_varint(#_b) line as ~33% of hello.Person 1KB
encode time, with another ~17% in encode_varint dispatch — together
~50% of encode time. Lifting the 1-byte fast path (the dominant case
for proto strings and small message bodies) to the call site removes
the function frame entirely for lengths < 128.

Applied at every LEN emit site: singular/repeated message body,
singular/repeated string|bytes, packed scalar bundle, packed enum
bundle, map entry. Map value pieces (emitMapPiece message branch)
left as-is — they sit inside a single slot assignment that would
require a deeper restructure, and maps are not on the current hot
benchmark.

Results (hello.Person full encode, msgs/s): 10B +6.9%, 100B +7.9%,
1KB +25.7%, 10KB +48.1%, 100KB +31.9%. Decode flat (unchanged path).
Runtime mode flat (descriptor dispatch still calls encode_varint).
JIT trace gate: 37/37. Test suite: 745/745.

Bench history saved to bench/PERF_LOG.md with full numbers and the
workflow this iteration follows.

beads-tarantool-protobuf-h8v
2656c977 — Eugene Blikh 3 months ago
bench,codec: pin proto2 paths, kill pairs() on extension hot path

JIT trace gate gains 12 proto2-specific checks (required encode/decode,
groups singular + repeated, extension encode/decode). The extension-encode
check fired NYI on bytecode 72 (ISNEXT) — pairs() over the new
extensions_by_full_name hash is the same trace-abort that gates map
encode. Fix: register_extension now also appends to extensions_list,
an array view. codec.encode_message, text.emit_message and
json.encode_message all switch to ipairs over the list. Hash tables
stay around for O(1) lookups in decode (extensions_by_id) and bracket-
name resolution (extensions_by_full_name).

Inline (full-mode) codegen learns to walk extensions too: before this
commit the inline encoder skipped t._extensions entirely (only the
runtime codec walked it), so a generated _encode silently dropped any
extension data set on the message. Add the array-walk after the field
loop and a decode_extension dispatch in the unknown-tag branch — the
inline path now matches runtime byte-for-byte.

bench/bench.lua parameterizes over a FIXTURES list so the baseline can
cover both hello.Person and a new proto2_basic.BenchPayload fixture
(required + group + extension + repeated). baseline.json restructured
under "schemas": [{schema, results}, …]; compare walks both. Fresh
numbers committed.

740/740 luatest cases pass; JIT gate 37/37; conformance 2806 binary+JSON
and 434 text-format, both 0 failures.
5a1dc35b — Eugene Blikh 3 months ago
codec,text,json: proto2 extensions end-to-end

Plugin iterates file.Extensions and per-message Extensions, emitting
\`pb.register_extension(extendee_desc, {…})\` calls. Each extension
becomes a field descriptor attached to the extendee under two
indices: extensions_by_id (wire-decode lookup) and
extensions_by_full_name (encode + textual emission ordering).

Codec encode_message walks data._extensions after the regular field
loop and calls encode_field through the extension's descriptor.
decode_message routes an unknown tag through decode_extension when
extensions_by_id matches; the helper mirrors the in-line dispatch
on field kind (scalar/enum/message/group, singular/repeated) and
stores into result._extensions[full_name].

Text decoder recognizes bracket syntax \`[pkg.ext_name]\` outside an
Any-target and looks the name up in extensions_by_full_name. Names
that don't resolve (e.g. \`[pkg.GroupField]\` for the CapitalCase
group type instead of the lowercase field name) raise a parse error
per spec. Text encoder emits each set extension as
\`[full.name]: value\` or \`[full.name] { … }\`.

JSON encoder emits set extensions under the bracketed full-name key
(\`"[pkg.ext_name]": value\`). JSON decoder recognizes the same
shape and stores into _extensions; unknown bracket-names stay
silently dropped to match the spec's tolerant behavior for unknown
JSON keys.

Extensions on google.protobuf.* descriptors (file/message/field
option extends) are skipped — those are meta-only and our WKT
module doesn't surface their descriptors.

Conformance baseline (strict --enforce_recommended, protobuf v34.1):
  Binary + JSON suite: 2806 ✓ / 0 failures   (was 1493 ✓ / 1313 skipped)
  Text-format suite:    434 ✓ / 0 failures   (was  416 ✓ /   18 skipped)

100% pass.
e9b650e9 — Eugene Blikh 3 months ago
text,codec: proto2 group text syntax + closed-enum semantics

Text decoder now resolves a group field reference by the group's
capitalized submessage name (e.g. \`Data\`, \`MultiWordGroupField\`)
and by the ASCII-lowercase fold (\`data\`, \`multiwordgroupfield\`),
in addition to the existing lowercase field-name lookup. The \`:\`
between field and \`{\` is optional for groups (matches the
message/map rule).

Proto2 enums are closed: parse_enum_value now rejects an integer
literal that doesn't map to any declared value. The plugin emits
\`closed = true\` on every proto2 enum descriptor (driven by
protoreflect's IsClosed()); proto3 enums stay open to preserve
forward-compatibility on the wire.

Conformance text-format suite: 16 unexpected failures → 3, all in
the remaining extension-bracketed-group cases.
5fa0fd00 — Eugene Blikh 3 months ago
codec: proto2 groups (SGROUP/EGROUP wire format)

Plugin: detect protoreflect.GroupKind, emit kind='group' in the
field descriptor and a SGROUP opening tag in inline codegen. The
group body bypasses the LEN-prefix path entirely — encoder emits
start-tag, body bytes, end-tag in three slots; decoder calls
pb.codec.decode_group(desc, buf, pos, field_id) which reads inner
tags until the matching EGROUP id.

Wire layer already understood SGROUP/EGROUP for skip_field; the new
decode_group reuses the per-tag dispatch from decode_message but
stops on EGROUP instead of end-of-buffer. Repeated groups bracket
each element with its own SGROUP/EGROUP pair.

Runtime parser now desugars `optional|required|repeated group Name
= id { body }` into (a) a nested message named Name and (b) a
synthetic field of kind=group whose lowercased name is `name` — so
source-parsed proto2 schemas behave the same as build-time codegen.

Text format renders groups under the submessage's capitalized name
(`SingleGroup { ... }`) rather than the lowercase field name,
matching mainline protoc's convention.

Adds a vendored, MessageSet-stripped test_messages_proto2.proto in
test/conformance/proto/ (header comment documents the patch).
10 new luatest cases (group_*) cover singular + repeated + text +
descriptor kind, across both codegen modes. 739 tests pass.
4fbdb65d — Eugene Blikh 3 months ago
codegen: proto2 baseline — required, optional, custom defaults

Lifts the proto3-only syntax gate in the plugin and threads three
new field-descriptor attributes through codegen and the codec:

  * required=true   — fields declared with the proto2 `required` keyword.
                      Inline codegen and the runtime codec both error when
                      a required field is missing on encode (vs the silent
                      elide that proto3 implicit-presence fields get).
  * optional=true   — already wired for proto3 explicit `optional`; in
                      proto2 every singular field carries it via the
                      existing HasOptionalKeyword() check, giving presence
                      semantics without a separate emission path.
  * default_value=… — proto2 [default = X] from the field descriptor,
                      rendered as a Lua literal (cdata for 64-bit ints,
                      symbolic name for enums) so consumers can surface
                      it; the codec itself does not auto-materialize
                      defaults on decode, matching how proto3 absent
                      fields stay nil.

Packed-by-default already flips correctly because we ask
protoreflect's `IsPacked()`, which is syntax-aware.

Adds test/proto/proto2_basic.proto with 33 luatest cases covering
required validation, optional presence, custom defaults, the proto2
unpacked-by-default repeated rule, nested-required messages, and
full-vs-runtime mode parity. `just gen-proto2-tests` regenerates the
fixture into examples/expected/{full,runtime}/.

Out of scope: extensions, extend, group; conformance harness still
skips TestAllTypesProto2.
7676fdc2 — Eugene Blikh 3 months ago
codegen: preserve descriptor options as `options = { ... }`

Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.

The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.

The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.

UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
14e19819 — Eugene Blikh 3 months ago
codegen: bracket-quote Lua-keyword field names

The plugin emitted bare-identifier field names in three positions:
the `M.<Type>_fields` table key, the inline encoder's `v = t.<name>`
load, and the inline decoder's `result.<name> = ...` store. When a
proto field name collided with a Lua reserved word the generated
`*_pb.lua` failed to load with `'(' expected near '<keyword>'`.

The real-world hit is pprof's profile.proto, which declares
`repeated Function function = 5` — that breaks all three emit sites.

Route every user-named identifier through new luaTableKey /
luaFieldAccess helpers that bracket-quote reserved words. Covers
field_names + oneof descriptors, oneof presence pre-pass, inline
encode/decode field bodies (including repeated and map paths), and
optional has/clear accessors.
06b2978a — Eugene Blikh 3 months ago
codegen: resolve (tarantool.lua_package) via global type registry

The custom file option was being dropped silently: protoc encoded it
correctly into the FileDescriptorProto, but protobuf-go parked the
unknown extension in the message's unknown-fields tail because
E_LuaPackage was never registered with protoregistry.GlobalTypes. As a
result proto.GetExtension returned "" and every override the README
documented was a no-op — every caller fell through to the default
"<pkg>.<file>_pb" path resolution.

Register E_LuaPackage in init() and add a luatest regression that
asserts both the output path and the cross-file require strings honor
the option, parameterized over both codegen modes.
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
428ee1f6 — Eugene Blikh 3 months ago
codegen: propagate proto comments into generated _pb.lua

Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:

- Message / enum descriptions become `---` blocks above `---@class` /
  `---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
  matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
  lines inside the table literals so readers without an LSP still see
  the proto-side context.

Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
f5c5ee6c — Eugene Blikh 3 months ago
codec: preserve -0.0 for proto3 float/double scalars

Proto3 default-elision dropped any singular float/double whose value
compared equal to 0 — `-0.0 == 0.0` in IEEE so the `if v ~= 0` guard
silently elided negative zero. The wire bytes for -0 differ from +0
and the TextFormatInput conformance corpus pins that -0 must survive
a round-trip; the failure surfaced as 10 unexpected text-suite
regressions covering FloatFieldNegativeZero (3 spellings × 2 outputs)
and Neg{Float,Double}FieldLargeNegativeExponentParsesAsNegZero
(2 types × 2 outputs).

Sign-bit guard via `1/v == math.huge` (positive zero yields +inf,
negative zero yields -inf). Applied in three layers:
  * runtime/pb/codec.lua: is_default_scalar + the specialized
    monomorphic writer for non-optional numeric scalars
  * runtime/pb/text.lua: is_proto3_default (text encoder elision)
  * inline codegen: scalarNotDefaultExpr emits the guard for
    float/double fields in full-mode `_encode` functions

Proto3 text-format conformance now sits at 416 ✓ / 18 skipped / 0
expected failures; binary+JSON holds at 1478 ✓. Unit-test regressions
cover both directions (preserved on encode + decode round-trip, +0
still elided) and use a runtime-computed -0.0 sentinel because LuaJIT
can constant-fold the literal `-0.0` to a sign-less zero in some
load paths.
7ec8f2b4 — Eugene Blikh 3 months ago
text: add pb.text.decode and wire it into conformance dispatch

Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.

Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.

cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.

Closes the text-conformance-output branch.
343e4ba4 — Eugene Blikh 3 months ago
text: render captured unknown fields, tolerate SGROUP in skip

Two changes close the proto3 text-format conformance suite:

  1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
     inner tags until a matching EGROUP, with field_id checked against
     the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
     full-mode `_pb.lua`) now pass the tag's field_id so groups inside
     unknown-field skips don't error.

  2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
     `opts.print_unknown_fields=true` and emits each entry in
     TextFormat numeric-field form:
       VARINT  -> "<id>: <uint64>"
       I64/I32 -> "<id>: 0x<hex>"
       LEN     -> speculative "<id> { <recurse> }"; rolls back to
                  byte-string form if the inner bytes don't parse as a
                  sub-message
       SGROUP  -> "<id> { <recurse> }" through matching EGROUP

`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.

Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
Next