~bigbes/tarantool

tarantool-protobuf

ref: 7a77bcfc1c6f4d29600ee7636ff174162c40087a tarantool-protobuf/test d---------
5c705557 — Eugene Blikh 2 months ago
c-accel: descriptor -> C plan compiler (bd-mq7)

First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.

What the plan carries (per docs/specs/c_accel_strategy.md):

  * Per-field records: field_number, wire_type, kind, repeated/packed/
    optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
    sub_plan_idx (1-based into the sub-plans table), enum_ref
    (luaL_ref for enum descriptor), oneof_idx back-pointer.
  * Map fields capture map_key_kind, map_value_kind, and the value's
    sub_plan_idx when the value is a message.
  * Oneofs as a parallel array of {name, member_indices[]}, with
    fields' oneof_idx pointing back to their group.
  * WKT override detection: when desc.encode/desc.decode are set, the
    plan flips has_override=1 and skips field-walk entirely.
  * Extension range hooks captured (proto2 scaffolding for bd-3i).
  * Field-name cache as a Lua table referenced via luaL_ref, so
    encode/decode can do lua_rawgeti instead of re-interning C strings.

Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.

Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.

Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
  - module surface + ABI version + KIND/WIRE constants
  - Address: 4 fields, names, kinds, wire types, tag bytes, optional
  - Person: 14 fields, scalars/enum/message/map/repeated/packed
  - Sub-plan resolution + Person.friends self-reference cycle break
  - Idempotent compile (second call returns cached plan)
  - Result.outcome oneof: 3 members, oneof_idx back-pointers
  - WKT Timestamp: has_override=true, field-walk skipped

Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).

Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.

Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
b1a11473 — Eugene Blikh 3 months ago
docs: retire PLAN.md in favor of Beads issue tracking

Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.

References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.

The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.

Closes the goal of bd-8k2.
2656c977 — Eugene Blikh 3 months ago
bench,codec: pin proto2 paths, kill pairs() on extension hot path

JIT trace gate gains 12 proto2-specific checks (required encode/decode,
groups singular + repeated, extension encode/decode). The extension-encode
check fired NYI on bytecode 72 (ISNEXT) — pairs() over the new
extensions_by_full_name hash is the same trace-abort that gates map
encode. Fix: register_extension now also appends to extensions_list,
an array view. codec.encode_message, text.emit_message and
json.encode_message all switch to ipairs over the list. Hash tables
stay around for O(1) lookups in decode (extensions_by_id) and bracket-
name resolution (extensions_by_full_name).

Inline (full-mode) codegen learns to walk extensions too: before this
commit the inline encoder skipped t._extensions entirely (only the
runtime codec walked it), so a generated _encode silently dropped any
extension data set on the message. Add the array-walk after the field
loop and a decode_extension dispatch in the unknown-tag branch — the
inline path now matches runtime byte-for-byte.

bench/bench.lua parameterizes over a FIXTURES list so the baseline can
cover both hello.Person and a new proto2_basic.BenchPayload fixture
(required + group + extension + repeated). baseline.json restructured
under "schemas": [{schema, results}, …]; compare walks both. Fresh
numbers committed.

740/740 luatest cases pass; JIT gate 37/37; conformance 2806 binary+JSON
and 434 text-format, both 0 failures.
a9036f1d — Eugene Blikh 3 months ago
conformance: dispatch proto2 TestAllTypesProto2

Register the proto2 test message in MESSAGE_REGISTRY and the
Any/JSON registry. The harness now picks up 1312 previously-skipped
binary + JSON cases; whole suite jumps to 2805 ✓ / 0 failures (was
1493 ✓ with proto2 fully skipped).

Text-format suite still has 16 unexpected failures, all bound up
in proto2 group-field text syntax (CamelCase submessage label, no
mandatory `:` before `{`, extension-bracketed group form). Tackled
in the next commit.

The conformance unit test that pinned the proto2-skipped behavior
now targets an editions test type instead so the skip-path still
has a hard assertion.
5fa0fd00 — Eugene Blikh 3 months ago
codec: proto2 groups (SGROUP/EGROUP wire format)

Plugin: detect protoreflect.GroupKind, emit kind='group' in the
field descriptor and a SGROUP opening tag in inline codegen. The
group body bypasses the LEN-prefix path entirely — encoder emits
start-tag, body bytes, end-tag in three slots; decoder calls
pb.codec.decode_group(desc, buf, pos, field_id) which reads inner
tags until the matching EGROUP id.

Wire layer already understood SGROUP/EGROUP for skip_field; the new
decode_group reuses the per-tag dispatch from decode_message but
stops on EGROUP instead of end-of-buffer. Repeated groups bracket
each element with its own SGROUP/EGROUP pair.

Runtime parser now desugars `optional|required|repeated group Name
= id { body }` into (a) a nested message named Name and (b) a
synthetic field of kind=group whose lowercased name is `name` — so
source-parsed proto2 schemas behave the same as build-time codegen.

Text format renders groups under the submessage's capitalized name
(`SingleGroup { ... }`) rather than the lowercase field name,
matching mainline protoc's convention.

Adds a vendored, MessageSet-stripped test_messages_proto2.proto in
test/conformance/proto/ (header comment documents the patch).
10 new luatest cases (group_*) cover singular + repeated + text +
descriptor kind, across both codegen modes. 739 tests pass.
798e8db9 — Eugene Blikh 3 months ago
json,text: presence-tracked fields skip proto3 default elision

Both codecs gated default-value elision on `f.optional or f.oneof` —
the proto3 implicit-presence shape. That elides proto2 `required`
fields set to zero and any other presence-tracked descriptor that
doesn't carry the proto3 explicit-optional flag.

Extend the bypass to `f.required` so a Cardinality{r=0} survives
the round-trip through pb.json.encode / pb.text.encode. 4 new
luatest cases pin the rule.
4db0366a — Eugene Blikh 3 months ago
runtime: parser + dynamic accept proto2 sources

Parser now captures `required=true` and `default_value=…` from the
proto2 keywords (instead of dropping `required` silently and ignoring
field options). Dynamic descriptor builder reads `parsed.syntax`,
flips the repeated-scalar packing default for proto2, and calls
`codec.compile_writers/compile_readers` so the per-field
required-writer specialization actually fires — without that the
generic encode_field path silently elides missing required fields.

64-bit-int defaults are coerced into the appropriate cdata type
inside `coerce_default` so the codec's value-comparison rules
match what generated code emits.

Adds 7 dynamic-mode luatest cases including a static-vs-dynamic
byte-parity check. 725 tests pass.
4fbdb65d — Eugene Blikh 3 months ago
codegen: proto2 baseline — required, optional, custom defaults

Lifts the proto3-only syntax gate in the plugin and threads three
new field-descriptor attributes through codegen and the codec:

  * required=true   — fields declared with the proto2 `required` keyword.
                      Inline codegen and the runtime codec both error when
                      a required field is missing on encode (vs the silent
                      elide that proto3 implicit-presence fields get).
  * optional=true   — already wired for proto3 explicit `optional`; in
                      proto2 every singular field carries it via the
                      existing HasOptionalKeyword() check, giving presence
                      semantics without a separate emission path.
  * default_value=… — proto2 [default = X] from the field descriptor,
                      rendered as a Lua literal (cdata for 64-bit ints,
                      symbolic name for enums) so consumers can surface
                      it; the codec itself does not auto-materialize
                      defaults on decode, matching how proto3 absent
                      fields stay nil.

Packed-by-default already flips correctly because we ask
protoreflect's `IsPacked()`, which is syntax-aware.

Adds test/proto/proto2_basic.proto with 33 luatest cases covering
required validation, optional presence, custom defaults, the proto2
unpacked-by-default repeated rule, nested-required messages, and
full-vs-runtime mode parity. `just gen-proto2-tests` regenerates the
fixture into examples/expected/{full,runtime}/.

Out of scope: extensions, extend, group; conformance harness still
skips TestAllTypesProto2.
7676fdc2 — Eugene Blikh 3 months ago
codegen: preserve descriptor options as `options = { ... }`

Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.

The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.

The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.

UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
14e19819 — Eugene Blikh 3 months ago
codegen: bracket-quote Lua-keyword field names

The plugin emitted bare-identifier field names in three positions:
the `M.<Type>_fields` table key, the inline encoder's `v = t.<name>`
load, and the inline decoder's `result.<name> = ...` store. When a
proto field name collided with a Lua reserved word the generated
`*_pb.lua` failed to load with `'(' expected near '<keyword>'`.

The real-world hit is pprof's profile.proto, which declares
`repeated Function function = 5` — that breaks all three emit sites.

Route every user-named identifier through new luaTableKey /
luaFieldAccess helpers that bracket-quote reserved words. Covers
field_names + oneof descriptors, oneof presence pre-pass, inline
encode/decode field bodies (including repeated and map paths), and
optional has/clear accessors.
7e9e3e43 — Eugene Blikh 3 months ago
json: M.encode(desc, t, opts) for use_proto_names / emit_defaults / indent

Canonical protojson options were silently ignored. Plumb a single opts
table through encode_message via a CURRENT_ENCODE_OPTS state (mirroring
the decode-side CURRENT_OPTS):

  * use_proto_names         — emit snake_case field names (proto wire
                              names) instead of the spec-default
                              lowerCamelCase.
  * emit_defaults           — emit zero-valued implicit-presence scalars,
    (alias: always_emit_zero_value)  empty repeated lists, and empty
                              maps. Explicit-presence fields (optional /
                              oneof) and singular message fields remain
                              absent — matches protobuf-go's behavior.
  * indent                  — pretty-print with the given indent string;
                              empty arrays/objects stay compact.

Tests parameterize over both codegen modes (descriptor-table contract is
shared between full and runtime) plus a dedicated parity group that
asserts byte-identical JSON across modes for each option.
06b2978a — Eugene Blikh 3 months ago
codegen: resolve (tarantool.lua_package) via global type registry

The custom file option was being dropped silently: protoc encoded it
correctly into the FileDescriptorProto, but protobuf-go parked the
unknown extension in the message's unknown-fields tail because
E_LuaPackage was never registered with protoregistry.GlobalTypes. As a
result proto.GetExtension returned "" and every override the README
documented was a no-op — every caller fell through to the default
"<pkg>.<file>_pb" path resolution.

Register E_LuaPackage in init() and add a luatest regression that
asserts both the output path and the cross-file require strings honor
the option, parameterized over both codegen modes.
76140b72 — Eugene Blikh 3 months ago
json: emit unbroken base64 for bytes fields ({nowrap = true})

Canonical proto3 JSON expects RFC 4648 unwrapped base64, but Tarantool's
`digest.base64_encode` defaults to RFC 2045 MIME-style 76-char line
wrapping. Any `bytes` payload past ~57 bytes used to land in the JSON
string with an embedded `\n`, which breaks every spec-compliant
consumer (grpc-gateway, protojson, protobuf-go's JSON, ...).

Pass `{nowrap = true}` at the two encode sites: the per-field bytes
encoder and the `google.protobuf.Any` opaque-fallback `value` encoder.

Surfaced by tarantool-etcd's `TestJSONGatewayBytesUnwrapped` over Range
responses whose `value` is >=57 bytes; its reference grpc-gateway never
emits the wrapped form.
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
428ee1f6 — Eugene Blikh 3 months ago
codegen: propagate proto comments into generated _pb.lua

Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:

- Message / enum descriptions become `---` blocks above `---@class` /
  `---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
  matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
  lines inside the table literals so readers without an LSP still see
  the proto-side context.

Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
37a18dad — Eugene Blikh 3 months ago
json: strict validation pass closes the proto3 conformance suite

Six classes of relaxation that the proto3 JSON conformance corpus
flagged are now enforced. All as Recommended.* tests; combined with
the -0.0 codec fix this empties known_failures.txt and brings the
proto3 binary+JSON suite to 1493 ✓ / 1313 skipped / 0 expected
failures / 0 unexpected failures.

  1. Duplicate JSON keys. Tarantool's json.decode is hash-backed and
     silently collapses `{"foo":1,"foo":2}` to one entry. A small
     byte-walker `find_duplicate_json_keys` runs before json.decode,
     tracks per-object brace frames and key sets, errors on the
     second occurrence. Closes Recommended.FieldNameDuplicate.

  2. camelCase / snake_case aliases of the same proto field appearing
     side-by-side. Detected inside decode_message via a `field_seen`
     set keyed by proto-name; second hit errors. Closes
     FieldNameDuplicateDifferentCasing{1,2}.

  3. JSON null inside repeated arrays and map values. Previously
     silently dropped; now errors before decode_field_value. Closes
     RepeatedField{Message,Primitive}ElementIsNull and
     MapFieldValueIsNull.

  4. Unknown enum *names* (not integers). decode_enum used to return
     nil so callers silently dropped them; now raises by default and
     returns nil only when M.decode's `ignore_unknown_fields=true`
     opt is set. Conformance dispatch in cmd/conformance/core.lua
     forwards this flag when req.test_category ==
     JSON_IGNORE_UNKNOWN_PARSING_TEST. Closes
     RejectUnknownEnumStringValueIn{Optional,Repeated,Map} and the
     paired IgnoreUnknownEnumStringValueIn* tests.

  5. google.protobuf.NullValue JSON canonical form. The single enum
     value renders as the literal JSON `null` (not the string
     "NULL_VALUE"); decode accepts either, encode emits null. The
     decode_field_value null-handling path also treats a JSON null
     on a NullValue-typed field as "set" rather than "absent" so a
     oneof gets marked active. Closes
     NullValueInOtherOneof{New,Old}Format.Validator.

  6. FieldMask strict round-trip. Path validity is checked on both
     sides: the snake_case wire form rejects uppercase letters,
     consecutive underscores, trailing underscore, and underscore
     followed by anything other than a lowercase letter — these
     break the snake↔camel round-trip. The JSON form rejects any
     underscore in the input (must be lowerCamelCase). Closes
     FieldMask{TooManyUnderscore,PathsDontRoundTrip,
     NumbersDontRoundTrip}.JsonOutput and JsonInput.FieldMaskInvalidCharacter.

The pre-existing "drop unknown enum strings" unit regressions in
test/conformance_test.lua were inverted to assert the new error
shape. New strict-validation regressions in test/json_test.lua pin
all six categories so they don't regress; the `json.strict` group
runs across both codegen modes via the shared descriptor table.
f5c5ee6c — Eugene Blikh 3 months ago
codec: preserve -0.0 for proto3 float/double scalars

Proto3 default-elision dropped any singular float/double whose value
compared equal to 0 — `-0.0 == 0.0` in IEEE so the `if v ~= 0` guard
silently elided negative zero. The wire bytes for -0 differ from +0
and the TextFormatInput conformance corpus pins that -0 must survive
a round-trip; the failure surfaced as 10 unexpected text-suite
regressions covering FloatFieldNegativeZero (3 spellings × 2 outputs)
and Neg{Float,Double}FieldLargeNegativeExponentParsesAsNegZero
(2 types × 2 outputs).

Sign-bit guard via `1/v == math.huge` (positive zero yields +inf,
negative zero yields -inf). Applied in three layers:
  * runtime/pb/codec.lua: is_default_scalar + the specialized
    monomorphic writer for non-optional numeric scalars
  * runtime/pb/text.lua: is_proto3_default (text encoder elision)
  * inline codegen: scalarNotDefaultExpr emits the guard for
    float/double fields in full-mode `_encode` functions

Proto3 text-format conformance now sits at 416 ✓ / 18 skipped / 0
expected failures; binary+JSON holds at 1478 ✓. Unit-test regressions
cover both directions (preserved on encode + decode round-trip, +0
still elided) and use a runtime-computed -0.0 sentinel because LuaJIT
can constant-fold the literal `-0.0` to a sign-less zero in some
load paths.
7ec8f2b4 — Eugene Blikh 3 months ago
text: add pb.text.decode and wire it into conformance dispatch

Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.

Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.

cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.

Closes the text-conformance-output branch.
343e4ba4 — Eugene Blikh 3 months ago
text: render captured unknown fields, tolerate SGROUP in skip

Two changes close the proto3 text-format conformance suite:

  1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
     inner tags until a matching EGROUP, with field_id checked against
     the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
     full-mode `_pb.lua`) now pass the tag's field_id so groups inside
     unknown-field skips don't error.

  2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
     `opts.print_unknown_fields=true` and emits each entry in
     TextFormat numeric-field form:
       VARINT  -> "<id>: <uint64>"
       I64/I32 -> "<id>: 0x<hex>"
       LEN     -> speculative "<id> { <recurse> }"; rolls back to
                  byte-string form if the inner bytes don't parse as a
                  sub-message
       SGROUP  -> "<id> { <recurse> }" through matching EGROUP

`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.

Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
b1d9c758 — Eugene Blikh 3 months ago
test: pin unknown-fields text-format conformance regressions

Adds eight tests under conformance.core mirroring the
Recommended.Proto3.ProtobufInput.*UnknownFields_*.TextFormatOutput tests
in the Google harness. Each sends the byte-exact upstream payload (field
IDs 1001..1011 from UnknownToTestAllTypes) through cmd/conformance/core
and pins current output, with the target assertion + needed runtime fix
documented in each failure message.

Two blockers surface:
  1. wire.skip_field rejects SGROUP/EGROUP (wire 3/4), so the four
     Group/Repeated tests fail at decode.
  2. pb.text.encode doesn't walk _unknown_fields, so the four *_Print
     tests serialize empty.

Lets us iterate on those fixes locally without the Docker round-trip.
Next