~bigbes/tarantool

tarantool-protobuf

ref: c85386b1692ea4af5ad52aa383a49576f7f50d2e tarantool-protobuf/runtime/pb/wkt.lua -rw-r--r-- 22.3 KiB
7ec8f2b4 — Eugene Blikh 3 months ago
text: add pb.text.decode and wire it into conformance dispatch

Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.

Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.

cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.

Closes the text-conformance-output branch.
2ffc6ebd — Eugene Blikh 3 months ago
wkt: split length-prefix emission to drop per-field concat

Apply the split-emit pattern from ac9b9c1 (inline.go + codec.lua) to the
WKT hand-rolled encoders. Replace `wire.encode_len(body)` — which
allocates `varint(#body) .. body` — with three out slots (tag, varint
length, body) wherever the result feeds a `table.concat(out)`
accumulator. For StringValue/BytesValue (whose encoders return a single
string and can't be split), inline encode_len's chain so LuaJIT folds
tag + varint + body into one multi-concat instead of two sequential
concats. Same final wire bytes.

Touched sites: struct_encode (split the outer entry wrap, inline the
inner key/value chain), list_encode, any_encode (type_url, value),
fieldmask_encode, and the WRAPPERS LEN-typed encoders.

The struct_encode inner entry stays a single chain rather than fanning
out into more `out` slots — eight slots per Struct entry regressed
B/op and gave no encode win on the wkt-event shape; three slots match
the codec.lua nested-message pattern and land the speedup.

bench/shapes (wkt-event encode, median across 4 runs):
  full:    125895 -> 129000 msgs/s (+2.5%)
  runtime: 115231 -> 120000 msgs/s (+4%)

make test 509/509, make jit-trace 23/23, docker conformance binary
1478/0 (unexpected), text suite clean.
343e4ba4 — Eugene Blikh 3 months ago
text: render captured unknown fields, tolerate SGROUP in skip

Two changes close the proto3 text-format conformance suite:

  1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
     inner tags until a matching EGROUP, with field_id checked against
     the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
     full-mode `_pb.lua`) now pass the tag's field_id so groups inside
     unknown-field skips don't error.

  2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
     `opts.print_unknown_fields=true` and emits each entry in
     TextFormat numeric-field form:
       VARINT  -> "<id>: <uint64>"
       I64/I32 -> "<id>: 0x<hex>"
       LEN     -> speculative "<id> { <recurse> }"; rolls back to
                  byte-string form if the inner bytes don't parse as a
                  sub-message
       SGROUP  -> "<id> { <recurse> }" through matching EGROUP

`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.

Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
e53089dd — Eugene Blikh 3 months ago
json: strict proto3 JSON conformance — close all Required failures

The conformance suite's Required JSON failures collapse to zero with
this pass. Local tests grow from 456 to 497 to pin every fix.

Decode side — `runtime/pb/json.lua`:
  * Strict scalar validators per type. Numeric strings must match the
    JSON number grammar (no leading whitespace, no partial numerics);
    out-of-range values, NaN/Inf surrogates from JSON literals, and
    type mismatches all become parse_error.
  * Quoted exponential ints ("1e5" -> 100000) are accepted via the
    JSON-number grammar path, matching Int32FieldQuotedExponentialValue.
  * Timestamp parser: strict RFC 3339 (uppercase T/Z, ±HH:MM offset,
    ≤9 frac digits, range check). Output uses a portable Hinnant-style
    epoch_to_ymdhms so year 0001 zero-pads correctly — glibc's POSIX
    %Y emits "1" for that year, breaking round-trip.
  * Duration parser/formatter: mandatory `s` suffix, ±10000-year range,
    sign-matching nanos, 0/3/6/9-digit fractional output.
  * Any: WKT-aware nesting under "value"; empty Any -> {}; Empty WKT
    inside Any omits "value" (reference parser rejects {"value":{}});
    @type URL without `/` rejected; empty @type with sibling fields
    rejected.
  * Reject NaN/Infinity in google.protobuf.Value.number_value (no JSON
    literal for these).
  * Duplicate oneof branches rejected; a null oneof branch does NOT
    count as set, so a sibling non-null branch is unambiguous.
  * Top-level JSON null rejected for messages; preserved for Value.
  * Repeated/map values must be JSON array/object (not bare scalar).

Encode side — hand-rolled JSON emitter:
  Tarantool's `json.encode` uses a fixed global precision so doubles
  like 0.1 don't round-trip and we can't change it per-value without
  polluting other users. Replace with a minimal emitter that picks the
  shortest-round-tripping precision (15 -> 16 -> 17) per double and
  handles NaN/Inf as quoted sentinel strings.

`runtime/pb/wkt.lua`: `timestamp_decode` now keeps invalid Timestamps as
a raw {seconds, nanos} table when `datetime.new` rejects them (negative
nanos, year > 9999, ...) so the JSON encoder can produce
serialize_error rather than the binary decoder raising parse_error.
Required by the Timestamp conformance suite.

`test/conformance_test.lua`: 41 new regression tests grouped under
Fixes 12-19, pinning every code path touched. Strict scalar
validators (one per rejection shape per type), Timestamp/Duration
strict parsing and canonical output, Any WKT/non-WKT/Empty handling,
Value NaN/Inf rejection, ValueAcceptNull round-trip, LuaJIT
NaN-boxing collision cases (0x7FFBCBA987654321, all-ones), shortest
double round-trip, oneof-null semantics.

`test/conformance/known_failures.txt`: refreshed. 15 Recommended-only
failures remain (FieldMask round-trip quirks, duplicate-field-name
detection, unknown-enum-string rejection, null-element-in-list,
NullValue oneof validator).
b1273f1b — Eugene Blikh 3 months ago
json: canonical lowerCamelCase + NullValue WKT descriptor

Two unrelated JSON-decoder gaps captured under the conformance triage:

1. to_camel's gsub pattern '_(%w)' didn't match consecutive underscores
   and didn't drop trailing underscores, so proto names like
   __field_name13 / field__name4_ / field_name17__ generated JSON keys
   that didn't match what protoc produces. The fix strips trailing _+
   and collapses '_+%w' to a capitalized letter; a leading underscore
   thus capitalizes the next character, matching the spec.

2. pb.wkt didn't export a descriptor for google.protobuf.NullValue, so
   any enum field whose type is NullValue (oneof_null_value, or the
   implicit one inside Value) crashed decode_enum with "attempt to
   index a nil value." Adds a minimal {by_name, by_value} descriptor.

Drops 3 entries from test/conformance/known_failures.txt
(FieldNameInSnakeCase, FieldNameWithDoubleUnderscores,
NullValueInOtherOneofOldFormat). Adds 4 regression tests covering
double underscore, leading underscore, trailing underscore, and the
NullValue enum decode path.
50ed9b68 — Eugene Blikh 3 months ago
wkt: auto-register WKT descriptors so Any JSON decode resolves @type

pb.wkt exported Timestamp/Duration/Empty/FieldMask/Any/Struct/Value/
ListValue and the nine Wrapper descriptors, but the REGISTRY they live in
was empty until callers manually invoked pb.register. json_to_any looked
up @type in that empty registry, fell through to the opaque base64
fallback, and errored on every Any-of-WKT JSON payload.

Iterates M for every *_descriptor entry at module load and self-registers
it. Also calls pb.register on TestAllTypesProto3 in the conformance runner
so Any tests that embed the user message type also resolve.

Drops 10 entries from test/conformance/known_failures.txt.
784dea4d — Eugene Blikh 3 months ago
Initial commit: protoc-gen-tarantool plugin + pb runtime

A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.

Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.