~bigbes/tarantool

tarantool-protobuf

ref: b1d9c7581bcc6b4f299155746dc04acc2330e78c tarantool-protobuf/test/conformance_test.lua -rw-r--r-- 62.3 KiB
b1d9c758 — Eugene Blikh 3 months ago
test: pin unknown-fields text-format conformance regressions

Adds eight tests under conformance.core mirroring the
Recommended.Proto3.ProtobufInput.*UnknownFields_*.TextFormatOutput tests
in the Google harness. Each sends the byte-exact upstream payload (field
IDs 1001..1011 from UnknownToTestAllTypes) through cmd/conformance/core
and pins current output, with the target assertion + needed runtime fix
documented in each failure message.

Two blockers surface:
  1. wire.skip_field rejects SGROUP/EGROUP (wire 3/4), so the four
     Group/Repeated tests fail at decode.
  2. pb.text.encode doesn't walk _unknown_fields, so the four *_Print
     tests serialize empty.

Lets us iterate on those fixes locally without the Docker round-trip.
39233ed5 — Eugene Blikh 3 months ago
text: wire pb.text into conformance runner TEXT_FORMAT output

The runner short-circuited every TEXT_FORMAT request to `skipped`, even
though pb.text.encode has been encode-capable since M7. Plug it into
cmd/conformance/core.lua so protobuf/JSON input → text output exercises
the existing encoder end-to-end. Text-format input remains deferred
(pb.text is encode-only).

Text-format suite: 0 ✓ / 430 skipped / 4 expected fails →
                   2 ✓ / 426 skipped / 6 expected fails (Scalar/Message
*_Print added — unknown-field rendering still missing).
e53089dd — Eugene Blikh 3 months ago
json: strict proto3 JSON conformance — close all Required failures

The conformance suite's Required JSON failures collapse to zero with
this pass. Local tests grow from 456 to 497 to pin every fix.

Decode side — `runtime/pb/json.lua`:
  * Strict scalar validators per type. Numeric strings must match the
    JSON number grammar (no leading whitespace, no partial numerics);
    out-of-range values, NaN/Inf surrogates from JSON literals, and
    type mismatches all become parse_error.
  * Quoted exponential ints ("1e5" -> 100000) are accepted via the
    JSON-number grammar path, matching Int32FieldQuotedExponentialValue.
  * Timestamp parser: strict RFC 3339 (uppercase T/Z, ±HH:MM offset,
    ≤9 frac digits, range check). Output uses a portable Hinnant-style
    epoch_to_ymdhms so year 0001 zero-pads correctly — glibc's POSIX
    %Y emits "1" for that year, breaking round-trip.
  * Duration parser/formatter: mandatory `s` suffix, ±10000-year range,
    sign-matching nanos, 0/3/6/9-digit fractional output.
  * Any: WKT-aware nesting under "value"; empty Any -> {}; Empty WKT
    inside Any omits "value" (reference parser rejects {"value":{}});
    @type URL without `/` rejected; empty @type with sibling fields
    rejected.
  * Reject NaN/Infinity in google.protobuf.Value.number_value (no JSON
    literal for these).
  * Duplicate oneof branches rejected; a null oneof branch does NOT
    count as set, so a sibling non-null branch is unambiguous.
  * Top-level JSON null rejected for messages; preserved for Value.
  * Repeated/map values must be JSON array/object (not bare scalar).

Encode side — hand-rolled JSON emitter:
  Tarantool's `json.encode` uses a fixed global precision so doubles
  like 0.1 don't round-trip and we can't change it per-value without
  polluting other users. Replace with a minimal emitter that picks the
  shortest-round-tripping precision (15 -> 16 -> 17) per double and
  handles NaN/Inf as quoted sentinel strings.

`runtime/pb/wkt.lua`: `timestamp_decode` now keeps invalid Timestamps as
a raw {seconds, nanos} table when `datetime.new` rejects them (negative
nanos, year > 9999, ...) so the JSON encoder can produce
serialize_error rather than the binary decoder raising parse_error.
Required by the Timestamp conformance suite.

`test/conformance_test.lua`: 41 new regression tests grouped under
Fixes 12-19, pinning every code path touched. Strict scalar
validators (one per rejection shape per type), Timestamp/Duration
strict parsing and canonical output, Any WKT/non-WKT/Empty handling,
Value NaN/Inf rejection, ValueAcceptNull round-trip, LuaJIT
NaN-boxing collision cases (0x7FFBCBA987654321, all-ones), shortest
double round-trip, oneof-null semantics.

`test/conformance/known_failures.txt`: refreshed. 15 Recommended-only
failures remain (FieldMask round-trip quirks, duplicate-field-name
detection, unknown-enum-string rejection, null-element-in-list,
NullValue oneof validator).
b1273f1b — Eugene Blikh 3 months ago
json: canonical lowerCamelCase + NullValue WKT descriptor

Two unrelated JSON-decoder gaps captured under the conformance triage:

1. to_camel's gsub pattern '_(%w)' didn't match consecutive underscores
   and didn't drop trailing underscores, so proto names like
   __field_name13 / field__name4_ / field_name17__ generated JSON keys
   that didn't match what protoc produces. The fix strips trailing _+
   and collapses '_+%w' to a capitalized letter; a leading underscore
   thus capitalizes the next character, matching the spec.

2. pb.wkt didn't export a descriptor for google.protobuf.NullValue, so
   any enum field whose type is NullValue (oneof_null_value, or the
   implicit one inside Value) crashed decode_enum with "attempt to
   index a nil value." Adds a minimal {by_name, by_value} descriptor.

Drops 3 entries from test/conformance/known_failures.txt
(FieldNameInSnakeCase, FieldNameWithDoubleUnderscores,
NullValueInOtherOneofOldFormat). Adds 4 regression tests covering
double underscore, leading underscore, trailing underscore, and the
NullValue enum decode path.
96ed328e — Eugene Blikh 3 months ago
json: treat null fields as absent (and Value's null as a real value)

Per the proto3 JSON spec, a null on any field means "use the field's
default" — encoded as missing — with the lone exception of
google.protobuf.Value, where JSON null is itself a Value carrying
NullValue.NULL_VALUE.

Three coupled bugs surfaced together:

1. decode_field_value used to fall through with v = box.NULL, leaving a
   useless box.NULL sitting in the result table for scalars. Now it
   returns nil for non-Value fields, PB_NULL for Value fields.

2. decode_message's repeated and map branches called `#jv` and
   `pairs(jv)` unconditionally; a JSON-null on either type crashed with
   "attempt to get length of 'void *'". Now both branches short-circuit
   when jv is box.NULL.

3. The nil-skip checks in the decode loop (`if dv ~= nil`) and in the
   codec / inline message encoders (`if v == nil then return end`)
   evaluated TRUE on box.NULL because Tarantool's cdata __eq aliases
   it to nil. Decode now uses rawequal(dv, nil); encode special-cases
   message kind by also accepting cdata, so the Value field's
   box.NULL sentinel survives all the way through to value_encode.

Drops 3 entries from test/conformance/known_failures.txt
(AllFieldAcceptNull, WrapperTypesWithNullValue, ValueAcceptNull).
Adds 5 regression tests covering scalar / repeated / map / wrapper
null treatment and the Value-NULL_VALUE exception.
43f7b869 — Eugene Blikh 3 months ago
wire: reject invalid UTF-8 in proto3 string fields

Per the proto3 spec, a string field's bytes must form valid UTF-8.
decode_string was aliased to decode_len, so any byte sequence was
accepted and round-tripped. Adds a pure-Lua RFC 3629 validator —
is_valid_utf8 — and routes M.decode_string through it. The bytes
type keeps the raw decode_len path so binary payloads still pass.

The validator covers stray continuation bytes, truncated sequences,
overlong encodings, UTF-16 surrogates (U+D800..U+DFFF), and code
points above U+10FFFF.

Codec, lazy, dynamic, and the generated inline code all consume the
same M.decode_string, so singular / repeated / oneof / map-key /
map-value string fields are all covered.

Drops 5 entries from test/conformance/known_failures.txt
(RejectInvalidUtf8.String.*) and adds 7 regression tests covering
each invalid form plus a positive multi-byte string round-trip and
a bytes-field control.
2ed0f4b5 — Eugene Blikh 3 months ago
wire: validate field number and minimal encoding in decode_tag

Three checks added on the multi-byte path:
1. Field number must be > 0 (IllegalZeroFieldNum_Case_0/1/3).
2. Field number must fit in 29 bits per the protobuf spec
   (BadTag_FieldNumberSlightlyTooHigh, BadTag_FieldNumberTooHigh).
3. Tag varint must be minimally encoded — a trailing 0 byte with more
   than one byte read is overlong (BadTag_OverlongVarint).

The field-number check runs on bit ops over the uint64 cdata returned
by decode_varint rather than after tonumber, otherwise field numbers
above 2^32 alias into the valid range (e.g., fn=2^31+1 was being
recovered as fn=1).

The single-byte fast path picks up the field-zero check directly via
the b >> 3 == 0 condition.

Drops 6 entries from test/conformance/known_failures.txt and adds 4
regression tests pinning each rejection path.
849b7662 — Eugene Blikh 3 months ago
codec: recursively merge repeated singular-message wire entries

Per proto3 spec, when the same singular message field (including a oneof
branch) appears twice on the wire, the two values must merge: scalar
fields last-wins, repeated fields concatenate, sub-messages merge
recursively, maps last-wins per key. The previous reader replaced the
prev value wholesale for oneof branches and overwrote repeated/nested
fields with `prev[k] = v` even outside oneofs, losing data unique to
the first occurrence.

Adds wire.codec.merge_message(desc, prev, decoded), a descriptor-driven
recursive merge, and routes both codec.lua's reader and the inline-mode
codegen through it. WKT message fields (custom decode) keep the replace
behavior because their decoded value is not a generic Lua table.

Exposes pb.codec to the generated inline code so the helper is reachable
without a per-call require.

Drops 3 entries from test/conformance/known_failures.txt:
ValidDataOneof.MESSAGE.Merge, ValidDataOneofBinary.MESSAGE.Merge,
RepeatedScalarMessageMerge. Adds 5 regression tests covering scalar
last-wins, oneof merge, recursive sub-message merge, repeated-in-
submessage concat, and oneof sibling clearing after merge.
2180b76a — Eugene Blikh 3 months ago
test: regression tests for each conformance fix in this branch

Pins each fix locally so the bug is caught without the Docker conformance
runner. Grouped by fix; multiple tests per fix because each fix touches
several code paths (per scalar type, per shape).

Fix 1 — varint 32-bit truncation (0c13fd7), 7 tests:
  int32 high-bits → 0, int32 sign-extend, uint32 low-32, sint32 zigzag
  after truncation, enum singular, packed-repeated int32 element-wise,
  repeated-scalar-selects-last after truncation.

Fix 2 — wire types 6/7 reject (7442ea2), 5 tests:
  wt=6 known, wt=7 known, wt=6 unknown, wt=6 mid-stream after a valid
  field, positive control that wts 0/2/5 still parse.

Fix 3 — WKT registry auto-register (50ed9b6), 6 tests:
  pb.lookup resolves all 11 WKT descriptors plus type-URL form;
  Any/Timestamp, Any/Duration, Any/Int32Value, Any/Struct, Any/user-type
  all round-trip via JSON input.

Fix 4 — skip_field bounds checks (1712192), 5 tests:
  truncated I64, truncated I32, truncated LEN fast path, truncated LEN
  multi-byte length, complete unknown round-trips intact.

Fix 5 — drop unknown enum names in JSON (a2f1209), 5 tests:
  unknown name elided in singular, dropped from repeated, dropped from
  map value, numeric unknown preserved, numeric-string unknown preserved.
784dea4d — Eugene Blikh 3 months ago
Initial commit: protoc-gen-tarantool plugin + pb runtime

A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.

Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.