json: strict validation pass closes the proto3 conformance suite
Six classes of relaxation that the proto3 JSON conformance corpus
flagged are now enforced. All as Recommended.* tests; combined with
the -0.0 codec fix this empties known_failures.txt and brings the
proto3 binary+JSON suite to 1493 ✓ / 1313 skipped / 0 expected
failures / 0 unexpected failures.
1. Duplicate JSON keys. Tarantool's json.decode is hash-backed and
silently collapses `{"foo":1,"foo":2}` to one entry. A small
byte-walker `find_duplicate_json_keys` runs before json.decode,
tracks per-object brace frames and key sets, errors on the
second occurrence. Closes Recommended.FieldNameDuplicate.
2. camelCase / snake_case aliases of the same proto field appearing
side-by-side. Detected inside decode_message via a `field_seen`
set keyed by proto-name; second hit errors. Closes
FieldNameDuplicateDifferentCasing{1,2}.
3. JSON null inside repeated arrays and map values. Previously
silently dropped; now errors before decode_field_value. Closes
RepeatedField{Message,Primitive}ElementIsNull and
MapFieldValueIsNull.
4. Unknown enum *names* (not integers). decode_enum used to return
nil so callers silently dropped them; now raises by default and
returns nil only when M.decode's `ignore_unknown_fields=true`
opt is set. Conformance dispatch in cmd/conformance/core.lua
forwards this flag when req.test_category ==
JSON_IGNORE_UNKNOWN_PARSING_TEST. Closes
RejectUnknownEnumStringValueIn{Optional,Repeated,Map} and the
paired IgnoreUnknownEnumStringValueIn* tests.
5. google.protobuf.NullValue JSON canonical form. The single enum
value renders as the literal JSON `null` (not the string
"NULL_VALUE"); decode accepts either, encode emits null. The
decode_field_value null-handling path also treats a JSON null
on a NullValue-typed field as "set" rather than "absent" so a
oneof gets marked active. Closes
NullValueInOtherOneof{New,Old}Format.Validator.
6. FieldMask strict round-trip. Path validity is checked on both
sides: the snake_case wire form rejects uppercase letters,
consecutive underscores, trailing underscore, and underscore
followed by anything other than a lowercase letter — these
break the snake↔camel round-trip. The JSON form rejects any
underscore in the input (must be lowerCamelCase). Closes
FieldMask{TooManyUnderscore,PathsDontRoundTrip,
NumbersDontRoundTrip}.JsonOutput and JsonInput.FieldMaskInvalidCharacter.
The pre-existing "drop unknown enum strings" unit regressions in
test/conformance_test.lua were inverted to assert the new error
shape. New strict-validation regressions in test/json_test.lua pin
all six categories so they don't regress; the `json.strict` group
runs across both codegen modes via the shared descriptor table.
text: add pb.text.decode and wire it into conformance dispatch
Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.
Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.
cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.
Closes the text-conformance-output branch.
text: render captured unknown fields, tolerate SGROUP in skip
Two changes close the proto3 text-format conformance suite:
1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
inner tags until a matching EGROUP, with field_id checked against
the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
full-mode `_pb.lua`) now pass the tag's field_id so groups inside
unknown-field skips don't error.
2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
`opts.print_unknown_fields=true` and emits each entry in
TextFormat numeric-field form:
VARINT -> "<id>: <uint64>"
I64/I32 -> "<id>: 0x<hex>"
LEN -> speculative "<id> { <recurse> }"; rolls back to
byte-string form if the inner bytes don't parse as a
sub-message
SGROUP -> "<id> { <recurse> }" through matching EGROUP
`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.
Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
text: wire pb.text into conformance runner TEXT_FORMAT output
The runner short-circuited every TEXT_FORMAT request to `skipped`, even
though pb.text.encode has been encode-capable since M7. Plug it into
cmd/conformance/core.lua so protobuf/JSON input → text output exercises
the existing encoder end-to-end. Text-format input remains deferred
(pb.text is encode-only).
Text-format suite: 0 ✓ / 430 skipped / 4 expected fails →
2 ✓ / 426 skipped / 6 expected fails (Scalar/Message
*_Print added — unknown-field rendering still missing).
json: enable conformance JSON output + close 459 tests
Three coupled changes that turn the JSON output path on for the
conformance harness:
1. string_to_int64 accepts cdata: Tarantool's json.decode parses raw
JSON integer literals outside double range as int64_t/uint64_t cdata,
not Lua numbers. The decoder errored "expected JSON string or number
for int64" on any unquoted 64-bit value (Int64FieldMaxValueNotQuoted
et al). cdata is now cast through directly, preserving precision.
2. encode_message marks output as a map: an empty proto3 message
serialized as `[]` because Tarantool's json defaults empty tables to
array shape. jsoncpp's strict comparator threw Json::LogicError and
aborted the whole suite. Setting __serialize='map' on the output
gives `{}` and unblocks all JsonOutput tests.
3. PB_CONFORMANCE_SKIP_JSON gate is opt-in by default. core.lua now
matches exactly "1" (so docker -e VAR= disables it), and the
Dockerfile no longer hard-codes "=1" — JSON output runs end-to-end
for everyone unless they re-enable the gate.
Conformance moves from 930 / 1869 / 11 to 1389 / 1313 / 79
(successes / skipped / expected fails). The 75 new expected fails
are canonical-form edge cases (Duration formatting sign handling,
Timestamp out-of-range rejection, double precision digits, NaN
canonicalization, JSON-input strict rejection) — left for a follow-up.
Drops 7 entries from test/conformance/known_failures.txt that this
change closes; adds 75 newly-visible ones.
wkt: auto-register WKT descriptors so Any JSON decode resolves @type
pb.wkt exported Timestamp/Duration/Empty/FieldMask/Any/Struct/Value/
ListValue and the nine Wrapper descriptors, but the REGISTRY they live in
was empty until callers manually invoked pb.register. json_to_any looked
up @type in that empty registry, fell through to the opaque base64
fallback, and errored on every Any-of-WKT JSON payload.
Iterates M for every *_descriptor entry at module load and self-registers
it. Also calls pb.register on TestAllTypesProto3 in the conformance runner
so Any tests that embed the user message type also resolve.
Drops 10 entries from test/conformance/known_failures.txt.
conformance: local Docker pipeline + cdata int64 map dedup
Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.
Six bugs surfaced and got fixed on the way to green:
1. conformance_test_runner uses execv (not execvp): bare `tarantool`
hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
2. The harness strips LUA_PATH from the child: the runner now
self-bootstraps package.path from debug.getinfo(1, 'S').source.
3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
full BUFSIZ before returning, deadlocking against the parent.
setvbuf('no') on stdin/stdout.
4. v34.1 fetches libjsoncpp via CMake FetchContent under
_deps/jsoncpp-build/...; the runtime image now COPYs the matching
.so* and runs ldconfig.
5. The harness's strict jsoncpp comparator crashes on our currently-
imperfect JSON output (enum numerics, map<K,V> shape, oneof
object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
set in the container ENV; host-side `make test` still exercises
the full JSON path.
6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
key map entries (per proto3's "last value wins" semantics) split
across hash buckets even though __eq matches. Codec walks the
map once on insert to find a canonical key, gated by a
precomputed `f.key_dedup` flag so the dedup only fires for
int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
same `for _k in pairs(map) do` walk only when the static key
kind is 64-bit, so string/int32-keyed map decode stays
JIT-traceable.
Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:
- Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
- Text-format suite: 0 ✓ / 430 skipped / 4 expected fails
403/403 luatest green, 19/19 jit-trace gate green.
Initial commit: protoc-gen-tarantool plugin + pb runtime
A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.
Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.