test: regression tests for each conformance fix in this branch
Pins each fix locally so the bug is caught without the Docker conformance
runner. Grouped by fix; multiple tests per fix because each fix touches
several code paths (per scalar type, per shape).
Fix 1 — varint 32-bit truncation (0c13fd7), 7 tests:
int32 high-bits → 0, int32 sign-extend, uint32 low-32, sint32 zigzag
after truncation, enum singular, packed-repeated int32 element-wise,
repeated-scalar-selects-last after truncation.
Fix 2 — wire types 6/7 reject (7442ea2), 5 tests:
wt=6 known, wt=7 known, wt=6 unknown, wt=6 mid-stream after a valid
field, positive control that wts 0/2/5 still parse.
Fix 3 — WKT registry auto-register (50ed9b6), 6 tests:
pb.lookup resolves all 11 WKT descriptors plus type-URL form;
Any/Timestamp, Any/Duration, Any/Int32Value, Any/Struct, Any/user-type
all round-trip via JSON input.
Fix 4 — skip_field bounds checks (1712192), 5 tests:
truncated I64, truncated I32, truncated LEN fast path, truncated LEN
multi-byte length, complete unknown round-trips intact.
Fix 5 — drop unknown enum names in JSON (a2f1209), 5 tests:
unknown name elided in singular, dropped from repeated, dropped from
map value, numeric unknown preserved, numeric-string unknown preserved.
json: drop unrecognized enum string names per proto3 spec
decode_enum returned the raw input string for unknown enum names,
which then propagated through the codec and errored at encode time
with "unknown enum value '...'". For repeated and map fields the
spec requires *dropping* the element (not substituting 0), so the
encoded output should be shorter than the input.
decode_enum now returns nil for unrecognized names. The optional /
repeated / map call sites in decode_message skip nil values: optional
leaves the field unset (encoded as default), repeated compacts the
array, map omits the entry. Numeric and known-name inputs are
unchanged.
Drops 5 entries from test/conformance/known_failures.txt:
IgnoreUnknownEnumStringValueIn{Optional,Repeated,RepeatedPart,
MapPart,MapValue}.
wire: bounds-check skip_field for I32/I64/LEN truncation
skip_field advanced pos by a fixed or length-prefixed amount with no
check against #buf, so a truncated unknown field was silently consumed:
the outer decode loop's `while pos <= len` exited without raising,
making the parser accept payloads it should have rejected.
Validates that the new position never exceeds #buf+1 for WIRE_I64,
WIRE_I32, and both WIRE_LEN fast and slow paths. WIRE_VARINT already
errored correctly via decode_varint's per-byte check.
Drops 15 entries from test/conformance/known_failures.txt
(PrematureEofBeforeUnknownValue.*, PrematureEofInsideUnknownValue.*,
PrematureEofInDelimitedDataForUnknownValue.*).
wkt: auto-register WKT descriptors so Any JSON decode resolves @type
pb.wkt exported Timestamp/Duration/Empty/FieldMask/Any/Struct/Value/
ListValue and the nine Wrapper descriptors, but the REGISTRY they live in
was empty until callers manually invoked pb.register. json_to_any looked
up @type in that empty registry, fell through to the opaque base64
fallback, and errored on every Any-of-WKT JSON payload.
Iterates M for every *_descriptor entry at module load and self-registers
it. Also calls pb.register on TestAllTypesProto3 in the conformance runner
so Any tests that embed the user message type also resolve.
Drops 10 entries from test/conformance/known_failures.txt.
wire: reject illegal wire types 6/7 in decode_tag
Wire types 6 and 7 are never assigned in the protobuf wire format; only
0 (VARINT), 1 (I64), 2 (LEN), 3/4 (SGROUP/EGROUP, proto2-only), and 5
(I32) are valid. skip_field already rejected them for unknown fields,
but the decode loop in codec.lua and inline.go dispatched to a typed
reader whenever the field ID was known — the reader ignored wt and
called the value-specific decoder anyway, accepting bytes that should
have been a parse error.
Moves the check into decode_tag so every parser path — codec, generated
inline code, lazy, and map sub-fields — rejects 6/7 uniformly.
Drops 24 entries from test/conformance/known_failures.txt.
wire: truncate varints to 32 bits in int32/uint32/sint32/enum decode
decode_int32/uint32/sint32 returned full uint64 values when the wire input
carried bits above bit 31, corrupting re-encode on 51 conformance tests that
exercise overlong or over-range varints. Per the proto3 spec the decoder
must keep only the low 32 bits (and sign-extend for signed types).
Adds wire.varint_to_int32 / varint_to_uint32 and routes every enum-varint
decode site through them: wire.lua typed decoders, codec.lua (5 enum
sites), lazy.lua (3 sites), and the generated code via inline.go (3 sites).
Drops 51 entries from test/conformance/known_failures.txt.
conformance: local Docker pipeline + cdata int64 map dedup
Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.
Six bugs surfaced and got fixed on the way to green:
1. conformance_test_runner uses execv (not execvp): bare `tarantool`
hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
2. The harness strips LUA_PATH from the child: the runner now
self-bootstraps package.path from debug.getinfo(1, 'S').source.
3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
full BUFSIZ before returning, deadlocking against the parent.
setvbuf('no') on stdin/stdout.
4. v34.1 fetches libjsoncpp via CMake FetchContent under
_deps/jsoncpp-build/...; the runtime image now COPYs the matching
.so* and runs ldconfig.
5. The harness's strict jsoncpp comparator crashes on our currently-
imperfect JSON output (enum numerics, map<K,V> shape, oneof
object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
set in the container ENV; host-side `make test` still exercises
the full JSON path.
6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
key map entries (per proto3's "last value wins" semantics) split
across hash buckets even though __eq matches. Codec walks the
map once on insert to find a canonical key, gated by a
precomputed `f.key_dedup` flag so the dedup only fires for
int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
same `for _k in pairs(map) do` walk only when the static key
kind is 64-bit, so string/int32-keyed map decode stays
JIT-traceable.
Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:
- Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
- Text-format suite: 0 ✓ / 430 skipped / 4 expected fails
403/403 luatest green, 19/19 jit-trace gate green.
codegen: protoc-gen-tarantool-doc — Markdown reference plugin
Sibling Go plugin under cmd/protoc-gen-tarantool-doc that emits one
Markdown file per input .proto. Sections (omitted when empty):
- Header (path, package, imports)
- Messages (per-message description + field table:
# | Field | Type | Label | Description)
- Enums (value table)
- Services (method table with unary/client/server/bidi label)
Field type cells render scalar names, full type names for message/enum
references, and `map<K, V>` for maps. Synthetic map-entry messages are
skipped. Leading comments preserved via SourceCodeInfo (squashed to a
single line inside table cells).
Built via `make build-doc`; sample output committed at
examples/docs/hello.md via `make gen-docs`. Smoke tests in
test/doc_test.lua build the plugin if needed and assert the expected
sections and labels appear. 403/403 luatest green.
runtime: pb.from_pb — build modules from a binary FileDescriptorSet
Complements pb.parse (which consumes .proto source) by accepting the
output of `protoc --descriptor_set_out=...`. Returns
{files={[name]=module}, order, lookup} where each per-file module has
the same shape as pb.parse() output.
Pipeline: hand-built descriptor.proto descriptors in descriptor_pb.lua
decode the wire bytes via pb.codec; fileset.lua translates each
FileDescriptorProto into the AST shape pb.parser emits;
pb.dynamic.build consumes the AST.
Handles map-entry reconstruction (synthetic entry messages skipped from
nested_messages, key/value lifted to the AST map field), proto3_optional
rehydrated as optional=true rather than a synthetic oneof, oneof
grouping, nested types, WKT references.
Tests parity against the statically-generated hello module — 17 cases
covering scalars/repeated/optional/oneof/maps/self-reference/enums/WKTs
plus a full round-trip. 403/403 luatest green.
codegen: text-format printer (pb.text.encode + <Msg>_text wrappers)
Adds a descriptor-driven text-format encoder mirroring `protoc --decode`
output: one field per line, 2-space indent, octal byte escapes,
`nan`/`inf` floats, `opts.single_line=true` for compact one-liners.
WKT-aware — Timestamp/Duration accept datetime cdata or {seconds,nanos},
wrappers print their unwrapped scalar, Struct/Value/ListValue walk the
tagged-table form, FieldMask flattens to `paths:` lines, Any stays
opaque.
Both codegen modes emit `M.<Type>_text(t, opts)`, surfaced as `pb.text`
on the public table. Encode-only; matching parser deferred.
Tests cover scalars, repeated, maps, oneof, optional presence, all WKT
types, single-line mode, and the generated wrapper across both modes
(74 cases). 403/403 luatest green, 19/19 jit-trace gate green.
codegen: EmmyLua type annotations for messages, enums, wrappers
Generated Lua modules now carry lua-language-server type annotations:
---@alias <full.Enum> integer per enum
---@class <full.Message> per message
---@field <name> <type> per field
---@param / ---@return per wrapper
Mappings:
bool -> boolean
string / bytes -> string
float / double -> number
all int kinds -> integer (64-bit cdata typed as integer;
LSP has no cdata model)
enum / message -> <full.Name> (resolves to declared alias/class)
repeated T -> T[]
map<K,V> -> table<K, V>
Presence markers (trailing `?` on field name):
proto3 explicit optional
oneof branches (only one is set at a time)
Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').
Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.
Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.
docs: PLAN.md lazy perf numbers after SoA refactor
The lazy entry's perf claims were written against the per-segment-table
implementation. Update with the post-SoA numbers — passthrough 1.6–1.9×,
sparse read 0.90–1.16×, mutate+reencode 1.09–1.26× — and drop the
caveat about sparse-read losing on flat shapes (no longer true).
lazy: SoA index layout (sparse-read 0.66× → 0.90×, passthrough 1.5× → 1.7×)
Replace per-segment Lua tables with four parallel integer arrays
(id, tag_start, val_start, next_start). For an emails-heavy Person at
100KB that's 4 tables of 2800 ints instead of 2800 tables of 4 keys —
~5× fewer table allocations on decode_lazy.
ArrayView and MapView are similarly flattened: each holds a single
int array of val_starts instead of one mini-table per element.
Packed-payload expansion produces the same shape so :at(i) is one
array index lookup + decode call.
Before / after on bench/lazy_bench.lua (1KB / 10KB / 100KB):
passthrough (decode + reencode)
full: 1.43× → 1.73× 1.13× → 1.62× 1.04× → 1.69×
runtime: 1.43× → 1.94× 1.15× → 1.73× 1.16× → 1.84×
sparse read (:get name + :get age)
full: 0.70× → 0.90× 0.65× → 0.94× 0.60× → 1.03×
runtime: 0.76× → 0.99× 0.66× → 1.05× 0.68× → 1.16×
rewrite name (decode + set + reencode)
full: 0.98× → 1.11× 0.82× → 1.09× 0.81× → 1.14×
runtime: 1.06× → 1.26× 0.95× → 1.15× 0.85× → 1.25×
Sparse read goes from a loss to break-even or better; passthrough
gain widens; mutate-then-reencode flips from regression to consistent
win. JIT-trace gate still 19/19 — the index loop's single side-trace
bridge (decode_tag at lazy.lua:49) was at decode_tag in the old code
too; same pattern, different line number.
Drops the unused wt field from the SoA: consumers never re-read the
tag wire type after the index pass. Unknown-field splice and lazy
:encode emit byte slices directly from tag_start..next_start.
docs: PLAN.md M6 lazy view entry
Marks the M6 "lazy view" bullet done with the API surface, conformance
claim (interop byte-equal through decode_lazy:encode), trace stability
(make jit-trace 19/19), and the honest perf characteristic from
bench/lazy_bench.lua — lazy wins passthrough re-encode (1.0–1.5×),
loses sparse-read 0.60–0.77× on the flat shape because per-segment
table allocation dominates index cost. Mutate-then-reencode is
roughly break-even.
Lazy is a byte-passthrough optimization, not a universal speedup.
bench: lazy decode/encode scenarios + jit-trace coverage
Adds bench/lazy_bench.lua comparing eager decode/encode against
decode_lazy:encode across three workloads (passthrough, sparse-read,
mutate-then-reencode) at 1KB/10KB/100KB on emails-heavy Person.
Output is stderr-only (varies with CPU load; not committed to
baseline.json).
Findings (LuaJIT 2.1.0-beta3, both modes):
- Passthrough re-encode: lazy 1.0–1.5× faster. Untouched views skip
field-walk entirely and return their original bytes verbatim.
- Sparse :get x2: lazy 0.60–0.77× of eager. Per-segment Lua tables
allocated during the index pass cancel the decode-skip savings on
this flat shape (no large subtrees to skip_field over).
- Mutate-then-reencode: lazy 0.81–1.07× of eager — roughly
break-even. Both paths traverse the full byte range; lazy
splices, eager re-emits per field.
Lazy is a byte-passthrough optimization, not a universal speedup. Use
it when you decode, touch few fields, and re-encode — the proxy /
router shape.
Also extends bench/jit_trace.lua with three lazy hot-path checks:
index pass, sparse :get x2, and untouched :encode. All compile with
no fatal aborts (19/19 trace-stability checks pass; one harmless
side-trace bridge per scenario at lazy.lua's index loop, same pattern
as the existing decode_varint bridge).
lazy: zero-copy decode view (decode_lazy) with passthrough re-encode
Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.
Surface (see runtime/pb/lazy.lua):
- MessageView: :get / :has / :which / :iter / :names / :set / :encode
- ArrayView: :len / :at / :iter / :tolist
- MapView: :get / :has / :keys / :iter / :totable
:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).
Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.
Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.
wire: document why encode_varint's fast path stays 1-byte only
Adds a comment explaining the failed experiment with 2/3/4-byte fast
paths: extending the function past the LuaJIT inline budget makes
parent traces stop inlining it, costing ~30% on 1-byte-dominant
workloads (and the bench payload is 1-byte-dominant since most
proto field tags, enum ordinals, and short-string length prefixes
fit in 7 bits). Future maintainers should resist the temptation.
No code change beyond the comment.
wire: encode_varint 1-byte fast path (up to 3.8× encode throughput)
For non-negative Lua numbers under 0x80, return string.char(n) directly
— no `to_uint64` cdata allocation, no `out` table, no table.concat.
Covers the dominant case for typical payloads: length prefixes for
strings <128 bytes, small int field values, enum ordinals, and most
tag bytes when codegen-precomputation isn't available.
Symmetric to the decode_varint 1-byte fast path in 9f3bfb8 but the
payoff is much bigger because the encode path was paying for both
a cdata allocation and a list-builder per call, not just a varint
loop.
Effect (bench/bench.lua, hello.Person):
full/10B encode: 13 → 30 MB/s (2.3×)
full/100B encode: 125 → 278 MB/s (2.2×)
full/1KB encode: 69 → 210 MB/s (3.0×)
full/10KB encode: 98 → 352 MB/s (3.6×)
full/100KB encode: 108 → 398 MB/s (3.7×)
runtime/10B encode: 9 → 16 MB/s (1.8×)
runtime/100B encode: 78 → 151 MB/s (1.9×)
runtime/1KB encode: 68 → 183 MB/s (2.7×)
runtime/10KB encode: 99 → 352 MB/s (3.5×)
runtime/100KB encode: 109 → 411 MB/s (3.8×)
decode: unchanged
alloc/op: unchanged (bench-compare clean)
codec: precompute per-field readers (runtime decode +10–17%)
Mirror of compile_writers on the decode side: pb.finalize_message now
also calls codec.compile_readers(desc), attaching `f._reader` to each
field whose shape can be specialized — singular scalar/enum/message
and repeated scalar/enum/message (packed and unpacked). Each reader
has signature (buf, pos, wt, result) -> new_pos and bakes in the
field name, decode function, packed-detection, list bookkeeping,
nested-message merge rules, and oneof sibling clearing.
The decode_message hot loop becomes:
id, wt, pos = decode_tag(buf, pos)
f = fbi[id]
if f and f._reader then pos = f._reader(buf, pos, wt, result)
else /* existing per-kind dispatch — map fields only */ end
Forward declaration `local decode_msg` so reader closures captured at
finalize-time can refer to it; the later `decode_msg = function...`
fills the upvalue.
Effect (bench/bench.lua, hello.Person):
runtime/100B decode: 246 → 272 MB/s (+11%)
runtime/1KB decode: 122 → 144 MB/s (+17%)
runtime/10KB decode: 176 → 200 MB/s (+13%)
runtime/100KB decode: 180 → 210 MB/s (+17%)
full mode decode: ~flat (already maximally inlined by codegen)
encode: unchanged
alloc/op: unchanged (bench-compare clean)
bridges across 10 jit-trace runs: 3 → 2
Runtime mode decode is now within ~10–15% of full mode across all
sizes, vs 24–28% gap before this commit.
codec: specialize writers for repeated fields (+20–80% encode across sizes)
Extends compile_writers to handle repeated scalar (packed and unpacked),
repeated message, and repeated enum (packed and unpacked). Each writer
knows its tag bytes, encoder function, and packed/unpacked shape; the
encode_message loop calls them directly without rediscovering the field
shape on every iteration.
Combined with the singular-field writers (previous commit), this lifts
runtime-mode encode throughput uniformly:
runtime/10B encode: 5.0 → 9.1 MB/s (+82%)
runtime/100B encode: 48 → 82 MB/s (+70%)
runtime/1KB encode: 50 → 69 MB/s (+39%)
runtime/10KB encode: 87 → 101 MB/s (+16%)
runtime/100KB encode: 90 → 111 MB/s (+22%)
full mode encode: +10% on small sizes, +10% on large (small bonus from
nested encode_msg calls going through writers too)
decode: small uplift on full mode 100B (323 → 340 MB/s), noise on rest
alloc/op: unchanged (bench-compare clean)
bridges across 10 jit-trace runs: 7 → 3 (-57%; total -89% from baseline)
Falls through to encode_field only for map fields and oneof branches —
both keep their existing dispatch path.