wire: decode wins — utf8.len validator + fast paths + FFI cast

Five focused decode optimizations, all in wire.lua, landing a 3.9-6.6x
speedup on bench/bench.lua decode and matching wins on lazy / shapes
benches. Per-helper numbers from bench/wire_bench.lua:

  is_valid_utf8(32B ASCII):  545 ns -> 36 ns  (15x)
  is_valid_utf8(1KB ASCII): 16329 ns -> 539 ns (30x)
  decode_string(32B):         140 ns -> 80 ns  (1.75x)
  decode_double:              299 ns -> 226 ns (1.32x)
  decode_fixed64:             196 ns -> 184 ns (1.07x)

1. utf8.len swap. Pure-Lua RFC 3629 validator replaced by
   `utf8.len(s) ~= nil` (ICU U8_NEXT-backed at src/lua/utf8.c:165).
   Source-verified to reject every proto3 case: stray continuation,
   overlong, surrogates, truncated, > U+10FFFF, 5-byte+ sequences.
   Conformance suite still at 1478/1478 expected passes.

2. decode_string / decode_bytes 1-byte LEN fast path. Strings <=127
   bytes (the RPC common case) skip two function-call layers
   (decode_string -> decode_len -> decode_varint).

3. decode_float / decode_double via ffi.cast(uint8_t*, buf) instead
   of buf:sub. Eliminates the per-call string-slice allocation.

4. decode_double Inf/NaN check via a uint32[2] union split. Bit ops
   on Lua numbers don't allocate; the previous uint64 cdata path
   produced 3+ intermediates per call.

5. decode_fixed64 reads via the new pb_u64_u_t union (ffi.copy +
   read .u). Replaces UINT64(lo) + bit.lshift(UINT64(hi), 32) which
   allocated 2-3 cdata per call.

All FFI cdef/union locals hoisted to the top of the file so the
fixed-width decoders all reference the same scratch buffers.

497/497 luatest pass.  19/19 jit-trace gates pass.  Conformance
(binary + text) shows zero unexpected failures.

json: strict proto3 JSON conformance — close all Required failures

The conformance suite's Required JSON failures collapse to zero with
this pass. Local tests grow from 456 to 497 to pin every fix.

Decode side — `runtime/pb/json.lua`:
  * Strict scalar validators per type. Numeric strings must match the
    JSON number grammar (no leading whitespace, no partial numerics);
    out-of-range values, NaN/Inf surrogates from JSON literals, and
    type mismatches all become parse_error.
  * Quoted exponential ints ("1e5" -> 100000) are accepted via the
    JSON-number grammar path, matching Int32FieldQuotedExponentialValue.
  * Timestamp parser: strict RFC 3339 (uppercase T/Z, ±HH:MM offset,
    ≤9 frac digits, range check). Output uses a portable Hinnant-style
    epoch_to_ymdhms so year 0001 zero-pads correctly — glibc's POSIX
    %Y emits "1" for that year, breaking round-trip.
  * Duration parser/formatter: mandatory `s` suffix, ±10000-year range,
    sign-matching nanos, 0/3/6/9-digit fractional output.
  * Any: WKT-aware nesting under "value"; empty Any -> {}; Empty WKT
    inside Any omits "value" (reference parser rejects {"value":{}});
    @type URL without `/` rejected; empty @type with sibling fields
    rejected.
  * Reject NaN/Infinity in google.protobuf.Value.number_value (no JSON
    literal for these).
  * Duplicate oneof branches rejected; a null oneof branch does NOT
    count as set, so a sibling non-null branch is unambiguous.
  * Top-level JSON null rejected for messages; preserved for Value.
  * Repeated/map values must be JSON array/object (not bare scalar).

Encode side — hand-rolled JSON emitter:
  Tarantool's `json.encode` uses a fixed global precision so doubles
  like 0.1 don't round-trip and we can't change it per-value without
  polluting other users. Replace with a minimal emitter that picks the
  shortest-round-tripping precision (15 -> 16 -> 17) per double and
  handles NaN/Inf as quoted sentinel strings.

`runtime/pb/wkt.lua`: `timestamp_decode` now keeps invalid Timestamps as
a raw {seconds, nanos} table when `datetime.new` rejects them (negative
nanos, year > 9999, ...) so the JSON encoder can produce
serialize_error rather than the binary decoder raising parse_error.
Required by the Timestamp conformance suite.

`test/conformance_test.lua`: 41 new regression tests grouped under
Fixes 12-19, pinning every code path touched. Strict scalar
validators (one per rejection shape per type), Timestamp/Duration
strict parsing and canonical output, Any WKT/non-WKT/Empty handling,
Value NaN/Inf rejection, ValueAcceptNull round-trip, LuaJIT
NaN-boxing collision cases (0x7FFBCBA987654321, all-ones), shortest
double round-trip, oneof-null semantics.

`test/conformance/known_failures.txt`: refreshed. 15 Recommended-only
failures remain (FieldMask round-trip quirks, duplicate-field-name
detection, unknown-enum-string rejection, null-element-in-list,
NullValue oneof validator).

wire: bypass LuaJIT NaN-boxing collision in decode_float/decode_double

Reading a NaN-payload double back through `tonumber(F.d)` runs into
LuaJIT 2.1's NaN-boxed value representation: certain IEEE NaN bit
patterns collide with internal type tags (nil, function, ...) so
`tonumber` yields a non-number and the field is silently dropped
during encoding.

Inspect the raw bit pattern via the uint32_t/uint64_t aliases of the
float/double unions before reaching for the Lua-level value. When the
exponent is all-ones we resolve Inf/NaN ourselves; only normal values
hit `tonumber`.

Closes the conformance suite's DoubleFieldNormalizeSignalingNan and
FloatFieldNormalizeSignalingNan JsonOutput tests, which used bit
patterns (0x7FFBCBA987654321, 0x7FBFFFFF) that hit the collision.

json: enable conformance JSON output + close 459 tests

Three coupled changes that turn the JSON output path on for the
conformance harness:

1. string_to_int64 accepts cdata: Tarantool's json.decode parses raw
   JSON integer literals outside double range as int64_t/uint64_t cdata,
   not Lua numbers. The decoder errored "expected JSON string or number
   for int64" on any unquoted 64-bit value (Int64FieldMaxValueNotQuoted
   et al). cdata is now cast through directly, preserving precision.

2. encode_message marks output as a map: an empty proto3 message
   serialized as `[]` because Tarantool's json defaults empty tables to
   array shape. jsoncpp's strict comparator threw Json::LogicError and
   aborted the whole suite. Setting __serialize='map' on the output
   gives `{}` and unblocks all JsonOutput tests.

3. PB_CONFORMANCE_SKIP_JSON gate is opt-in by default. core.lua now
   matches exactly "1" (so docker -e VAR= disables it), and the
   Dockerfile no longer hard-codes "=1" — JSON output runs end-to-end
   for everyone unless they re-enable the gate.

Conformance moves from 930 / 1869 / 11 to 1389 / 1313 / 79
(successes / skipped / expected fails). The 75 new expected fails
are canonical-form edge cases (Duration formatting sign handling,
Timestamp out-of-range rejection, double precision digits, NaN
canonicalization, JSON-input strict rejection) — left for a follow-up.

Drops 7 entries from test/conformance/known_failures.txt that this
change closes; adds 75 newly-visible ones.

json: canonical lowerCamelCase + NullValue WKT descriptor

Two unrelated JSON-decoder gaps captured under the conformance triage:

1. to_camel's gsub pattern '_(%w)' didn't match consecutive underscores
   and didn't drop trailing underscores, so proto names like
   __field_name13 / field__name4_ / field_name17__ generated JSON keys
   that didn't match what protoc produces. The fix strips trailing _+
   and collapses '_+%w' to a capitalized letter; a leading underscore
   thus capitalizes the next character, matching the spec.

2. pb.wkt didn't export a descriptor for google.protobuf.NullValue, so
   any enum field whose type is NullValue (oneof_null_value, or the
   implicit one inside Value) crashed decode_enum with "attempt to
   index a nil value." Adds a minimal {by_name, by_value} descriptor.

Drops 3 entries from test/conformance/known_failures.txt
(FieldNameInSnakeCase, FieldNameWithDoubleUnderscores,
NullValueInOtherOneofOldFormat). Adds 4 regression tests covering
double underscore, leading underscore, trailing underscore, and the
NullValue enum decode path.

json: treat null fields as absent (and Value's null as a real value)

Per the proto3 JSON spec, a null on any field means "use the field's
default" — encoded as missing — with the lone exception of
google.protobuf.Value, where JSON null is itself a Value carrying
NullValue.NULL_VALUE.

Three coupled bugs surfaced together:

1. decode_field_value used to fall through with v = box.NULL, leaving a
   useless box.NULL sitting in the result table for scalars. Now it
   returns nil for non-Value fields, PB_NULL for Value fields.

2. decode_message's repeated and map branches called `#jv` and
   `pairs(jv)` unconditionally; a JSON-null on either type crashed with
   "attempt to get length of 'void *'". Now both branches short-circuit
   when jv is box.NULL.

3. The nil-skip checks in the decode loop (`if dv ~= nil`) and in the
   codec / inline message encoders (`if v == nil then return end`)
   evaluated TRUE on box.NULL because Tarantool's cdata __eq aliases
   it to nil. Decode now uses rawequal(dv, nil); encode special-cases
   message kind by also accepting cdata, so the Value field's
   box.NULL sentinel survives all the way through to value_encode.

Drops 3 entries from test/conformance/known_failures.txt
(AllFieldAcceptNull, WrapperTypesWithNullValue, ValueAcceptNull).
Adds 5 regression tests covering scalar / repeated / map / wrapper
null treatment and the Value-NULL_VALUE exception.

wire: reject invalid UTF-8 in proto3 string fields

Per the proto3 spec, a string field's bytes must form valid UTF-8.
decode_string was aliased to decode_len, so any byte sequence was
accepted and round-tripped. Adds a pure-Lua RFC 3629 validator —
is_valid_utf8 — and routes M.decode_string through it. The bytes
type keeps the raw decode_len path so binary payloads still pass.

The validator covers stray continuation bytes, truncated sequences,
overlong encodings, UTF-16 surrogates (U+D800..U+DFFF), and code
points above U+10FFFF.

Codec, lazy, dynamic, and the generated inline code all consume the
same M.decode_string, so singular / repeated / oneof / map-key /
map-value string fields are all covered.

Drops 5 entries from test/conformance/known_failures.txt
(RejectInvalidUtf8.String.*) and adds 7 regression tests covering
each invalid form plus a positive multi-byte string round-trip and
a bytes-field control.

wire: validate field number and minimal encoding in decode_tag

Three checks added on the multi-byte path:
1. Field number must be > 0 (IllegalZeroFieldNum_Case_0/1/3).
2. Field number must fit in 29 bits per the protobuf spec
   (BadTag_FieldNumberSlightlyTooHigh, BadTag_FieldNumberTooHigh).
3. Tag varint must be minimally encoded — a trailing 0 byte with more
   than one byte read is overlong (BadTag_OverlongVarint).

The field-number check runs on bit ops over the uint64 cdata returned
by decode_varint rather than after tonumber, otherwise field numbers
above 2^32 alias into the valid range (e.g., fn=2^31+1 was being
recovered as fn=1).

The single-byte fast path picks up the field-zero check directly via
the b >> 3 == 0 condition.

Drops 6 entries from test/conformance/known_failures.txt and adds 4
regression tests pinning each rejection path.

codec: recursively merge repeated singular-message wire entries

Per proto3 spec, when the same singular message field (including a oneof
branch) appears twice on the wire, the two values must merge: scalar
fields last-wins, repeated fields concatenate, sub-messages merge
recursively, maps last-wins per key. The previous reader replaced the
prev value wholesale for oneof branches and overwrote repeated/nested
fields with `prev[k] = v` even outside oneofs, losing data unique to
the first occurrence.

Adds wire.codec.merge_message(desc, prev, decoded), a descriptor-driven
recursive merge, and routes both codec.lua's reader and the inline-mode
codegen through it. WKT message fields (custom decode) keep the replace
behavior because their decoded value is not a generic Lua table.

Exposes pb.codec to the generated inline code so the helper is reachable
without a per-call require.

Drops 3 entries from test/conformance/known_failures.txt:
ValidDataOneof.MESSAGE.Merge, ValidDataOneofBinary.MESSAGE.Merge,
RepeatedScalarMessageMerge. Adds 5 regression tests covering scalar
last-wins, oneof merge, recursive sub-message merge, repeated-in-
submessage concat, and oneof sibling clearing after merge.

json: drop unrecognized enum string names per proto3 spec

decode_enum returned the raw input string for unknown enum names,
which then propagated through the codec and errored at encode time
with "unknown enum value '...'". For repeated and map fields the
spec requires *dropping* the element (not substituting 0), so the
encoded output should be shorter than the input.

decode_enum now returns nil for unrecognized names. The optional /
repeated / map call sites in decode_message skip nil values: optional
leaves the field unset (encoded as default), repeated compacts the
array, map omits the entry. Numeric and known-name inputs are
unchanged.

Drops 5 entries from test/conformance/known_failures.txt:
IgnoreUnknownEnumStringValueIn{Optional,Repeated,RepeatedPart,
MapPart,MapValue}.

wire: bounds-check skip_field for I32/I64/LEN truncation

skip_field advanced pos by a fixed or length-prefixed amount with no
check against #buf, so a truncated unknown field was silently consumed:
the outer decode loop's `while pos <= len` exited without raising,
making the parser accept payloads it should have rejected.

Validates that the new position never exceeds #buf+1 for WIRE_I64,
WIRE_I32, and both WIRE_LEN fast and slow paths. WIRE_VARINT already
errored correctly via decode_varint's per-byte check.

Drops 15 entries from test/conformance/known_failures.txt
(PrematureEofBeforeUnknownValue.*, PrematureEofInsideUnknownValue.*,
PrematureEofInDelimitedDataForUnknownValue.*).

wkt: auto-register WKT descriptors so Any JSON decode resolves @type

pb.wkt exported Timestamp/Duration/Empty/FieldMask/Any/Struct/Value/
ListValue and the nine Wrapper descriptors, but the REGISTRY they live in
was empty until callers manually invoked pb.register. json_to_any looked
up @type in that empty registry, fell through to the opaque base64
fallback, and errored on every Any-of-WKT JSON payload.

Iterates M for every *_descriptor entry at module load and self-registers
it. Also calls pb.register on TestAllTypesProto3 in the conformance runner
so Any tests that embed the user message type also resolve.

Drops 10 entries from test/conformance/known_failures.txt.

wire: reject illegal wire types 6/7 in decode_tag

Wire types 6 and 7 are never assigned in the protobuf wire format; only
0 (VARINT), 1 (I64), 2 (LEN), 3/4 (SGROUP/EGROUP, proto2-only), and 5
(I32) are valid. skip_field already rejected them for unknown fields,
but the decode loop in codec.lua and inline.go dispatched to a typed
reader whenever the field ID was known — the reader ignored wt and
called the value-specific decoder anyway, accepting bytes that should
have been a parse error.

Moves the check into decode_tag so every parser path — codec, generated
inline code, lazy, and map sub-fields — rejects 6/7 uniformly.

Drops 24 entries from test/conformance/known_failures.txt.

wire: truncate varints to 32 bits in int32/uint32/sint32/enum decode

decode_int32/uint32/sint32 returned full uint64 values when the wire input
carried bits above bit 31, corrupting re-encode on 51 conformance tests that
exercise overlong or over-range varints. Per the proto3 spec the decoder
must keep only the low 32 bits (and sign-extend for signed types).

Adds wire.varint_to_int32 / varint_to_uint32 and routes every enum-varint
decode site through them: wire.lua typed decoders, codec.lua (5 enum
sites), lazy.lua (3 sites), and the generated code via inline.go (3 sites).

Drops 51 entries from test/conformance/known_failures.txt.

conformance: local Docker pipeline + cdata int64 map dedup

Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.

Six bugs surfaced and got fixed on the way to green:

  1. conformance_test_runner uses execv (not execvp): bare `tarantool`
     hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
  2. The harness strips LUA_PATH from the child: the runner now
     self-bootstraps package.path from debug.getinfo(1, 'S').source.
  3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
     full BUFSIZ before returning, deadlocking against the parent.
     setvbuf('no') on stdin/stdout.
  4. v34.1 fetches libjsoncpp via CMake FetchContent under
     _deps/jsoncpp-build/...; the runtime image now COPYs the matching
     .so* and runs ldconfig.
  5. The harness's strict jsoncpp comparator crashes on our currently-
     imperfect JSON output (enum numerics, map<K,V> shape, oneof
     object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
     set in the container ENV; host-side `make test` still exercises
     the full JSON path.
  6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
     key map entries (per proto3's "last value wins" semantics) split
     across hash buckets even though __eq matches. Codec walks the
     map once on insert to find a canonical key, gated by a
     precomputed `f.key_dedup` flag so the dedup only fires for
     int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
     same `for _k in pairs(map) do` walk only when the static key
     kind is 64-bit, so string/int32-keyed map decode stays
     JIT-traceable.

Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:

  - Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
  - Text-format suite:     0 ✓ /  430 skipped /   4 expected fails

403/403 luatest green, 19/19 jit-trace gate green.

runtime: pb.from_pb — build modules from a binary FileDescriptorSet

Complements pb.parse (which consumes .proto source) by accepting the
output of `protoc --descriptor_set_out=...`. Returns
{files={[name]=module}, order, lookup} where each per-file module has
the same shape as pb.parse() output.

Pipeline: hand-built descriptor.proto descriptors in descriptor_pb.lua
decode the wire bytes via pb.codec; fileset.lua translates each
FileDescriptorProto into the AST shape pb.parser emits;
pb.dynamic.build consumes the AST.

Handles map-entry reconstruction (synthetic entry messages skipped from
nested_messages, key/value lifted to the AST map field), proto3_optional
rehydrated as optional=true rather than a synthetic oneof, oneof
grouping, nested types, WKT references.

Tests parity against the statically-generated hello module — 17 cases
covering scalars/repeated/optional/oneof/maps/self-reference/enums/WKTs
plus a full round-trip. 403/403 luatest green.

codegen: text-format printer (pb.text.encode + <Msg>_text wrappers)

Adds a descriptor-driven text-format encoder mirroring `protoc --decode`
output: one field per line, 2-space indent, octal byte escapes,
`nan`/`inf` floats, `opts.single_line=true` for compact one-liners.
WKT-aware — Timestamp/Duration accept datetime cdata or {seconds,nanos},
wrappers print their unwrapped scalar, Struct/Value/ListValue walk the
tagged-table form, FieldMask flattens to `paths:` lines, Any stays
opaque.

Both codegen modes emit `M.<Type>_text(t, opts)`, surfaced as `pb.text`
on the public table. Encode-only; matching parser deferred.

Tests cover scalars, repeated, maps, oneof, optional presence, all WKT
types, single-line mode, and the generated wrapper across both modes
(74 cases). 403/403 luatest green, 19/19 jit-trace gate green.

codegen: EmmyLua type annotations for messages, enums, wrappers

Generated Lua modules now carry lua-language-server type annotations:
  ---@alias <full.Enum> integer            per enum
  ---@class <full.Message>                  per message
  ---@field <name> <type>                   per field
  ---@param / ---@return                    per wrapper

Mappings:
  bool                       -> boolean
  string / bytes             -> string
  float / double             -> number
  all int kinds              -> integer  (64-bit cdata typed as integer;
                                          LSP has no cdata model)
  enum / message             -> <full.Name>  (resolves to declared alias/class)
  repeated T                 -> T[]
  map<K,V>                   -> table<K, V>

Presence markers (trailing `?` on field name):
  proto3 explicit optional
  oneof branches             (only one is set at a time)

Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').

Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.

Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.

lazy: SoA index layout (sparse-read 0.66× → 0.90×, passthrough 1.5× → 1.7×)

Replace per-segment Lua tables with four parallel integer arrays
(id, tag_start, val_start, next_start). For an emails-heavy Person at
100KB that's 4 tables of 2800 ints instead of 2800 tables of 4 keys —
~5× fewer table allocations on decode_lazy.

ArrayView and MapView are similarly flattened: each holds a single
int array of val_starts instead of one mini-table per element.
Packed-payload expansion produces the same shape so :at(i) is one
array index lookup + decode call.

Before / after on bench/lazy_bench.lua (1KB / 10KB / 100KB):

  passthrough (decode + reencode)
    full:    1.43× → 1.73×   1.13× → 1.62×   1.04× → 1.69×
    runtime: 1.43× → 1.94×   1.15× → 1.73×   1.16× → 1.84×

  sparse read (:get name + :get age)
    full:    0.70× → 0.90×   0.65× → 0.94×   0.60× → 1.03×
    runtime: 0.76× → 0.99×   0.66× → 1.05×   0.68× → 1.16×

  rewrite name (decode + set + reencode)
    full:    0.98× → 1.11×   0.82× → 1.09×   0.81× → 1.14×
    runtime: 1.06× → 1.26×   0.95× → 1.15×   0.85× → 1.25×

Sparse read goes from a loss to break-even or better; passthrough
gain widens; mutate-then-reencode flips from regression to consistent
win. JIT-trace gate still 19/19 — the index loop's single side-trace
bridge (decode_tag at lazy.lua:49) was at decode_tag in the old code
too; same pattern, different line number.

Drops the unused wt field from the SoA: consumers never re-read the
tag wire type after the index pass. Unknown-field splice and lazy
:encode emit byte slices directly from tag_start..next_start.

lazy: zero-copy decode view (decode_lazy) with passthrough re-encode

Adds pb.decode_lazy(desc, bytes) returning a MessageView/ArrayView/MapView
that indexes the wire bytes in a single pass and decodes individual
fields only on :get / :at access. Nested messages return more lazy
sub-views; WKT descriptors (those carrying desc.decode) are
eager-wrapped so the API stays uniform.

Surface (see runtime/pb/lazy.lua):
  - MessageView: :get / :has / :which / :iter / :names / :set / :encode
  - ArrayView:   :len / :at / :iter / :tolist
  - MapView:     :get / :has / :keys / :iter / :totable

:encode is three-modes: WKT delegates to desc.encode on the materialized
table; untouched views return their original bytes verbatim
(passthrough); mixed views walk fields in id order, splicing clean
segments and re-emitting dirty ones. Sub-MessageView mutations
propagate to parent encode via a flat _sub_msg_views array (walked with
ipairs, so :is_dirty stays on a single JIT trace — pairs over a hash
is NYI in LuaJIT 2.1).

Codegen emits M.<Type>_decode_lazy in both modes as a one-line
delegation to pb.decode_lazy(<desc>, b); no inline expansion.
codec.encode_field is exposed so the lazy passthrough emitter can
splice fresh bytes for a single dirty field without rebuilding the
whole message.

Tests: 40 new lazy_test.lua cases parameterized over both codegen
modes; all 11 interop fixtures round-trip byte-equal through
decode_lazy(b):encode() in both modes. Total: 300/300 luatest, up
from 226.