runtime: pb.decode_unsafe + M.<Name>_decode_unsafe in runtime mode (58u)
Completes the unsafe-decode story 6bb started in full mode. Runtime mode
now exposes the same API via parallel `f._reader_unsafe` closures
compiled in pb.finalize_message against a swapped scalar table where
`string` maps to the bytes handler (no utf8_len). build_reader /
build_repeated_reader / decode_one are now parameterized on
(scalar_tbl, decode_msg_fn, decode_group_fn) so the same builders emit
both reader shapes. decode_message_unsafe, decode_group_unsafe, and
decode_extension_unsafe are literal clones of their safe twins with
three substitutions (documented in codec.lua): the _reader field, the
scalar table in slow paths, and the sub-message / group / extension
dispatchers. Tests in test/decode_unsafe_test.lua are now parameterized
over both modes (14 cases, including a map<string, int32> invalid-key
case that exercises the decode_one map-fallback path).
Runtime-mode microbench shows ~8% throughput vs validating decode on
the string-heavy 1KB Person; smaller than full mode's ~20% because the
descriptor dispatch + closure indirection swamp utf8_len, but still a
net win and the perf-cost-of-validating story is now consistent across
modes. Conformance 3240/3240 + JIT trace 37/37 still pass.
Closes 58u, also closes b12 (already fixed in 2656c97; never closed).
kyt still tracks unifying _decode_unsafe with C accel.
codegen: CHARS[_len] lookup replaces string.char(_len) at length-prefix sites (2ri)
Profile attributed 39% of Person_encode's trace share (~28% of total) to a
single line emitting `string.char(_len)` at every inlined length-prefix
site. The `_len` argument is rarely a compile-time constant (lengths come
from user data), so the JIT can't fold the C-function call, and the cost
compounds — the 1KB Person fixture fires ~30 length-prefix sites per
encode.
Replace with a 256-entry lookup table `wire.CHARS` (built once at module
load, byte i -> string.char(i)). Codegen header now emits
`local CHARS = wire.CHARS` alongside the other hot-path localizers; the
single emit site in `emitInlineLenPrefix` swaps `string.char(_len)` for
`CHARS[_len]`. Parity is by construction — both return the same interned
1-byte string.
Tests: 752/752 pass. Bench (work.lab.local, median of 3, hello.Person
encode): 10B +4.7%, 1KB +16.8%, 10KB +21.1%, 100KB +34.2%, proto2 mid
+11.1%. Decode unchanged. See bench/PERF_LOG.md 2026-05-24 2ri entry.
Closes lkz and 86g (ffi.new buffer rewrite paths) — separate hand-spike
of that shape regressed 0.31x-0.77x across all sizes; the perceived
buffering inefficiency wasn't there, and 2ri captured the single hottest
line. Remaining encode-perf headroom is c0i (C-runtime backend).
codegen: inline proto2 extension writers/readers (qwt) + table.new(N,0) for packed lists (2sn)
qwt closes the proto2_basic.BenchPayload `min` bench's worst data point:
encode 1.89M → 3.30M msgs/s (+75%), decode 1.02M → 1.26M msgs/s (+23%).
Mechanism: collectExtsByExtendee groups all `extend Foo { ... }`
declarations by extendee FullName across input files; emitInlineEncode
emits one dedicated writer per extension instead of dispatching through
pb.codec.encode_field, and emitInlineDecode adds an elseif arm per
extension id straight into result._extensions[full_name]. Dynamic
extensions_list walk preserved for forward compat, gated on
#_elist > N so it pays one int compare when no runtime extension was
registered.
2sn pre-sizes packed-scalar repeated lists via table.new(N, 0) where
the count is recoverable from the LEN payload — exact (lim >> 2 or
lim >> 3) for fixed-width, upper bound (lim) for varint-packed. The
allocation is deferred into the `if wt == 2 then` branch so the
per-element fallback and non-packable types keep the bare-`{}`
alloc. Avoids u39's regression mode because the call cost is paid
once per repeated-field-first-occurrence, not per message decode.
Bench summary for hello.Person full mode: encode +3-5% across sizes,
decode +1-3% across sizes. Tests: 752/752 pure-Lua, 1043/1043 with
PB_ENABLE_C=1, JIT 37/37. PERF_LOG entry covers the rationale and
caveats.
codegen: inline 1-byte LEN fast path for string/bytes decode
Singular and repeated string/bytes fields in generated full-mode _decode
now read the length byte and dispatch in-line instead of calling
wire.decode_string / wire.decode_bytes. For length < 128 (the common
short-string RPC case), the path stays inside the parent JIT trace:
no child-trace stitch, no per-element function frame, utf8_len lookup
hoisted to a generated-file upvalue.
The original a6n design — one ffi.cast(U8CP, buf) at the top of each
_decode — was abandoned: the cdata wrapper is 24 bytes per call and
LuaJIT can't sink the allocation because the pointer local lives
across wire.decode_* call frames. Net regression at small sizes
(+11-16%) overwhelmed the single-byte-read savings.
Bench (hello.Person full-mode decode):
size ns/op before ns/op after delta
10B 419 390 -6.9%
100B 515 478 -7.2%
1KB 9004 8042 -10.7%
10KB 64644 56745 -12.2%
100KB 625638 537750 -14.0%
Zero allocation impact across all sizes. Tests: 1043 pass, 37 JIT
trace gates pass. (a6n)
c_runtime: 64-bit cdata fidelity tests (ra6 3l)
Adds the c_int64.Wide fixture (one singular field per 64-bit kind)
plus an 18-test luatest group that round-trips each kind past 2^53
through the C runtime in both codegen modes. Confirms encode accepts
both LuaJIT int64_t/uint64_t cdata and Lua numbers, and decode
surfaces values >DBL_INT_MAX as cdata (matching msgpackffi /
net.box / box.tuple / built-in protobuf convention).
The C runtime already had the dispatch — to_int64_at / to_uint64_at
flow through luaL_toint64 / luaL_touint64 for cdata inputs, and
dec_push_one calls luaL_pushint64 / luaL_pushuint64 for every 64-bit
kind. This change pins the behavior under acceptance.
Closes tarantool-protobuf-awv (ra6 3l)
c_runtime: repeated + packed scalar encode/decode (ra6 3e)
Add repeated dispatch to the C-runtime encode/decode loop. Encode
side: encode_repeated_field walks Lua arrays via lua_objlen + per-
index rawgeti, dispatches on element kind. Packed numerics build
their payload in a stack-backed sub-buffer then emit `tag(LEN) +
varint(len) + body`; unpacked emit `tag + value` per element via
encode_one_field with force_emit=1 to bypass zero-suppression;
strings/bytes flow through the same path (never packable); repeated
messages reuse encode_submessage_field per element.
Decode side: per-field stack-slot cache (list_stack_idx[]) tied to
list_count[] avoids the per-element lua_getfield(result, name) round
trip the c-accel spike measured at 2x slower at 100KB. On first hit
for a repeated field we lua_createtable + write result[name] AND dup-
push the list onto the stack; subsequent hits lua_rawseti through the
cached absolute stack index. Lists stay valid across recursive sub-
message decodes because each child decode_body cleans up its own
scratch back to the caller's frame.
Packed/unpacked symmetry on read: a wt==LEN payload for any packable
scalar is decoded as a packed blob regardless of the schema's packed
flag, and a per-element-tagged stream is decoded element-by-element
even on a schema that defaults to packed — per proto3 reader rules.
New test/proto/c_repeated.proto fixture carries packed + explicit-
unpacked + repeated string/bytes + repeated message branches. The
encode and decode tests round-trip at 10/100/1000 elements per
branch. The two existing "skip repeated and map" marker tests
collapse to "skip map" — only map fields remain out of scope for
3e (bd-asz / 3h lands them next). 854 → 896 passing tests.
bd-jc9
c_runtime: encode/decode singular sub-messages (ra6 3d)
Refactor encode_lua/decode_lua into encode_body/decode_body so the
field-walk loop is callable recursively, then dispatch the MESSAGE
kind into a per-side sub-handler. Repeated and map fields still skip
at the field-walk level — 3e (jc9) and 3h (asz) land them next.
encode_submessage_field force-establishes the parent enc_buf's
heap_idx via a no-op ebuf_grow before recursing. Without that the
final ebuf_reserve on the parent could land its new userdata above
sub-encode's leaked stack slots, making the closing lua_settop drop
the parent's heap.
decode_submessage_field bounds the inner read by temporarily
shrinking c->len to the sub-message end offset; the wire-prim
helpers already bounds-check against c->len, so a malformed inner
payload can't over-read into the outer message's bytes.
New fixture test/proto/c_nested.proto carries a 5-level singular
chain (L1->L2->L3->L4->L5) for the depth test. The two existing
"skip message" tests are renamed to "skip repeated and map" — sub-
messages now encode and decode end-to-end.
bd-hwe
codegen: inline 1-byte tag fast path at decode call sites
Hoists wire.decode_tag's 1-byte fast path into every generated
M.X_decode while-loop, falling back to the helper for multi-byte
tags (field ids > 15). The 1-byte case covers every protobuf field
with id 1..15 and is the dominant decode dispatch in real payloads.
Header now localizes string.byte, bit.band, and bit.rshift so the
inlined ops compile to straight local calls.
Bench (Person full decode, msgs/s, median-of-3 vs proper post-h8v
3-run baseline): 10B +14.7%, 100B +8.9%, 1KB +7.5%, 10KB +7.0%,
100KB +8.5%. Full encode is flat to small (-0.1% to -2.6%) at large
sizes, plausibly from header-upvalue layout. JIT trace gate: 37/37,
all bridges still 0. Tests: 745/745.
Also documented in bench/PERF_LOG.md, including the methodology
note that h8v's earlier numbers used single-run baselines and are
therefore ~3-5% optimistic; medians-of-3 are the standard now.
beads-tarantool-protobuf-4kj
bench,codec: pin proto2 paths, kill pairs() on extension hot path
JIT trace gate gains 12 proto2-specific checks (required encode/decode,
groups singular + repeated, extension encode/decode). The extension-encode
check fired NYI on bytecode 72 (ISNEXT) — pairs() over the new
extensions_by_full_name hash is the same trace-abort that gates map
encode. Fix: register_extension now also appends to extensions_list,
an array view. codec.encode_message, text.emit_message and
json.encode_message all switch to ipairs over the list. Hash tables
stay around for O(1) lookups in decode (extensions_by_id) and bracket-
name resolution (extensions_by_full_name).
Inline (full-mode) codegen learns to walk extensions too: before this
commit the inline encoder skipped t._extensions entirely (only the
runtime codec walked it), so a generated _encode silently dropped any
extension data set on the message. Add the array-walk after the field
loop and a decode_extension dispatch in the unknown-tag branch — the
inline path now matches runtime byte-for-byte.
bench/bench.lua parameterizes over a FIXTURES list so the baseline can
cover both hello.Person and a new proto2_basic.BenchPayload fixture
(required + group + extension + repeated). baseline.json restructured
under "schemas": [{schema, results}, …]; compare walks both. Fresh
numbers committed.
740/740 luatest cases pass; JIT gate 37/37; conformance 2806 binary+JSON
and 434 text-format, both 0 failures.
codec,text,json: proto2 extensions end-to-end
Plugin iterates file.Extensions and per-message Extensions, emitting
\`pb.register_extension(extendee_desc, {…})\` calls. Each extension
becomes a field descriptor attached to the extendee under two
indices: extensions_by_id (wire-decode lookup) and
extensions_by_full_name (encode + textual emission ordering).
Codec encode_message walks data._extensions after the regular field
loop and calls encode_field through the extension's descriptor.
decode_message routes an unknown tag through decode_extension when
extensions_by_id matches; the helper mirrors the in-line dispatch
on field kind (scalar/enum/message/group, singular/repeated) and
stores into result._extensions[full_name].
Text decoder recognizes bracket syntax \`[pkg.ext_name]\` outside an
Any-target and looks the name up in extensions_by_full_name. Names
that don't resolve (e.g. \`[pkg.GroupField]\` for the CapitalCase
group type instead of the lowercase field name) raise a parse error
per spec. Text encoder emits each set extension as
\`[full.name]: value\` or \`[full.name] { … }\`.
JSON encoder emits set extensions under the bracketed full-name key
(\`"[pkg.ext_name]": value\`). JSON decoder recognizes the same
shape and stores into _extensions; unknown bracket-names stay
silently dropped to match the spec's tolerant behavior for unknown
JSON keys.
Extensions on google.protobuf.* descriptors (file/message/field
option extends) are skipped — those are meta-only and our WKT
module doesn't surface their descriptors.
Conformance baseline (strict --enforce_recommended, protobuf v34.1):
Binary + JSON suite: 2806 ✓ / 0 failures (was 1493 ✓ / 1313 skipped)
Text-format suite: 434 ✓ / 0 failures (was 416 ✓ / 18 skipped)
100% pass.
text,codec: proto2 group text syntax + closed-enum semantics
Text decoder now resolves a group field reference by the group's
capitalized submessage name (e.g. \`Data\`, \`MultiWordGroupField\`)
and by the ASCII-lowercase fold (\`data\`, \`multiwordgroupfield\`),
in addition to the existing lowercase field-name lookup. The \`:\`
between field and \`{\` is optional for groups (matches the
message/map rule).
Proto2 enums are closed: parse_enum_value now rejects an integer
literal that doesn't map to any declared value. The plugin emits
\`closed = true\` on every proto2 enum descriptor (driven by
protoreflect's IsClosed()); proto3 enums stay open to preserve
forward-compatibility on the wire.
Conformance text-format suite: 16 unexpected failures → 3, all in
the remaining extension-bracketed-group cases.
codec: proto2 groups (SGROUP/EGROUP wire format)
Plugin: detect protoreflect.GroupKind, emit kind='group' in the
field descriptor and a SGROUP opening tag in inline codegen. The
group body bypasses the LEN-prefix path entirely — encoder emits
start-tag, body bytes, end-tag in three slots; decoder calls
pb.codec.decode_group(desc, buf, pos, field_id) which reads inner
tags until the matching EGROUP id.
Wire layer already understood SGROUP/EGROUP for skip_field; the new
decode_group reuses the per-tag dispatch from decode_message but
stops on EGROUP instead of end-of-buffer. Repeated groups bracket
each element with its own SGROUP/EGROUP pair.
Runtime parser now desugars `optional|required|repeated group Name
= id { body }` into (a) a nested message named Name and (b) a
synthetic field of kind=group whose lowercased name is `name` — so
source-parsed proto2 schemas behave the same as build-time codegen.
Text format renders groups under the submessage's capitalized name
(`SingleGroup { ... }`) rather than the lowercase field name,
matching mainline protoc's convention.
Adds a vendored, MessageSet-stripped test_messages_proto2.proto in
test/conformance/proto/ (header comment documents the patch).
10 new luatest cases (group_*) cover singular + repeated + text +
descriptor kind, across both codegen modes. 739 tests pass.
codegen: proto2 baseline — required, optional, custom defaults
Lifts the proto3-only syntax gate in the plugin and threads three
new field-descriptor attributes through codegen and the codec:
* required=true — fields declared with the proto2 `required` keyword.
Inline codegen and the runtime codec both error when
a required field is missing on encode (vs the silent
elide that proto3 implicit-presence fields get).
* optional=true — already wired for proto3 explicit `optional`; in
proto2 every singular field carries it via the
existing HasOptionalKeyword() check, giving presence
semantics without a separate emission path.
* default_value=… — proto2 [default = X] from the field descriptor,
rendered as a Lua literal (cdata for 64-bit ints,
symbolic name for enums) so consumers can surface
it; the codec itself does not auto-materialize
defaults on decode, matching how proto3 absent
fields stay nil.
Packed-by-default already flips correctly because we ask
protoreflect's `IsPacked()`, which is syntax-aware.
Adds test/proto/proto2_basic.proto with 33 luatest cases covering
required validation, optional presence, custom defaults, the proto2
unpacked-by-default repeated rule, nested-required messages, and
full-vs-runtime mode parity. `just gen-proto2-tests` regenerates the
fixture into examples/expected/{full,runtime}/.
Out of scope: extensions, extend, group; conformance harness still
skips TestAllTypesProto2.
codegen: preserve descriptor options as `options = { ... }`
Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.
The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.
The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.
UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
docs: full reference + how-to set, migrate Makefile to Justfile
Documentation overhaul that adds the missing user-facing surface:
four reference pages (runtime-api, generated-api, cli, grpc-contract),
twelve how-tos walking from first-message through custom transports,
a troubleshooting page, and a docs/index map. Every how-to references
a runnable artifact under examples/, all of them verified end-to-end.
Build system migration: the Makefile is gone; the Justfile is now
the canonical entry point and absorbs every target. examples/Justfile
ships one recipe per runnable example, forwarded via top-level
'just examples <name>'. The 'examples are part of the documented
surface' convention is pinned in CLAUDE.md, alongside a dedicated
section on updating the conformance harness (PROTOBUF_TAG bumps,
libjsoncpp path drift, new test-category wiring).
Stale-number sweep across README/PLAN/CLAUDE: fixture count 18→10,
test count 613/130→639, wire.lua LOC dropped, M7 marked done.
Descriptor-shape block deduplicated against codegen.md as the
canonical source. gRPC transports spec status reframed from
'draft / decision deferred' to 'shipped contract; external
transports deferred'.
.gitignore picks up *.snap / *.xlog / *.vylog / *.run / *.pid /
512.lock so example state can't leak into the working tree.
codegen: emit strict <Type>_fields / _oneofs constants for lazy view
Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.
Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.
Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
codegen: propagate proto comments into generated _pb.lua
Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:
- Message / enum descriptions become `---` blocks above `---@class` /
`---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
lines inside the table literals so readers without an LSP still see
the proto-side context.
Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
text: add pb.text.decode and wire it into conformance dispatch
Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.
Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.
cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.
Closes the text-conformance-output branch.
codegen: text-format printer (pb.text.encode + <Msg>_text wrappers)
Adds a descriptor-driven text-format encoder mirroring `protoc --decode`
output: one field per line, 2-space indent, octal byte escapes,
`nan`/`inf` floats, `opts.single_line=true` for compact one-liners.
WKT-aware — Timestamp/Duration accept datetime cdata or {seconds,nanos},
wrappers print their unwrapped scalar, Struct/Value/ListValue walk the
tagged-table form, FieldMask flattens to `paths:` lines, Any stays
opaque.
Both codegen modes emit `M.<Type>_text(t, opts)`, surfaced as `pb.text`
on the public table. Encode-only; matching parser deferred.
Tests cover scalars, repeated, maps, oneof, optional presence, all WKT
types, single-line mode, and the generated wrapper across both modes
(74 cases). 403/403 luatest green, 19/19 jit-trace gate green.
codegen: EmmyLua type annotations for messages, enums, wrappers
Generated Lua modules now carry lua-language-server type annotations:
---@alias <full.Enum> integer per enum
---@class <full.Message> per message
---@field <name> <type> per field
---@param / ---@return per wrapper
Mappings:
bool -> boolean
string / bytes -> string
float / double -> number
all int kinds -> integer (64-bit cdata typed as integer;
LSP has no cdata model)
enum / message -> <full.Name> (resolves to declared alias/class)
repeated T -> T[]
map<K,V> -> table<K, V>
Presence markers (trailing `?` on field name):
proto3 explicit optional
oneof branches (only one is set at a time)
Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').
Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.
Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.