c-accel: descriptor -> C plan compiler (bd-mq7)
First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.
What the plan carries (per docs/specs/c_accel_strategy.md):
* Per-field records: field_number, wire_type, kind, repeated/packed/
optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
sub_plan_idx (1-based into the sub-plans table), enum_ref
(luaL_ref for enum descriptor), oneof_idx back-pointer.
* Map fields capture map_key_kind, map_value_kind, and the value's
sub_plan_idx when the value is a message.
* Oneofs as a parallel array of {name, member_indices[]}, with
fields' oneof_idx pointing back to their group.
* WKT override detection: when desc.encode/desc.decode are set, the
plan flips has_override=1 and skips field-walk entirely.
* Extension range hooks captured (proto2 scaffolding for bd-3i).
* Field-name cache as a Lua table referenced via luaL_ref, so
encode/decode can do lua_rawgeti instead of re-interning C strings.
Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.
Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.
Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
- module surface + ABI version + KIND/WIRE constants
- Address: 4 fields, names, kinds, wire types, tag bytes, optional
- Person: 14 fields, scalars/enum/message/map/repeated/packed
- Sub-plan resolution + Person.friends self-reference cycle break
- Idempotent compile (second call returns cached plan)
- Result.outcome oneof: 3 members, oneof_idx back-pointers
- WKT Timestamp: has_override=true, field-walk skipped
Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).
Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.
Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
docs: retire PLAN.md in favor of Beads issue tracking
Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.
References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.
The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.
Closes the goal of bd-8k2.
bench,codec: pin proto2 paths, kill pairs() on extension hot path
JIT trace gate gains 12 proto2-specific checks (required encode/decode,
groups singular + repeated, extension encode/decode). The extension-encode
check fired NYI on bytecode 72 (ISNEXT) — pairs() over the new
extensions_by_full_name hash is the same trace-abort that gates map
encode. Fix: register_extension now also appends to extensions_list,
an array view. codec.encode_message, text.emit_message and
json.encode_message all switch to ipairs over the list. Hash tables
stay around for O(1) lookups in decode (extensions_by_id) and bracket-
name resolution (extensions_by_full_name).
Inline (full-mode) codegen learns to walk extensions too: before this
commit the inline encoder skipped t._extensions entirely (only the
runtime codec walked it), so a generated _encode silently dropped any
extension data set on the message. Add the array-walk after the field
loop and a decode_extension dispatch in the unknown-tag branch — the
inline path now matches runtime byte-for-byte.
bench/bench.lua parameterizes over a FIXTURES list so the baseline can
cover both hello.Person and a new proto2_basic.BenchPayload fixture
(required + group + extension + repeated). baseline.json restructured
under "schemas": [{schema, results}, …]; compare walks both. Fresh
numbers committed.
740/740 luatest cases pass; JIT gate 37/37; conformance 2806 binary+JSON
and 434 text-format, both 0 failures.
conformance: dispatch proto2 TestAllTypesProto2
Register the proto2 test message in MESSAGE_REGISTRY and the
Any/JSON registry. The harness now picks up 1312 previously-skipped
binary + JSON cases; whole suite jumps to 2805 ✓ / 0 failures (was
1493 ✓ with proto2 fully skipped).
Text-format suite still has 16 unexpected failures, all bound up
in proto2 group-field text syntax (CamelCase submessage label, no
mandatory `:` before `{`, extension-bracketed group form). Tackled
in the next commit.
The conformance unit test that pinned the proto2-skipped behavior
now targets an editions test type instead so the skip-path still
has a hard assertion.
codec: proto2 groups (SGROUP/EGROUP wire format)
Plugin: detect protoreflect.GroupKind, emit kind='group' in the
field descriptor and a SGROUP opening tag in inline codegen. The
group body bypasses the LEN-prefix path entirely — encoder emits
start-tag, body bytes, end-tag in three slots; decoder calls
pb.codec.decode_group(desc, buf, pos, field_id) which reads inner
tags until the matching EGROUP id.
Wire layer already understood SGROUP/EGROUP for skip_field; the new
decode_group reuses the per-tag dispatch from decode_message but
stops on EGROUP instead of end-of-buffer. Repeated groups bracket
each element with its own SGROUP/EGROUP pair.
Runtime parser now desugars `optional|required|repeated group Name
= id { body }` into (a) a nested message named Name and (b) a
synthetic field of kind=group whose lowercased name is `name` — so
source-parsed proto2 schemas behave the same as build-time codegen.
Text format renders groups under the submessage's capitalized name
(`SingleGroup { ... }`) rather than the lowercase field name,
matching mainline protoc's convention.
Adds a vendored, MessageSet-stripped test_messages_proto2.proto in
test/conformance/proto/ (header comment documents the patch).
10 new luatest cases (group_*) cover singular + repeated + text +
descriptor kind, across both codegen modes. 739 tests pass.
json,text: presence-tracked fields skip proto3 default elision
Both codecs gated default-value elision on `f.optional or f.oneof` —
the proto3 implicit-presence shape. That elides proto2 `required`
fields set to zero and any other presence-tracked descriptor that
doesn't carry the proto3 explicit-optional flag.
Extend the bypass to `f.required` so a Cardinality{r=0} survives
the round-trip through pb.json.encode / pb.text.encode. 4 new
luatest cases pin the rule.
runtime: parser + dynamic accept proto2 sources
Parser now captures `required=true` and `default_value=…` from the
proto2 keywords (instead of dropping `required` silently and ignoring
field options). Dynamic descriptor builder reads `parsed.syntax`,
flips the repeated-scalar packing default for proto2, and calls
`codec.compile_writers/compile_readers` so the per-field
required-writer specialization actually fires — without that the
generic encode_field path silently elides missing required fields.
64-bit-int defaults are coerced into the appropriate cdata type
inside `coerce_default` so the codec's value-comparison rules
match what generated code emits.
Adds 7 dynamic-mode luatest cases including a static-vs-dynamic
byte-parity check. 725 tests pass.
codegen: proto2 baseline — required, optional, custom defaults
Lifts the proto3-only syntax gate in the plugin and threads three
new field-descriptor attributes through codegen and the codec:
* required=true — fields declared with the proto2 `required` keyword.
Inline codegen and the runtime codec both error when
a required field is missing on encode (vs the silent
elide that proto3 implicit-presence fields get).
* optional=true — already wired for proto3 explicit `optional`; in
proto2 every singular field carries it via the
existing HasOptionalKeyword() check, giving presence
semantics without a separate emission path.
* default_value=… — proto2 [default = X] from the field descriptor,
rendered as a Lua literal (cdata for 64-bit ints,
symbolic name for enums) so consumers can surface
it; the codec itself does not auto-materialize
defaults on decode, matching how proto3 absent
fields stay nil.
Packed-by-default already flips correctly because we ask
protoreflect's `IsPacked()`, which is syntax-aware.
Adds test/proto/proto2_basic.proto with 33 luatest cases covering
required validation, optional presence, custom defaults, the proto2
unpacked-by-default repeated rule, nested-required messages, and
full-vs-runtime mode parity. `just gen-proto2-tests` regenerates the
fixture into examples/expected/{full,runtime}/.
Out of scope: extensions, extend, group; conformance harness still
skips TestAllTypesProto2.
codegen: preserve descriptor options as `options = { ... }`
Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.
The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.
The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.
UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
codegen: bracket-quote Lua-keyword field names
The plugin emitted bare-identifier field names in three positions:
the `M.<Type>_fields` table key, the inline encoder's `v = t.<name>`
load, and the inline decoder's `result.<name> = ...` store. When a
proto field name collided with a Lua reserved word the generated
`*_pb.lua` failed to load with `'(' expected near '<keyword>'`.
The real-world hit is pprof's profile.proto, which declares
`repeated Function function = 5` — that breaks all three emit sites.
Route every user-named identifier through new luaTableKey /
luaFieldAccess helpers that bracket-quote reserved words. Covers
field_names + oneof descriptors, oneof presence pre-pass, inline
encode/decode field bodies (including repeated and map paths), and
optional has/clear accessors.
json: M.encode(desc, t, opts) for use_proto_names / emit_defaults / indent
Canonical protojson options were silently ignored. Plumb a single opts
table through encode_message via a CURRENT_ENCODE_OPTS state (mirroring
the decode-side CURRENT_OPTS):
* use_proto_names — emit snake_case field names (proto wire
names) instead of the spec-default
lowerCamelCase.
* emit_defaults — emit zero-valued implicit-presence scalars,
(alias: always_emit_zero_value) empty repeated lists, and empty
maps. Explicit-presence fields (optional /
oneof) and singular message fields remain
absent — matches protobuf-go's behavior.
* indent — pretty-print with the given indent string;
empty arrays/objects stay compact.
Tests parameterize over both codegen modes (descriptor-table contract is
shared between full and runtime) plus a dedicated parity group that
asserts byte-identical JSON across modes for each option.
codegen: resolve (tarantool.lua_package) via global type registry
The custom file option was being dropped silently: protoc encoded it
correctly into the FileDescriptorProto, but protobuf-go parked the
unknown extension in the message's unknown-fields tail because
E_LuaPackage was never registered with protoregistry.GlobalTypes. As a
result proto.GetExtension returned "" and every override the README
documented was a no-op — every caller fell through to the default
"<pkg>.<file>_pb" path resolution.
Register E_LuaPackage in init() and add a luatest regression that
asserts both the output path and the cross-file require strings honor
the option, parameterized over both codegen modes.
json: emit unbroken base64 for bytes fields ({nowrap = true})
Canonical proto3 JSON expects RFC 4648 unwrapped base64, but Tarantool's
`digest.base64_encode` defaults to RFC 2045 MIME-style 76-char line
wrapping. Any `bytes` payload past ~57 bytes used to land in the JSON
string with an embedded `\n`, which breaks every spec-compliant
consumer (grpc-gateway, protojson, protobuf-go's JSON, ...).
Pass `{nowrap = true}` at the two encode sites: the per-field bytes
encoder and the `google.protobuf.Any` opaque-fallback `value` encoder.
Surfaced by tarantool-etcd's `TestJSONGatewayBytesUnwrapped` over Range
responses whose `value` is >=57 bytes; its reference grpc-gateway never
emits the wrapped form.
codegen: emit strict <Type>_fields / _oneofs constants for lazy view
Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.
Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.
Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
codegen: propagate proto comments into generated _pb.lua
Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:
- Message / enum descriptions become `---` blocks above `---@class` /
`---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
lines inside the table literals so readers without an LSP still see
the proto-side context.
Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
json: strict validation pass closes the proto3 conformance suite
Six classes of relaxation that the proto3 JSON conformance corpus
flagged are now enforced. All as Recommended.* tests; combined with
the -0.0 codec fix this empties known_failures.txt and brings the
proto3 binary+JSON suite to 1493 ✓ / 1313 skipped / 0 expected
failures / 0 unexpected failures.
1. Duplicate JSON keys. Tarantool's json.decode is hash-backed and
silently collapses `{"foo":1,"foo":2}` to one entry. A small
byte-walker `find_duplicate_json_keys` runs before json.decode,
tracks per-object brace frames and key sets, errors on the
second occurrence. Closes Recommended.FieldNameDuplicate.
2. camelCase / snake_case aliases of the same proto field appearing
side-by-side. Detected inside decode_message via a `field_seen`
set keyed by proto-name; second hit errors. Closes
FieldNameDuplicateDifferentCasing{1,2}.
3. JSON null inside repeated arrays and map values. Previously
silently dropped; now errors before decode_field_value. Closes
RepeatedField{Message,Primitive}ElementIsNull and
MapFieldValueIsNull.
4. Unknown enum *names* (not integers). decode_enum used to return
nil so callers silently dropped them; now raises by default and
returns nil only when M.decode's `ignore_unknown_fields=true`
opt is set. Conformance dispatch in cmd/conformance/core.lua
forwards this flag when req.test_category ==
JSON_IGNORE_UNKNOWN_PARSING_TEST. Closes
RejectUnknownEnumStringValueIn{Optional,Repeated,Map} and the
paired IgnoreUnknownEnumStringValueIn* tests.
5. google.protobuf.NullValue JSON canonical form. The single enum
value renders as the literal JSON `null` (not the string
"NULL_VALUE"); decode accepts either, encode emits null. The
decode_field_value null-handling path also treats a JSON null
on a NullValue-typed field as "set" rather than "absent" so a
oneof gets marked active. Closes
NullValueInOtherOneof{New,Old}Format.Validator.
6. FieldMask strict round-trip. Path validity is checked on both
sides: the snake_case wire form rejects uppercase letters,
consecutive underscores, trailing underscore, and underscore
followed by anything other than a lowercase letter — these
break the snake↔camel round-trip. The JSON form rejects any
underscore in the input (must be lowerCamelCase). Closes
FieldMask{TooManyUnderscore,PathsDontRoundTrip,
NumbersDontRoundTrip}.JsonOutput and JsonInput.FieldMaskInvalidCharacter.
The pre-existing "drop unknown enum strings" unit regressions in
test/conformance_test.lua were inverted to assert the new error
shape. New strict-validation regressions in test/json_test.lua pin
all six categories so they don't regress; the `json.strict` group
runs across both codegen modes via the shared descriptor table.
codec: preserve -0.0 for proto3 float/double scalars
Proto3 default-elision dropped any singular float/double whose value
compared equal to 0 — `-0.0 == 0.0` in IEEE so the `if v ~= 0` guard
silently elided negative zero. The wire bytes for -0 differ from +0
and the TextFormatInput conformance corpus pins that -0 must survive
a round-trip; the failure surfaced as 10 unexpected text-suite
regressions covering FloatFieldNegativeZero (3 spellings × 2 outputs)
and Neg{Float,Double}FieldLargeNegativeExponentParsesAsNegZero
(2 types × 2 outputs).
Sign-bit guard via `1/v == math.huge` (positive zero yields +inf,
negative zero yields -inf). Applied in three layers:
* runtime/pb/codec.lua: is_default_scalar + the specialized
monomorphic writer for non-optional numeric scalars
* runtime/pb/text.lua: is_proto3_default (text encoder elision)
* inline codegen: scalarNotDefaultExpr emits the guard for
float/double fields in full-mode `_encode` functions
Proto3 text-format conformance now sits at 416 ✓ / 18 skipped / 0
expected failures; binary+JSON holds at 1478 ✓. Unit-test regressions
cover both directions (preserved on encode + decode round-trip, +0
still elided) and use a runtime-computed -0.0 sentinel because LuaJIT
can constant-fold the literal `-0.0` to a sign-less zero in some
load paths.
text: add pb.text.decode and wire it into conformance dispatch
Hand-written recursive-descent parser for the textproto grammar
covering every bucket the proto3 conformance suite exercises:
decimal/hex/octal integer literals with full 32/64-bit range checks,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflows to ±0), C-style + \u/\U string escapes
with adjacent-literal concat and surrogate rejection, aggregate {} /
<> bodies, repeated short-form `[a, b, c]`, `key: K value: V` map
entries, the `[type.googleapis.com/...]` inline Any form alongside
the direct `type_url:`/`value:` form, enum-by-name-or-number,
reserved-name silent drop, numeric-field-ID tolerance, and
duplicate-singular-field rejection.
Plugin gains a small reserved_names emitter so the parser can match
mainline TextFormat::Parser's "silently drop reserved" rule. The Any
WKT descriptor advertises its real fields (type_url + string,
value + bytes) so the generic body walker can populate it directly
when the input doesn't use the inline-URL form.
cmd/conformance/core.lua stops short-circuiting text_payload to
`skipped` and runs it through pb.text.decode. The proto3 TextFormat
input suite climbs from 8 ✓ / 426 skipped to 406 ✓ / 18 skipped / 10
expected failures. The 10 surviving failures all share one cause
(proto3 -0.0 elision in the codec, not a parser bug — documented in
test/conformance/known_failures_text.txt). Binary+JSON conformance
holds at 1478 ✓. 591 unit tests pass across both codegen modes.
Closes the text-conformance-output branch.
text: render captured unknown fields, tolerate SGROUP in skip
Two changes close the proto3 text-format conformance suite:
1. `wire.skip_field` learns SGROUP/EGROUP. Wire 3 recurses through
inner tags until a matching EGROUP, with field_id checked against
the SGROUP's id. Callers (codec.lua, lazy.lua, wkt.lua, generated
full-mode `_pb.lua`) now pass the tag's field_id so groups inside
unknown-field skips don't error.
2. `pb.text.encode` walks the captured `_unknown_fields` buffer when
`opts.print_unknown_fields=true` and emits each entry in
TextFormat numeric-field form:
VARINT -> "<id>: <uint64>"
I64/I32 -> "<id>: 0x<hex>"
LEN -> speculative "<id> { <recurse> }"; rolls back to
byte-string form if the inner bytes don't parse as a
sub-message
SGROUP -> "<id> { <recurse> }" through matching EGROUP
`cmd/conformance/core.lua` threads `req.print_unknown_fields` into
`pb.text.encode` so `_Drop` tests drop unknowns and `_Print` tests
render them.
Conformance: text-format suite goes from 2 ✓ / 6 expected fails to
8 ✓ / 0 expected fails. All eight regression tests in
`conformance_test.lua` (one per upstream test, plus the fixed field-1011
tag bytes that were miscomputed earlier) now assert the target output.
test: pin unknown-fields text-format conformance regressions
Adds eight tests under conformance.core mirroring the
Recommended.Proto3.ProtobufInput.*UnknownFields_*.TextFormatOutput tests
in the Google harness. Each sends the byte-exact upstream payload (field
IDs 1001..1011 from UnknownToTestAllTypes) through cmd/conformance/core
and pins current output, with the target assertion + needed runtime fix
documented in each failure message.
Two blockers surface:
1. wire.skip_field rejects SGROUP/EGROUP (wire 3/4), so the four
Group/Repeated tests fail at decode.
2. pb.text.encode doesn't walk _unknown_fields, so the four *_Print
tests serialize empty.
Lets us iterate on those fixes locally without the Docker round-trip.