M PLAN.md => PLAN.md +115 -65
@@ 17,66 17,81 @@ A first-class Protocol Buffers + gRPC stack for Tarantool that:
- Is comfortable to use from real Tarantool apps: clean integration with
`net.box`, `fiber`, `box.session`, and the 3.x config framework.
-## 2. Current state (M0)
+## 2. Current state (post-M7)
See `README.md` for the user-visible summary. Internally:
-- Plugin: Go, single mode (descriptor + runtime dispatch).
-- Runtime: pure Lua + LuaJIT FFI; ~260 LOC wire layer, ~250 LOC dispatch.
-- Tests: 25 hand-rolled assertions in one file, no luatest.
-- Demo proto: 1 file, 11 fields, no map/oneof/services.
-
-This is the floor we're building on.
+- **Plugin**: Go, two emission modes (`mode=full` inline + `mode=runtime`
+ descriptor-delegating wrappers). Sibling `protoc-gen-tarantool-doc`
+ emits Markdown reference per `.proto`.
+- **Runtime**: pure Lua + LuaJIT FFI. `wire.lua` (~660 LOC) + `codec.lua`
+ + `lazy.lua` (zero-copy view) + `text.lua` (encode + decode) +
+ `json.lua` (strict proto3 JSON) + `wkt.lua` (all 9 well-known types) +
+ `grpc.lua` (transport + loopback) + `parser.lua` / `dynamic.lua` /
+ `fileset.lua` (three runtime descriptor producers — `.proto` source,
+ AST, FileDescriptorSet bytes).
+- **Tests**: 613 luatest assertions across 19 groups, parametrized over
+ both codegen modes where applicable.
+- **Conformance**: proto3 binary+JSON suite **1493 ✓ / 0 failures**;
+ proto3 text-format suite **416 ✓ / 0 failures** (Google's
+ `conformance_test_runner` v34.1).
+- **Bench**: `bench/baseline.json` tracks allocation/op (regression
+ gate at 5%); `make jit-trace` pins LuaJIT trace stability.
+
+The remaining unfinished bullets in section 3 are mostly M8 (release
+engineering) and a handful of optional perf items in M6.
## 3. Phased roadmap
Each milestone ends with a green CI run, an updated README, and a tagged
release on sourcecraft.dev.
-### M1 — Two codegen modes + luatest harness *(in progress)*
+### M1 — Two codegen modes + luatest harness *(done)*
-- [ ] Promote scalar encode/decode to typed helpers in `pb.wire`.
+- [x] Promote scalar encode/decode to typed helpers in `pb.wire`.
`wire.encode_int32(v)`, `wire.decode_string(buf, pos)`, etc., for all
15 scalar proto types. Both modes consume the same primitives.
-- [ ] **Full (inline) codegen**: emit per-message `_encode` / `_decode`
+- [x] **Full (inline) codegen**: emit per-message `_encode` / `_decode`
functions with no descriptor lookup. Tag bytes precomputed at gen
time as Lua string literals. This is the JIT-friendly hot path.
-- [ ] **Runtime codegen**: keep current behavior. Useful for introspection,
+- [x] **Runtime codegen**: keep current behavior. Useful for introspection,
schema registries, and forward-compat with descriptor-only consumers.
-- [ ] Plugin parameter `mode=full|runtime` (default: `full`).
-- [ ] Migrate tests to luatest groups. Every behavior tested against **both**
+- [x] Plugin parameter `mode=full|runtime` (default: `full`).
+- [x] Migrate tests to luatest groups. Every behavior tested against **both**
generated modules to prove parity.
-- [ ] Generate side-by-side outputs in `examples/expected/{full,runtime}/`
+- [x] Generate side-by-side outputs in `examples/expected/{full,runtime}/`
for visual diffing.
-**Done when**: same `.proto` produces two modules; identical wire bytes; one
-luatest run covers both.
-
-### M2 — Composite types
+### M2 — Composite types *(done)*
-- [ ] **`map<K,V>`**: emit as repeated synthetic `*Entry` messages with
+- [x] **`map<K,V>`**: emit as repeated synthetic `*Entry` messages with
`key`/`value` fields per spec. Decode merges into a Lua table; encode
iterates with `pairs`. Key types limited to scalars + string per spec.
-- [ ] **`oneof`**: descriptor includes `oneof_index` per field. Encode
+- [x] **`oneof`**: descriptor includes `oneof_index` per field. Encode
emits at most one branch (last assignment wins). Decode clears prior
- oneof siblings on assignment.
-- [ ] **proto3 explicit `optional`**: respect presence — emit field even
+ oneof siblings on assignment. Hot-path lookup goes through
+ `desc.oneofs_list` (flat array) to keep the trace JIT-stable.
+- [x] **proto3 explicit `optional`**: respect presence — emit field even
when value equals scalar default. Generated descriptor exposes
`has_<name>(t)` / `clear_<name>(t)` helpers.
-**Done when**: a "composite" example proto with map<string,int32>, a oneof
-with 3 cases, and an explicit-optional bool round-trips through both modes
-with parity-against-protoc.
-
-### M3 — Well-known types
-
-- [ ] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module
- (epoch + nsec mapping).
-- [ ] `google.protobuf.Duration` ↔ `interval` (or seconds+nanos table).
-- [ ] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar:
- pass plain Lua value → auto-wrap; decode → auto-unwrap or keep.
-- [x] `Empty`, `FieldMask`. FieldMask is a `repeated string`; JSON mapping
- is the canonical comma-joined `lowerCamelCase` form.
+### M3 — Well-known types *(done)*
+
+- [x] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module
+ (epoch + nsec mapping). Out-of-spec inputs (negative nanos,
+ year > 9999) keep the raw `{seconds, nanos}` table so JSON
+ serialization can reject them with `serialize_error` instead of
+ crashing on decode.
+- [x] `google.protobuf.Duration` ↔ `{seconds, nanos}` table (interval
+ proved a poor fit — it carries months/days that don't map
+ cleanly).
+- [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar:
+ pass plain Lua value → auto-wrap; decode → auto-unwrap.
+- [x] FieldMask: `repeated string`. Strict round-trip validation —
+ snake_case paths must use only `[a-z0-9_]`, no leading/trailing
+ `_`, no `__`, and `_` must precede a lowercase letter (not a
+ digit). JSON form rejects any `_` (must be lowerCamelCase).
+- [x] `Empty`.
- [x] `Any`: opaque `{type_url, value}` form by default; `pb.register(desc)`
+ `pb.any.pack(desc, t)` / `pb.any.unpack(any_t)` for typed
round-trips. JSON canonical mapping emits the flat `{"@type": ...}`
@@ 125,41 140,59 @@ with parity-against-protoc.
fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.
-### M5 — Conformance + interop *(runner shipped; pass-rate tracking pending)*
+### M5 — Conformance + interop *(proto3 closed; CI wire-up + proto2 deferred)*
- [x] Wire up Google's [protobuf conformance test runner][conformance].
`cmd/conformance-runner.lua` is the testee: reads length-prefixed
- `ConformanceRequest` on stdin, runs it through our codec / JSON,
- writes a length-prefixed `ConformanceResponse` on stdout. Loops to
- EOF. Core dispatch is factored into `cmd/conformance/core.lua` and
- exercised directly by `test/conformance_test.lua` (10 dispatch
- cases + 3 stdin/stdout framing cases).
+ `ConformanceRequest` on stdin, runs it through our codec / JSON /
+ text codecs, writes a length-prefixed `ConformanceResponse` on
+ stdout. Loops to EOF. Core dispatch lives in
+ `cmd/conformance/core.lua` and is exercised directly by
+ `test/conformance_test.lua` so the inner dev loop doesn't need
+ Docker.
- [x] Run the canonical `conformance_test_runner` binary locally and
track the pass-rate as a numeric metric (regression gate).
`docker/conformance.Dockerfile` builds `conformance_test_runner`
from upstream protobuf v34.1 source (matching the host's
- `libprotoc 34.1`) and includes Tarantool 3 from the official deb;
+ `libprotoc 34.1`) and bundles Tarantool 3 from the official deb;
`just conformance` regenerates Lua then runs the harness against
`cmd/conformance-runner.lua` with the repo mounted as a volume.
- Watchlists at `test/conformance/known_failures.txt` (main suite)
- and `test/conformance/known_failures_text.txt` (text-format
- suite). Current baseline (2026-05-16):
- - Binary+JSON suite: 1478 ✓ / 1313 skipped / 15 expected fails
- - Text-format suite: 8 ✓ / 426 skipped / 0 expected fails
- JSON output runs end-to-end; remaining expected fails are
- Recommended-only edge cases (FieldMask round-trip,
- duplicate-field-name rejection, null-in-collection rejection,
- unknown-enum-name rejection, NullValue oneof validator).
- Text-format output is wired through `pb.text.encode` and the
- proto3 text suite is fully clean: SGROUP/EGROUP are tolerated in
- `wire.skip_field` and `pb.text` renders captured unknown bytes in
- numeric field-ID form under `opts.print_unknown_fields`. The 426
- still-skipped tests are proto2/editions message types we don't
- register. Text-format *input* parsing is still deferred. The
- `PB_CONFORMANCE_SKIP_JSON=1` env var still short-circuits JSON
- output if a new encoder bug crashes jsoncpp.
- CI wire-up pending — the image build is the long pole (~10–15 min
- on a clean cache).
+ Watchlists at `test/conformance/known_failures.txt` (binary +
+ JSON suite) and `test/conformance/known_failures_text.txt`
+ (text-format suite); **both are empty for the proto3 suites as
+ of 2026-05-16.**
+
+ Current baseline:
+ - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
+ - Text-format suite: **416 ✓ / 18 skipped / 0 failures**
+
+ The 1313 + 18 skipped all target
+ `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
+ is a separate slice — see [docs/codegen.md](../docs/codegen.md)
+ for what it would take.
+
+ Strict-validation closures landed across three commits on the
+ `text-conformance-output` branch:
+ - `pb.text.decode` — full grammar coverage (recursive-descent
+ parser, ~580 LOC; see
+ [docs/text_format_parser_brief.md](../docs/text_format_parser_brief.md)).
+ - `codec` -0.0 preservation — float/double `is_default_scalar`
+ and the inline-codegen elision both gained a sign-bit guard
+ (`1/v == math.huge`).
+ - `pb.json` strict-validation pass — duplicate-key rejection
+ (literal + camel/snake alias detection via a byte-walking
+ pre-scan), null-in-container rejection, unknown-enum-name
+ rejection (with `ignore_unknown_fields` opt for the
+ `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
+ `google.protobuf.NullValue` round-trip as JSON `null`,
+ strict FieldMask round-trip validation.
+
+ The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
+ short-circuit JSON output if a future encoder bug starts
+ crashing jsoncpp.
+- [ ] CI wire-up. The Docker image build is the long pole (~10–15 min
+ on a clean cache); a registry push from a scheduled job would
+ let CI runs reuse a warm cache.
- [x] Cross-impl interop: 18-fixture corpus in `test/interop/fixtures/`
produced by mainline `protoc --encode`; tests assert byte-for-byte
equality.
@@ 282,10 315,27 @@ fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
duplicate singular fields, and threads through the conformance
runner — `cmd/conformance/core.lua` no longer skips `text_payload`.
- Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 406 ✓
- / 18 skipped / 10 expected failures** (the 10 are `-0` float/double
- preservation; their root cause is in the codec, not the parser —
- see `test/conformance/known_failures_text.txt`).
+ Plugin gained a small `reserved_names` emitter so the parser can
+ match mainline TextFormat::Parser's "silently drop reserved" rule.
+ Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 416
+ ✓ / 18 skipped / 0 failures** (the 18 are the proto2 message-type
+ bucket; everything in scope passes).
+- [x] JSON strict-validation pass. Six classes of relaxation that the
+ proto3 JSON conformance corpus flagged are now enforced — together
+ with the -0 codec fix this empties `known_failures.txt`:
+ 1. Duplicate JSON keys (`{"foo":1,"foo":2}`) rejected via a
+ byte-walking pre-scan that runs before `json.decode`.
+ 2. camelCase / snake_case aliases of the same proto field
+ rejected via a per-message `field_seen` set.
+ 3. JSON `null` inside repeated arrays and map values rejected.
+ 4. Unknown enum *names* rejected by default; the
+ `ignore_unknown_fields=true` opt silently drops them (and
+ the conformance dispatch forwards this flag when
+ `req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST`).
+ 5. `google.protobuf.NullValue` JSON canonical form: literal
+ `null`, not the string `"NULL_VALUE"`. Null on a
+ NullValue-typed oneof member marks the oneof active.
+ 6. Strict FieldMask round-trip (see M3 entry).
- [x] `protoc-gen-tarantool-doc`: sibling Go plugin under
`cmd/protoc-gen-tarantool-doc/` that emits one Markdown file per
input `.proto`. Sections: header (package + imports), messages
M README.md => README.md +111 -54
@@ 15,7 15,8 @@ path. This project fills those gaps with:
## Status
-MVP — proto3 messages and enums, end-to-end round-trip verified.
+Proto3 conformance is closed: every `Required.*` and `Recommended.*` test in
+both the binary+JSON and text-format suites passes.
| Feature | State |
|----------------------------------|--------------|
@@ 26,9 27,11 @@ MVP — proto3 messages and enums, end-to-end round-trip verified.
| Enums (open semantics) | ✅ |
| 64-bit integers as LuaJIT cdata | ✅ |
| Two codegen modes (full + runtime) | ✅ |
+| Zero-copy lazy decode views | ✅ |
| `map<K,V>` (scalar/message values) | ✅ |
| `oneof` | ✅ |
-| proto3 explicit `optional` + has_/clear_ | ✅ |
+| proto3 explicit `optional` + `has_*`/`clear_*` | ✅ |
+| `-0.0` preserved for float/double | ✅ |
| gRPC service stubs (unary) | ✅ |
| gRPC streaming (server / client / bidi) | ✅ |
| Loopback / multiplex transport | ✅ |
@@ 36,17 39,18 @@ MVP — proto3 messages and enums, end-to-end round-trip verified.
| WKT: Duration, Empty, wrappers | ✅ |
| WKT: Struct, Value, ListValue | ✅ |
| WKT: Any (opaque + registry pack/unpack) | ✅ |
-| WKT: FieldMask | ✅ |
-| Byte-for-byte interop with `protoc` | ✅ (18 fixtures) |
-| Google conformance runner (`cmd/conformance-runner.lua`) | ✅ |
-| Runtime `.proto` parsing (`pb.parse`) | ✅ |
-| Runtime `FileDescriptorSet` ingest (`pb.from_pb`) | ✅ |
-| Markdown doc generator (`protoc-gen-tarantool-doc`) | ✅ |
-| proto3 JSON (`pb.json.encode`/`.decode`) | ✅ |
-| Text format printer (`pb.text.encode` / `<Msg>_text`) | ✅ (encode-only) |
-| Unknown-field passthrough (`_unknown_fields`) | ✅ |
+| WKT: FieldMask (strict round-trip) | ✅ |
+| Byte-for-byte interop with `protoc` (18 fixtures) | ✅ |
+| **Google conformance suite — proto3 binary+JSON** | **1493 ✓ / 0 failures** |
+| **Google conformance suite — proto3 text format** | **416 ✓ / 0 failures** |
+| Runtime `.proto` parsing (`pb.parse`) | ✅ |
+| Runtime `FileDescriptorSet` ingest (`pb.from_pb`) | ✅ |
+| Markdown doc generator (`protoc-gen-tarantool-doc`) | ✅ |
+| proto3 JSON (`pb.json.encode`/`.decode`) | ✅ |
+| Text format (`pb.text.encode` / `pb.text.decode`) | ✅ |
+| Unknown-field passthrough (`_unknown_fields`) | ✅ |
| Microbenchmark + alloc regression gate (`make bench`) | ✅ |
-| proto2 / editions | ❌ out of scope |
+| proto2 / editions | ❌ deferred (separate slice; see [docs/codegen.md](docs/codegen.md)) |
## Quick start
@@ 74,14 78,20 @@ For each message `Foo` the plugin emits:
```lua
local M = require('myapp.proto.foo')
-M.Foo_descriptor -- the descriptor table consumed by the runtime
-M.Foo_new(t) -- returns t (or {}); placeholder for future validation
-M.Foo_encode(t) -- table -> wire bytes (string)
-M.Foo_decode(b) -- wire bytes (string) -> table
-M.Foo_decode_lazy(b) -- wire bytes -> MessageView (zero-copy view)
-M.Foo_text(t, opts) -- table -> protoc-style text format (debug printer)
+M.Foo_descriptor -- the descriptor table consumed by the runtime
+M.Foo_new(t) -- returns t (or {}); placeholder for future validation
+M.Foo_encode(t) -- table -> wire bytes (string)
+M.Foo_decode(b) -- wire bytes (string) -> table
+M.Foo_decode_lazy(b) -- wire bytes -> MessageView (zero-copy view)
+M.Foo_text(t, opts) -- table -> protoc-style text format (debug printer)
+M.Foo_has_<field>(t) -- only emitted for proto3 explicit-`optional` fields
+M.Foo_clear_<field>(t) -- same
```
+Text-format **decoding** is exposed on the runtime as `pb.text.decode(desc,
+text, opts)` (no per-message wrapper — it's used from a few places, like
+the conformance runner, and didn't warrant codegen surface).
+
For each enum `Color`:
```lua
@@ 94,23 104,64 @@ Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers
`uint64_t` cdata — lossless and the same convention used by Tarantool's
`net.box`, `msgpack`, and built-in `protobuf` modules.
+**Three API modes live side-by-side.** Same descriptor, three call shapes —
+the inline (full) generated API is the default, the descriptor-driven runtime
+API is for dynamic schemas, and the lazy API is a zero-copy view for sparse
+reads and proxy / router workloads. See [docs/api-modes.md](docs/api-modes.md)
+for when to pick which, with measured trade-offs.
+
## Layout
```
-cmd/protoc-gen-tarantool/ # Go plugin
- main.go # reads CodeGeneratorRequest, hands off to gen
- internal/gen/ # codegen package
-runtime/pb/ # Lua runtime (require('pb'))
- init.lua # public surface
- wire.lua # varint / zigzag / fixed / float / LEN
- codec.lua # descriptor-driven encode/decode
-options/tarantool/ # custom proto options
- tarantool.proto # (tarantool.lua_package) file option
-examples/proto/ # demo .proto inputs
-examples/expected/ # generated output (committed for inspection)
-test/ # roundtrip test
+cmd/
+ protoc-gen-tarantool/ Go plugin (the codegen)
+ main.go reads CodeGeneratorRequest, hands off to gen
+ internal/gen/ per-message emission for both modes
+ protoc-gen-tarantool-doc/ separate Go plugin that emits Markdown docs
+ conformance/ Lua conformance dispatch (loaded by the runner)
+ conformance-runner.lua stdin/stdout framing for `conformance_test_runner`
+
+runtime/pb/ pure-Lua runtime (`require('pb')`)
+ init.lua public surface
+ wire.lua varint / zigzag / fixed / float / LEN primitives
+ codec.lua descriptor-driven encode/decode
+ lazy.lua zero-copy MessageView / ArrayView / MapView
+ text.lua text format encode + decode (proto3)
+ json.lua proto3 JSON encode + decode (strict)
+ wkt.lua Timestamp / Duration / Empty / Wrappers /
+ Struct / Value / ListValue / Any / FieldMask
+ grpc.lua transport interface + loopback / multiplex
+ parser.lua pure-Lua proto3 schema parser (.proto → AST)
+ dynamic.lua AST → descriptor module
+ fileset.lua FileDescriptorSet bytes → descriptor module
+ descriptor_pb.lua hand-built descriptors of descriptor.proto
+
+options/tarantool/ custom proto file options
+ tarantool.proto (tarantool.lua_package) — Lua require path override
+
+examples/proto/ demo .proto inputs
+examples/expected/ generated output for full + runtime modes
+ (both committed for inspection + parametrized tests)
+test/ luatest groups, conformance regressions, fixtures
+docs/ codegen notes, API mode comparison, design briefs
+bench/ per-helper bench + JIT-trace gate + alloc baseline
```
+## Documentation
+
+- **[docs/api-modes.md](docs/api-modes.md)** — when to use `Foo_encode` (full),
+ `pb.encode(desc, t)` (runtime / reflect), or `pb.decode_lazy(desc, b)` (lazy
+ view). Measured allocation + throughput trade-offs.
+- **[docs/codegen.md](docs/codegen.md)** — plugin internals, descriptor shape
+ contract, how to add a new wire type or scalar, the LuaJIT hot-path rules
+ the generated code observes.
+- **[docs/text_format_parser_brief.md](docs/text_format_parser_brief.md)** —
+ retrospective brief on the text-format parser slice (`pb.text.decode`).
+- **PLAN.md** — phased roadmap and per-feature design notes.
+- **CLAUDE.md** — invariants and conventions enforced across the codebase
+ (no `pairs()` on hot paths, 64-bit ints as cdata, SoA over AoS for large
+ index structures, etc.).
+
## Conformance
`cmd/conformance-runner.lua` speaks the [Google protobuf conformance
@@ 132,30 183,29 @@ just conformance
```
(Mounts the repo into the container — generated Lua from `make gen` on the
-host is what gets tested.) Known failures live in `test/conformance/known_failures.txt` (binary +
-JSON suite) and `test/conformance/known_failures_text.txt` (text-format
-suite); the runner exits zero only when actual failures match those
-lists exactly.
-
-Current baseline (2026-05-15, protobuf v34.1):
-
-| Suite | Successes | Skipped | Expected failures |
-|-------|-----------|---------|-------------------|
-| Binary + JSON | 1389 | 1313 | 79 |
-| Text-format | 0 | 430 | 4 |
-
-JSON output runs end-to-end through the conformance harness. The
-remaining failures are mostly canonical-form edge cases — Duration /
-Timestamp serialization rules, double-precision formatting, NaN
-canonicalization, JSON-input rejection rules — listed in
-`test/conformance/known_failures.txt`. Set `PB_CONFORMANCE_SKIP_JSON=1`
-to short-circuit JSON output requests with `skipped` if a new encoder
-bug starts crashing jsoncpp again.
-
-The runner currently supports `protobuf_test_messages.proto3.TestAllTypesProto3`
-in both protobuf and JSON formats; proto2 / editions / JSPB / text-format
-cases return `skipped`. The self-test in `test/conformance_test.lua`
-exercises the runner with crafted requests on every `make test` run.
+host is what gets tested.) Known failures live in
+`test/conformance/known_failures.txt` (binary + JSON suite) and
+`test/conformance/known_failures_text.txt` (text-format suite); both are
+empty for the proto3 suites as of 2026-05-16.
+
+Current baseline (2026-05-16, protobuf v34.1):
+
+| Suite | Successes | Skipped | Expected failures | Unexpected |
+|-------|-----------|---------|-------------------|------------|
+| Binary + JSON | **1493** | 1313 | 0 | 0 |
+| Text-format | **416** | 18 | 0 | 0 |
+
+The 1313 + 18 skipped tests all target
+`protobuf_test_messages.proto2.TestAllTypesProto2`, which we don't generate
+Lua for. Proto2 support is a separate slice — see
+[docs/codegen.md#proto2-deferral](docs/codegen.md) for what it would take.
+
+The runner supports `protobuf_test_messages.proto3.TestAllTypesProto3` in
+binary, JSON, and text-format input/output, including the
+`JSON_IGNORE_UNKNOWN_PARSING_TEST` category (forwarded as
+`ignore_unknown_fields=true` to `pb.json.decode`). The self-test in
+`test/conformance_test.lua` exercises the runner with crafted requests on
+every `make test` run.
[gconf]: https://github.com/protocolbuffers/protobuf/tree/main/conformance
@@ 181,6 231,13 @@ Current baseline (LuaJIT 2.1, hello.Person):
| 10 KB | 47.5 / 48.6 KB | 59.5 / 59.6 KB |
| 100 KB | 444 / 445 KB | 568 / 568 KB |
+Lazy decode trades a much higher *index-build* cost (one MessageView
+table + four SoA arrays) for near-zero allocation on subsequent
+field reads — best when you touch a small fraction of fields, or when
+you re-encode mostly-unchanged messages (proxy / router workloads). See
+[docs/api-modes.md](docs/api-modes.md) for the bench numbers and the
+cross-over point.
+
## Why named `pb` instead of `protobuf`?
Tarantool's loader prefers the built-in `require('protobuf')` over any
A docs/api-modes.md => docs/api-modes.md +265 -0
@@ 0,0 1,265 @@
+# API modes: full, runtime, lazy
+
+The same descriptor table feeds three encode/decode call shapes. They are
+**not exclusive** — every generated `_pb.lua` exposes all three (`_encode`,
+`_decode`, `_decode_lazy`), and `pb.encode` / `pb.decode` / `pb.decode_lazy`
+work directly against any descriptor. Picking is per-call-site, not
+per-message and not per-build.
+
+## TL;DR
+
+| Mode | Call shape | Best for | Pays |
+|---|---|---|---|
+| **Full** (inline codegen) | `M.Foo_encode(t)` / `M.Foo_decode(b)` | Hot RPC paths, anything you call >1k times/sec on a known type | Larger generated `_pb.lua`; one wrapper per message |
+| **Runtime** (descriptor-driven / reflect) | `pb.encode(desc, t)` / `pb.decode(desc, b)` | Dynamic schemas (`pb.parse`, `pb.from_pb`, reflection); cases where you only have a descriptor at runtime | One extra indirection per field (descriptor table dispatch) |
+| **Lazy** (zero-copy view) | `M.Foo_decode_lazy(b)` / `pb.decode_lazy(desc, b)` | Proxy / router shapes that touch a few fields and re-encode; sparse reads from large payloads | Higher fixed cost per decode (index build); per-field GC pressure on dense reads |
+
+If you don't know which to use: **start with full**. Switch to lazy when
+profiling shows you're decoding more than you read; switch to runtime when
+the schema isn't known at build time.
+
+## Full (inline codegen)
+
+The default, generated by `mode=full` (the codegen default). For each
+message the plugin emits dedicated `_encode` and `_decode` functions with
+the wire calls and tag bytes inlined.
+
+```lua
+local hello = require('myapp.proto.hello')
+
+local bytes = hello.Person_encode({
+ name = 'Alice',
+ user_id = require('ffi').cast('uint64_t', 42),
+ emails = {'a@x', 'b@x'},
+})
+local p = hello.Person_decode(bytes)
+```
+
+Under the hood the generated `Person_encode` looks like:
+
+```lua
+function M.Person_encode(t)
+ if type(t) ~= 'table' then
+ error("expected table for hello.Person, got " .. type(t), 0)
+ end
+ local out, n = {}, 0
+ local v
+ v = t.name
+ if v ~= nil and v ~= '' then
+ n = n + 1; out[n] = "\x0a" -- precomputed tag bytes
+ n = n + 1; out[n] = wire.encode_varint(#v)
+ n = n + 1; out[n] = v
+ end
+ v = t.user_id
+ if v ~= nil and v ~= 0 then
+ n = n + 1; out[n] = "\x10"
+ n = n + 1; out[n] = wire.encode_uint64(v)
+ end
+ -- ... one branch per field ...
+ return table.concat(out)
+end
+```
+
+No descriptor table lookups, no kind dispatch, no `pairs()` over a hash
+on the hot path. Tag bytes are precomputed Lua string literals. The LuaJIT
+trace compiler sees a flat sequence of monomorphic wire calls and
+specializes the whole encode into a single trace.
+
+This is what `make bench` measures by default. The
+`bench/baseline.json` numbers under the "full" column are this path.
+
+## Runtime (descriptor-driven / reflect)
+
+The plugin's `mode=runtime` emits thin wrappers that delegate to
+`pb.encode(desc, t)` and `pb.decode(desc, b)`:
+
+```lua
+function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
+function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end
+```
+
+You can also call those directly with any descriptor — that's the
+"reflect" use case:
+
+```lua
+local pb = require('pb')
+
+-- Parse a .proto file at runtime; no codegen, no Go plugin.
+local mod = pb.parse(io.open('hello.proto'):read('*a'))
+local desc = mod.Person_descriptor
+
+-- Same encode / decode surface, just bound to a runtime-built descriptor.
+local bytes = pb.encode(desc, {name = 'Alice'})
+local p = pb.decode(desc, bytes)
+
+-- Or ingest a binary FileDescriptorSet from protoc --descriptor_set_out:
+local set = pb.from_pb(io.open('build/all.pb', 'rb'):read('*a'))
+local desc = set.lookup('hello.Person')
+local p = pb.decode(desc, bytes)
+```
+
+The descriptor table is the contract:
+
+```lua
+{
+ name = 'hello.Person',
+ fields = {
+ {name='name', id=1, kind='scalar', proto_type='string'},
+ {name='user_id', id=2, kind='scalar', proto_type='uint64'},
+ {name='emails', id=3, kind='scalar', proto_type='string', repeated=true},
+ {name='status', id=4, kind='enum', enum=<enum_descriptor>},
+ {name='address', id=5, kind='message', message=<other_descriptor>},
+ {name='ages_by_nickname', id=13, kind='map',
+ key={kind='scalar', proto_type='string'},
+ value={kind='scalar', proto_type='int32'}},
+ -- modifiers: repeated, packed, oneof, optional
+ },
+ field_by_id = {[1]=<field>, ...}, -- filled by pb.finalize_message
+ field_by_name = {name=<field>, ...},
+ oneofs = {<oneof_name> = {<field_names>}},
+ reserved_names = {[<name>]=true}, -- for text-format parser
+}
+```
+
+Three producers emit this shape — generated codegen (runtime mode),
+`pb.parse` (from .proto source), `pb.from_pb` (from
+FileDescriptorSet bytes). All three flow through the same `pb.codec`.
+
+The `pb.finalize_message(desc)` helper builds `field_by_id` /
+`field_by_name` / `oneofs_list` and attaches per-field `_writer` /
+`_reader` specializations. Always call it on hand-rolled descriptors.
+
+**Cost vs full**: per-field table lookup + kind dispatch. Allocation
+overhead is one extra `pairs()` walk in encode and a small per-field
+function-call cost; on `make bench` numbers the runtime mode is ~5-15%
+slower for encode and within noise for decode (the decoder fast paths
+are shared via `f._reader` closures).
+
+WKTs (`google.protobuf.Timestamp` etc) carry their own `desc.encode` /
+`desc.decode` overrides; the runtime codec dispatches to them in place
+of the generic field walk. That's the extension point for any
+descriptor that wants custom handling without a special case in the
+codec.
+
+## Lazy (zero-copy view)
+
+`pb.decode_lazy(desc, bytes)` returns a `MessageView` instead of a Lua
+table. Nothing past the field index gets decoded eagerly: scalar
+materialization happens on `:get`, repeated/map fields become
+`ArrayView` / `MapView` sub-views, sub-messages become nested
+`MessageView`s on access.
+
+```lua
+local view = hello.Person_decode_lazy(bytes)
+
+-- Direct field reads (decoded on demand, then cached on the view).
+local name = view:get('name')
+local uid = view:get('user_id')
+
+-- Repeated field: ArrayView.
+local emails = view:get('emails') -- ArrayView, no per-element table
+for i, e in emails:iter() do
+ print(i, e)
+end
+print('count:', emails:len())
+
+-- Map field: MapView.
+local ages = view:get('ages_by_nickname')
+print(ages:get('alice'))
+
+-- Sub-message: a nested MessageView.
+local addr = view:get('address')
+print(addr:get('city'))
+
+-- Presence + oneof which-branch checks without forcing a decode.
+view:has('user_id') -- bool
+view:which('outcome') -- string?
+
+-- Iterate fields actually present on the wire (skip-aware).
+for name, val in view:iter() do print(name, val) end
+
+-- Mutate, then re-encode. Untouched fields are spliced byte-for-byte
+-- from the original payload; only dirty fields run through encode.
+view:set('user_id', require('ffi').cast('uint64_t', 99))
+local new_bytes = view:encode()
+
+-- Force a fully-materialized table when you actually need one.
+local t = view:totable()
+```
+
+### When lazy wins
+
+- **Sparse reads** — payload is large, you only need a few fields.
+ Index build is ~one Lua table + four SoA int arrays sized to
+ `<wire-segment-count>`. Per-field reads are O(1) after the index.
+- **Mostly-passthrough re-encode** — proxy or router shapes that
+ decode, touch a couple of fields, and re-encode. The byte-splice
+ path skips encode entirely for untouched fields; only dirty fields
+ pay the encode cost. On the 100 KB hello.Person bench this puts the
+ mutate-then-reencode workload at **1.09-1.26× of eager** despite the
+ extra index build.
+- **Avoiding intermediate table churn** when iterating large repeated
+ fields: `ArrayView:iter()` doesn't allocate a flat array; values
+ flow through one at a time.
+
+### When lazy loses
+
+- **Dense reads** — if you read every field, the index build is wasted
+ work and each `:get` adds a Lua-call boundary the eager decoder
+ avoided. Eager wins by ~10-30% on full-field walks.
+- **Tiny messages** — for a 10 B payload the index build dominates.
+ The cross-over (on hello.Person) is around 1 KB.
+- **Anything that wants the result as a plain Lua table** — `:totable()`
+ reverses the win.
+
+### Implementation note (why SoA)
+
+The lazy index is structure-of-arrays, not array-of-structs: for
+N wire segments the indexer keeps four int arrays of size N (`id`,
+`tag_start`, `val_start`, `next_start`) instead of N tiny tables of 4
+keys each. Tiny per-entry tables in LuaJIT carry header overhead +
+hash dispatch costs that dominate for large N. Switching from AoS to
+SoA on the protobuf-lazy slice took sparse-read from 0.66× of eager
+back to 1.0×-1.16×.
+
+If you write your own index-style structure with many small entries,
+follow the same pattern. See `runtime/pb/lazy.lua`'s `index_bytes`
+for the reference shape.
+
+## Mixing modes
+
+All three modes interoperate freely:
+
+- A `MessageView` from `decode_lazy` can be `:set(...)` and `:encode()`'d
+ back to wire bytes that `pb.decode` (eager) parses identically.
+- A descriptor from `pb.parse` (runtime mode) plugs into
+ `pb.decode_lazy` exactly like a generated descriptor.
+- The codec's `desc.encode` / `desc.decode` override mechanism lets WKT
+ and user-registered descriptors short-circuit any of the three paths
+ without special cases in the codec.
+
+The contract is the descriptor table — the codec, both codegen modes,
+the dynamic parser, the JSON codec, the text codec, and the lazy view
+all consume the same shape. That's the design property that lets these
+three APIs coexist without forks.
+
+## Choosing per call site, not per build
+
+Both codegen modes always emit `Foo_decode_lazy` and `Foo_text`
+wrappers. The plugin's `mode=full`/`mode=runtime` switch only changes
+how `Foo_encode` / `Foo_decode` are emitted — everything else lives in
+the shared runtime. So you can:
+
+- Generate everything with `mode=full` (the default) and still call
+ `pb.encode(desc, t)` against the descriptor when you need a
+ descriptor-driven path.
+- Generate everything with `mode=runtime` if you want minimum
+ generated-code size, and pay the modest descriptor-dispatch overhead
+ uniformly.
+- Reach for lazy on a per-call basis when the workload shape calls for
+ it — typically in handlers that operate on large messages but only
+ inspect a few fields.
+
+The test suite runs every behavior test against both codegen modes
+(`full` and `runtime`) via the same descriptor; cross-mode parity is
+pinned by `parity.full_vs_runtime.*` test groups.
A docs/codegen.md => docs/codegen.md +391 -0
@@ 0,0 1,391 @@
+# Codegen notes
+
+How `protoc-gen-tarantool` works, what it emits, and where to extend it.
+
+This is a companion to [docs/api-modes.md](api-modes.md). API-modes
+covers the *user-facing* difference between full / runtime / lazy; this
+doc covers what the plugin does to produce the code those modes rely on.
+
+## The pipeline
+
+```
+.proto sources
+ │
+ ▼ protoc (mainline binary, host-installed)
+CodeGeneratorRequest (proto descriptor bytes on stdin)
+ │
+ ▼ cmd/protoc-gen-tarantool/main.go
+ │ advertises FEATURE_PROTO3_OPTIONAL so protoc surfaces
+ │ explicit-optional fields (without this flag, it omits them)
+ │
+ ▼ cmd/protoc-gen-tarantool/internal/gen.GenerateFile
+ │
+ ├─ ModeFull → inline `_encode` / `_decode` (the default)
+ └─ ModeRuntime → wrappers that delegate to pb.encode / pb.decode
+
+one `<lua_pkg>.lua` per input `.proto`
+```
+
+The plugin only handles `syntax = "proto3"`. Proto2 input is rejected
+at the top of `GenerateFile`. See [proto2 deferral](#proto2-deferral)
+for the rationale.
+
+## CLI options
+
+Passed via `protoc --tarantool_opt=<key>=<value>,...`:
+
+| Option | Values | Meaning |
+|---|---|---|
+| `mode` | `full` (default) or `runtime` | Inline `_encode`/`_decode` vs descriptor-delegating wrappers. See [api-modes.md](api-modes.md). |
+| `prefix` | any Lua-require path | Prepended to every generated module's require path **and** its on-disk subpath. The Makefile uses this to generate both modes side-by-side into `examples/expected/{full,runtime}/`. |
+
+The `(tarantool.lua_package)` file option (defined in
+`options/tarantool/tarantool.proto`) overrides the per-file Lua module
+path:
+
+```proto
+import "tarantool/tarantool.proto";
+option (tarantool.lua_package) = "myapp.proto.foo";
+```
+
+Without that option the path mirrors `package` (e.g.
+`package my.app; foo.proto` → `my/app/foo_pb.lua`,
+`require('my.app.foo_pb')`).
+
+## What gets emitted per .proto
+
+```lua
+-- Code generated by protoc-gen-tarantool. DO NOT EDIT.
+-- source: hello.proto
+-- syntax: proto3
+
+local pb = require('pb')
+local wire = pb.wire
+
+local M = {}
+
+-- 1. Enums first (no forward-reference problems).
+M.Status_descriptor = pb.enum('hello.Status', { ACTIVE=0, BANNED=1, ... })
+M.Status = M.Status_descriptor.by_name
+
+-- 2. Predeclare all message descriptor tables. Self-references and
+-- mutual recursion resolve here because every descriptor is named
+-- before any field references it.
+M.Person_descriptor = {name = 'hello.Person'}
+M.Address_descriptor = {name = 'hello.Address'}
+M.Result_descriptor = {name = 'hello.Result'}
+
+-- 3. Fill in fields[] per message, then call pb.finalize_message.
+M.Person_descriptor.fields = {
+ {name='name', id=1, kind='scalar', proto_type='string'},
+ {name='user_id', id=2, kind='scalar', proto_type='uint64'},
+ -- ...
+}
+M.Person_descriptor.oneofs = { outcome = {'text', 'code', 'details'} }
+M.Person_descriptor.reserved_names = { ['old_field'] = true }
+pb.finalize_message(M.Person_descriptor)
+
+-- 4. EmmyLua / lua-language-server type annotations.
+---@class hello.Person
+---@field name string
+---@field user_id integer
+-- ...
+
+-- 5. Wrappers: _new / _encode / _decode / _decode_lazy / _text
+-- + _has_<field> / _clear_<field> for explicit-optional fields.
+function M.Person_new(t) return t or {} end
+function M.Person_encode(t) -- inline body in full mode, delegation in runtime mode
+function M.Person_decode(b) -- same
+function M.Person_decode_lazy(b) return pb.decode_lazy(M.Person_descriptor, b) end
+function M.Person_text(t, opts) return pb.text.encode(M.Person_descriptor, t, opts) end
+
+-- 6. Services (mode-independent — they reuse the per-message wrappers).
+M.Greeter_service = { ... }
+function M.Greeter_client(transport) ... end
+function M.Greeter_server(impl) ... end
+
+return M
+```
+
+Generation order matters: enums and predeclared descriptors come before
+field tables so cross-references inside `fields = {...}` resolve in one
+pass.
+
+## The descriptor table — the contract
+
+The shape consumed by the runtime is:
+
+```lua
+{
+ name = 'pkg.Foo',
+ fields = {
+ {name='x', id=1, kind='scalar', proto_type='int32'},
+ {name='y', id=2, kind='message', message=<other_descriptor>},
+ {name='z', id=3, kind='enum', enum=<enum_descriptor>},
+ {name='m', id=4, kind='map', key=<sub_field>, value=<sub_field>},
+ -- modifiers: repeated, packed, oneof, optional
+ },
+ -- Filled in by pb.finalize_message:
+ field_by_id = {[1]=<field>, ...},
+ field_by_name = {x=<field>, ...},
+ oneofs_list = {{name=<n>, members={<names>}}, ...},
+ -- Optional, set by the codegen:
+ oneofs = {<name> = {<member-names>}}, -- input form, flattened above
+ reserved_names = {[<name>] = true}, -- for text-format parser
+ encode = <fn>, decode = <fn>, -- override hook (WKT use this)
+ text = <fn>, -- text-format encode override
+}
+```
+
+**Three producers emit this exact shape.** They live in different
+codepaths but consume the same runtime:
+
+1. `cmd/protoc-gen-tarantool/internal/gen` — build-time codegen.
+2. `runtime/pb/dynamic.lua` — runtime synthesis from an AST that
+ `runtime/pb/parser.lua` produces from `.proto` source.
+3. `runtime/pb/fileset.lua` — runtime synthesis from a
+ `FileDescriptorSet` binary (`protoc --descriptor_set_out=...`).
+
+Hand-rolled WKT descriptors in `runtime/pb/wkt.lua` use the same shape
+but additionally set `desc.encode` / `desc.decode` to take over the wire
+path entirely (Timestamp, Duration, Wrappers, Struct, Value, ListValue,
+Any, FieldMask, Empty).
+
+**Pinning this shape is what lets us run every behavior test against
+both codegen modes.** The test harness in `test/protobuf_test.lua` and
+friends parametrizes by mode (`{'full', 'runtime'}`) and runs the same
+assertions against `require('full.hello.hello_pb')` and
+`require('runtime.hello.hello_pb')`.
+
+## Inline (full) mode — what it looks like
+
+`inline.go` walks each field and emits Lua statements that call into
+`runtime/pb/wire.lua` primitives directly. Tag bytes are precomputed at
+codegen time as Lua string literals so the runtime never re-encodes
+them.
+
+```lua
+function M.Person_encode(t)
+ if type(t) ~= 'table' then
+ error("expected table for hello.Person, got " .. type(t), 0)
+ end
+ local out, n = {}, 0
+ local v
+
+ -- field 1: name (string, singular)
+ v = t.name
+ if v ~= nil and v ~= '' then -- proto3 default elision
+ n = n + 1; out[n] = "\x0a" -- tag(1, LEN), precomputed
+ n = n + 1; out[n] = wire.encode_varint(#v)
+ n = n + 1; out[n] = v
+ end
+
+ -- field 11: weight_kg (double, singular)
+ v = t.weight_kg
+ if v ~= nil and (v ~= 0 or 1/v == -math.huge) then -- -0 preserved
+ n = n + 1; out[n] = "\x59"
+ n = n + 1; out[n] = wire.encode_double(v)
+ end
+
+ -- ... oneof pre-pass picks active branch; map fields walk pairs(); etc.
+
+ return table.concat(out)
+end
+```
+
+The decoder is similarly inlined:
+
+```lua
+function M.Person_decode(b)
+ local result = {}
+ local pos, len = 1, #b
+ while pos <= len do
+ local id, wt
+ id, wt, pos = wire.decode_tag(b, pos)
+ if id == 1 then -- name
+ result.name, pos = wire.decode_string(b, pos)
+ elseif id == 2 then -- user_id
+ result.user_id, pos = wire.decode_uint64(b, pos)
+ -- ... if-elseif chain on field id ...
+ else
+ pos = wire.skip_field(b, pos, wt, id) -- unknown
+ end
+ end
+ return result
+end
+```
+
+**Why an `if-elseif` chain rather than a dispatch table?** LuaJIT
+compiles short chains into a tight branch sequence on a single trace.
+The crossover where a jump table would win is at field counts higher
+than anything we hit in practice; pinned by `bench/jit_trace.lua`.
+
+## Runtime mode — the thin wrappers
+
+`mode=runtime` skips the inline emission and produces:
+
+```lua
+function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
+function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end
+```
+
+`pb.encode` / `pb.decode` (in `runtime/pb/codec.lua`) walk the
+descriptor table to pick wire primitives at runtime. Slower per
+field — a hash lookup + closure call instead of a hard-coded branch —
+but smaller generated output (no per-message `_encode` / `_decode`
+bodies) and useful when the same code path needs to handle dynamic
+descriptors.
+
+`pb.finalize_message` (in `runtime/pb/init.lua`) builds two
+specialization layers on top of the descriptor at finalize-time:
+
+1. **Per-field writer closures** (`f._writer`): pre-bound to the
+ field's tag, encoder, and default predicate. The encode loop calls
+ `f._writer(data, out)` per field instead of dispatching on
+ `f.kind` / `f.proto_type` inside the loop.
+2. **Per-field reader closures** (`f._reader`): same idea for decode,
+ handling typed value extraction + repeated bookkeeping + nested
+ merge + oneof sibling clearing in one closure per field.
+
+These specializations bridge most of the gap between runtime mode and
+full mode for the *decoder*; encode is still meaningfully slower in
+runtime mode because the field walk goes through one extra Lua-function
+boundary per field.
+
+## The hot-path rules the generated code observes
+
+These come from CLAUDE.md and are enforced across both the codegen and
+the runtime. New code (codegen output or hand-written) needs to follow
+them or the LuaJIT trace compiler bails out.
+
+- **No `pairs()` on hot paths.** `pairs()` over a hash compiles to
+ bytecode `ISNEXT`, which is NYI in Tarantool's LuaJIT 2.1 fork —
+ the trace aborts and the code path falls off the JIT. Map fields are
+ the one accepted exception; the JIT-trace gate in `bench/jit_trace.lua`
+ pins the limitation. When a descriptor needs hash-shaped lookup state
+ (e.g. oneof grouping), build a parallel array view at
+ `pb.finalize_message` time (`desc.oneofs_list`) and iterate the array
+ on the hot path.
+- **64-bit integers stay as cdata.** `int64`/`uint64`/`sint64`/
+ `fixed64`/`sfixed64` are LuaJIT `int64_t` / `uint64_t` cdata
+ everywhere. Same convention as Tarantool's `msgpackffi`, `net.box`,
+ `box.tuple`, and built-in `protobuf`. Don't narrow through
+ `tonumber` — precision loss past 2^53 silently corrupts IDs and
+ timestamp nanos.
+- **`ipairs` / `for i=1,#t`, never `pairs()`, on generated arrays.**
+ Repeated fields are 1-based contiguous Lua arrays. Generated encode
+ walks them with `for i = 1, #v do`.
+- **Keep hot helpers small.** Adding "more fast paths" to a helper that
+ the JIT already inlines (e.g. extending `encode_varint`'s 1-byte
+ fast path with 2/3/4-byte branches) can push the function past the
+ LuaJIT inline budget — parent traces stop inlining it and the
+ dominant 1-byte case regresses by ~30%. Multi-byte fast paths live
+ in the *slow* helper, not the inlined one.
+- **SoA over AoS for many-entry index structures.** For per-element
+ records where N is in the thousands, use parallel int arrays, not
+ one small Lua table per entry. Each per-entry table is a separate
+ allocation with header overhead and hash dispatch; for large N the
+ allocations dominate. See `runtime/pb/lazy.lua`'s `index_bytes`
+ for the reference shape.
+- **WKT routing.** When codegen sees a field referencing a
+ `google.protobuf.*` type, it emits a reference to
+ `pb.wkt.<Name>_descriptor` instead of treating it as an ordinary
+ cross-file import. The plugin does NOT generate a Lua module for
+ imported WKT `.proto` files — those are fulfilled by
+ `runtime/pb/wkt.lua`.
+
+## Plugin source layout
+
+```
+cmd/protoc-gen-tarantool/
+ main.go # reads CodeGeneratorRequest from stdin, sets up
+ # the plugin options + FEATURE_PROTO3_OPTIONAL,
+ # dispatches to gen.GenerateFile per .proto
+ internal/gen/
+ gen.go # per-file orchestration + runtime-mode wrappers,
+ # imports collection, enum + message emission,
+ # descriptor field tables, oneofs, reserved_names
+ inline.go # full-mode inline _encode / _decode body emission,
+ # field-kind dispatch, repeated / map / oneof shapes
+ service.go # gRPC client / server factories
+ name.go # Lua name + path mangling rules
+ types.go # protoreflect.Kind → scalar name mapping
+ options.go # (tarantool.lua_package) file option lookup
+ emmylua.go # EmmyLua / lua-language-server type annotation
+ # emission (---@class, ---@field, ---@param)
+```
+
+**Adding a new wire type or scalar** means touching three places:
+
+1. `runtime/pb/wire.lua` — add the `encode_xxx` / `decode_xxx`
+ primitives + a `TYPE_INFO[xxx]` entry. That's the single source of
+ truth: the codec, both codegen modes, and the dynamic parser all
+ pick it up from there.
+2. `cmd/protoc-gen-tarantool/internal/gen/types.go` —
+ `scalarName(k)` mapping for the new `protoreflect.Kind`.
+3. `cmd/protoc-gen-tarantool/internal/gen/inline.go` — emission for
+ the new type (singular + repeated + packed branches).
+
+If the new type is `LEN`-typed (string/bytes-shaped), check the
+existing string/bytes paths for the split-emission pattern
+(`tag + varint(#v) + body` as three separate `out` slots instead of
+one concatenated `wire.encode_len(v)` call) — that avoids a per-field
+string allocation.
+
+## Sibling plugin: protoc-gen-tarantool-doc
+
+`cmd/protoc-gen-tarantool-doc/` is a second Go plugin that consumes
+the same `CodeGeneratorRequest` and emits one Markdown file per input
+`.proto`. Build with `make build-doc`; sample docs land in
+`examples/docs/` via `make gen-docs`. It's deliberately separate from
+the Lua codegen to keep the codegen plugin small.
+
+## Proto2 deferral
+
+The plugin rejects `syntax = "proto2"` outright. The conformance
+suites' 1331 skipped tests are all `TestAllTypesProto2`.
+
+Proto2 support is genuinely a separate slice. The wire format is
+identical to proto3 (with the one exception of `group`s — wire-types 3
+and 4, SGROUP/EGROUP, which we already tolerate on the skip path but
+don't encode/decode). The schema features that *don't* exist in proto3
+are what would need work:
+
+- **`required` fields** — codec must error on missing required at both
+ encode and decode (we currently treat absence as default).
+- **`optional` everywhere** — proto2 fields are presence-tracked by
+ default. proto3's explicit-optional path already handles this; just
+ needs the plugin to mark every proto2 field as `optional=true`.
+- **Extensions** (`extensions 100 to 199;` + `extend Foo {...}`)
+ — proto2-only mechanism for adding fields to messages defined
+ elsewhere. Round-trip overlaps with our existing
+ `_unknown_fields` machinery but explicit support needs a separate
+ descriptor table per extension and a way to expose it on user
+ messages.
+- **Groups** (`optional group Foo = 1 { ... }`) — actual encode/decode
+ rather than the skip-only tolerance we have today, plus a different
+ text-format rendering (`Foo { ... }` with the group's capitalized
+ name instead of the submessage's field name).
+- **Custom defaults** (`[default = X]`) — only matter when callers
+ query "what's the default value of field X"; for round-trips they
+ can be ignored.
+- **`MessageSet`** wire format — a few specific tests.
+
+Realistic phasing (if you take this on):
+
+1. Drop the proto3-only gate in the plugin; mark proto2 fields as
+ `optional=true`; skip extensions / groups / required harmlessly
+ (warn but emit the rest of the message). Re-run conformance —
+ should unlock most of the 1007 binary tests and ~200 of the JSON
+ tests immediately because the wire format is the same.
+2. Implement group encode/decode + the text-format group rendering.
+ Picks up the remaining text tests and a handful of binary tests.
+3. Required-field validation. Small, targeted at the
+ `TestAllRequiredTypesProto2` cases.
+4. Extensions, MessageSet, custom defaults — long tail, optional
+ depending on demand.
+
+This is a meaningfully larger slice than the typical addition here —
+plan on a few sessions, not an evening. The trade-off is "proto2 is
+mostly legacy" (see the README's status table for who actually uses
+it) vs "1331 extra ✓ in the conformance numbers looks good".