@@ 1,724 0,0 @@
-# tarantool-protobuf — Roadmap & Test Plan
-
-This document tracks the long-form design + testing strategy for
-`protoc-gen-tarantool` and the `pb` Lua runtime. Short-term task tracking
-lives in the in-session task list; this file is the slow-changing record.
-
-## 1. Vision
-
-A first-class Protocol Buffers + gRPC stack for Tarantool that:
-
-- Generates idiomatic Lua from `.proto` files with **two modes** (full inline
- vs descriptor-driven runtime) so users can pick speed-vs-flexibility.
-- Interoperates byte-for-byte with mainline protoc implementations (Go,
- Python, C++) — passes Google's protobuf conformance suite for proto3.
-- Ships with a small, FFI-aware runtime that respects Tarantool conventions
- (LuaJIT cdata for 64-bit ints, no global state, no monkey-patching).
-- Is comfortable to use from real Tarantool apps: clean integration with
- `net.box`, `fiber`, `box.session`, and the 3.x config framework.
-
-## 2. Current state (post-M7)
-
-See `README.md` for the user-visible summary. Internally:
-
-- **Plugin**: Go, two emission modes (`mode=full` inline + `mode=runtime`
- descriptor-delegating wrappers). Sibling `protoc-gen-tarantool-doc`
- emits Markdown reference per `.proto`.
-- **Runtime**: pure Lua + LuaJIT FFI. `wire.lua` + `codec.lua`
- + `lazy.lua` (zero-copy view) + `text.lua` (encode + decode) +
- `json.lua` (strict proto3 JSON) + `wkt.lua` (all 9 well-known types) +
- `grpc.lua` (transport + loopback) + `parser.lua` / `dynamic.lua` /
- `fileset.lua` (three runtime descriptor producers — `.proto` source,
- AST, FileDescriptorSet bytes).
-- **Tests**: 639 luatest tests across 15 test files (39 groups), parametrized over
- both codegen modes where applicable.
-- **Conformance**: proto3 binary+JSON suite **1493 ✓ / 0 failures**;
- proto3 text-format suite **416 ✓ / 0 failures** (Google's
- `conformance_test_runner` v34.1).
-- **Bench**: `bench/baseline.json` tracks allocation/op (regression
- gate at 5%); `just jit-trace` pins LuaJIT trace stability.
-
-The remaining unfinished bullets in section 3 are mostly M8 (release
-engineering) and a handful of optional perf items in M6.
-
-## 3. Phased roadmap
-
-Each milestone ends with a green CI run, an updated README, and a tagged
-release on sourcecraft.dev.
-
-### M1 — Two codegen modes + luatest harness *(done)*
-
-- [x] Promote scalar encode/decode to typed helpers in `pb.wire`.
- `wire.encode_int32(v)`, `wire.decode_string(buf, pos)`, etc., for all
- 15 scalar proto types. Both modes consume the same primitives.
-- [x] **Full (inline) codegen**: emit per-message `_encode` / `_decode`
- functions with no descriptor lookup. Tag bytes precomputed at gen
- time as Lua string literals. This is the JIT-friendly hot path.
-- [x] **Runtime codegen**: keep current behavior. Useful for introspection,
- schema registries, and forward-compat with descriptor-only consumers.
-- [x] Plugin parameter `mode=full|runtime` (default: `full`).
-- [x] Migrate tests to luatest groups. Every behavior tested against **both**
- generated modules to prove parity.
-- [x] Generate side-by-side outputs in `examples/expected/{full,runtime}/`
- for visual diffing.
-
-### M2 — Composite types *(done)*
-
-- [x] **`map<K,V>`**: emit as repeated synthetic `*Entry` messages with
- `key`/`value` fields per spec. Decode merges into a Lua table; encode
- iterates with `pairs`. Key types limited to scalars + string per spec.
-- [x] **`oneof`**: descriptor includes `oneof_index` per field. Encode
- emits at most one branch (last assignment wins). Decode clears prior
- oneof siblings on assignment. Hot-path lookup goes through
- `desc.oneofs_list` (flat array) to keep the trace JIT-stable.
-- [x] **proto3 explicit `optional`**: respect presence — emit field even
- when value equals scalar default. Generated descriptor exposes
- `has_<name>(t)` / `clear_<name>(t)` helpers.
-
-### M3 — Well-known types *(done)*
-
-- [x] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module
- (epoch + nsec mapping). Out-of-spec inputs (negative nanos,
- year > 9999) keep the raw `{seconds, nanos}` table so JSON
- serialization can reject them with `serialize_error` instead of
- crashing on decode.
-- [x] `google.protobuf.Duration` ↔ `{seconds, nanos}` table (interval
- proved a poor fit — it carries months/days that don't map
- cleanly).
-- [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar:
- pass plain Lua value → auto-wrap; decode → auto-unwrap.
-- [x] FieldMask: `repeated string`. Strict round-trip validation —
- snake_case paths must use only `[a-z0-9_]`, no leading/trailing
- `_`, no `__`, and `_` must precede a lowercase letter (not a
- digit). JSON form rejects any `_` (must be lowerCamelCase).
-- [x] `Empty`.
-- [x] `Any`: opaque `{type_url, value}` form by default; `pb.register(desc)`
- + `pb.any.pack(desc, t)` / `pb.any.unpack(any_t)` for typed
- round-trips. JSON canonical mapping emits the flat `{"@type": ...}`
- object when the type is registered, falls back to base64 opaque form
- otherwise.
-- [x] `Struct`, `Value`, `ListValue` ↔ idiomatic Lua tables.
- `box.NULL` is the null_value sentinel (re-exported as `pb.NULL`).
- `pb.wkt.struct(t)` / `pb.wkt.list(t)` tag tables when the auto-detect
- heuristic (`t[1] ~= nil` → list, else struct) needs to be overridden,
- and decode preserves the tag for byte-stable round trips.
-
-**Done when**: WKT-using protos round-trip and integrate visibly with
-`datetime` (e.g. `os.date(...)`-comparable timestamps).
-
-### M4 — gRPC services (transport-agnostic) *(done)*
-
-- [x] Per-service descriptor: methods with full path `/pkg.Svc/Method`,
- input/output type refs, `client_streaming` / `server_streaming` flags.
-- [x] **Client stub**: `MyService_client(transport)` returns a table with
- one function per method. Transport interface, all served by the
- shipped loopback + multiplex implementations:
- ```lua
- transport:unary(path, req_bytes, ctx) -> resp_bytes
- transport:server_stream(path, req_bytes, ctx) -> stream
- transport:client_stream(path, ctx) -> stream
- transport:bidi(path, ctx) -> stream
- ```
- Streaming returns a `stream` object with `send` / `close_send` /
- `recv() -> msg, err` / `cancel`. End-of-stream is `(nil, nil)`;
- handler errors surface as `(nil, err_string)`.
-- [x] **Server side**: `MyService_server(impl)` returns
- `{service, methods, streams}` consumable by the loopback /
- multiplex transports. Handler signatures by RPC kind:
- ```lua
- unary: function(req, ctx) -> resp
- server_stream: function(req, stream, ctx)
- client_stream: function(stream, ctx) -> resp
- bidi: function(stream, ctx)
- ```
-- [ ] Reference network transports (separate projects, deferred). Full
- design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md):
- - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via
- `tarantool/http`. Default external transport; works with browsers
- and `curl` without an HTTP/2 proxy.
- - `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls.
- First-class in-cluster path.
- - `pb.grpc.transport.http_client_unary` — outbound, unary only,
- via `http_client`.
- - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in
- front; the transport contract is HTTP/2-shaped so the same
- generated code works behind it.
- - Conformance anchor: `connectrpc/conformance` (same framed-runner
- shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect
- from one harness).
-
-**Done.** The loopback transport runs each streaming handler on its own
-fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
-(unary + 3 streaming) are exercised by parameterized luatest groups.
-
-### M5 — Conformance + interop *(proto2 + proto3 both closed; CI wire-up still pending)*
-
-- [x] Wire up Google's [protobuf conformance test runner][conformance].
- `cmd/conformance-runner.lua` is the testee: reads length-prefixed
- `ConformanceRequest` on stdin, runs it through our codec / JSON /
- text codecs, writes a length-prefixed `ConformanceResponse` on
- stdout. Loops to EOF. Core dispatch lives in
- `cmd/conformance/core.lua` and is exercised directly by
- `test/conformance_test.lua` so the inner dev loop doesn't need
- Docker.
-- [x] Run the canonical `conformance_test_runner` binary locally and
- track the pass-rate as a numeric metric (regression gate).
- `docker/conformance.Dockerfile` builds `conformance_test_runner`
- from upstream protobuf v34.1 source (matching the host's
- `libprotoc 34.1`) and bundles Tarantool 3 from the official deb;
- `just conformance` regenerates Lua then runs the harness against
- `cmd/conformance-runner.lua` with the repo mounted as a volume.
- Watchlists at `test/conformance/known_failures.txt` (binary +
- JSON suite) and `test/conformance/known_failures_text.txt`
- (text-format suite); **both are empty for the proto3 suites as
- of 2026-05-16.**
-
- Current baseline:
- - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
- - Text-format suite: **416 ✓ / 18 skipped / 0 failures**
-
- Both proto2 and proto3 schemas now run through the harness:
- Binary + JSON 2806 ✓ / 0 failures; Text-format 434 ✓ / 0
- failures (was 1493 + 416 ✓ with proto2 fully skipped). The four
- MessageSet-flavored nested messages remain stripped from our
- vendored copy of `test_messages_proto2.proto` because
- protobuf-go's protoreflect rejects `message_set_wire_format`;
- see [docs/codegen.md#proto2-support](../docs/codegen.md).
-
- Strict-validation closures landed across three commits on the
- `text-conformance-output` branch:
- - `pb.text.decode` — full grammar coverage (recursive-descent
- parser, ~580 LOC).
- - `codec` -0.0 preservation — float/double `is_default_scalar`
- and the inline-codegen elision both gained a sign-bit guard
- (`1/v == math.huge`).
- - `pb.json` strict-validation pass — duplicate-key rejection
- (literal + camel/snake alias detection via a byte-walking
- pre-scan), null-in-container rejection, unknown-enum-name
- rejection (with `ignore_unknown_fields` opt for the
- `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
- `google.protobuf.NullValue` round-trip as JSON `null`,
- strict FieldMask round-trip validation.
-
- The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
- short-circuit JSON output if a future encoder bug starts
- crashing jsoncpp.
-- [ ] CI wire-up. The Docker image build is the long pole (~10–15 min
- on a clean cache); a registry push from a scheduled job would
- let CI runs reuse a warm cache.
-- [x] Cross-impl interop: 18-fixture corpus in `test/interop/fixtures/`
- produced by mainline `protoc --encode`; tests assert byte-for-byte
- equality.
-
-[conformance]: https://github.com/protocolbuffers/protobuf/tree/main/conformance
-
-### M6 — Performance + production polish *(delayed; bench harness shipped, further perf work parked)*
-
-- [x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for
- messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped
- as `bench/bench.lua`, run via `just bench`. Throughput is stderr-only
- (varies with CPU load); JSON document on stdout.
-- [x] Allocation profiling — bytes per encode/decode op, measured via GC
- delta with `collectgarbage('stop')` framing. Committed as
- `bench/baseline.json`. Regression gate: `just bench-compare` exits
- non-zero if alloc/op grows >5% vs baseline. Allocations are
- deterministic to ~10 bytes regardless of hardware.
-- [x] Trace stability — `just jit-trace` (`bench/jit_trace.lua`)
- attaches a `jit.attach('trace')` listener over the hot
- encode/decode paths and asserts no aborts in our source files
- fall into the fatal set (NYI bytecode, blacklisting, persistent
- type instability). Pass: 13/13 scenarios. Two fixes shipped:
- decode_varint grew a 1-byte fast path so callers no longer drag
- an inner loop into the root trace; `pb.finalize_message` now
- precomputes `desc.oneofs_list` so runtime-mode oneof encoding
- uses `ipairs` instead of `pairs` (the latter compiles to bytecode
- ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
- interpreter-bridge counts as a benchmark-quality metric (decoders
- have 0–4 per run depending on JIT timing — caused by side traces
- returning from inlined `decode_varint` calls, which LuaJIT 2.1
- can't stitch back cleanly; small per-call overhead, structural
- to the engine). Scope caveat: map fields still encode via
- `pairs()` and remain off-trace — pinned by the gate's last
- scenario so we notice if upstream lifts the restriction.
-- [ ] Optional output: ibuf-based encoder that writes into a caller-owned
- `ffi.cdata` byte buffer instead of building a string list. Targets
- hot RPC paths where allocation cost dominates.
-- [x] Decoder fast path that returns an `msgpack.object`-like lazy view
- for nested messages; only materializes touched fields.
- Shipped as `runtime/pb/lazy.lua` + `M.<Type>_decode_lazy` codegen
- stubs in both modes. Surface: `:get / :has / :which / :iter /
- :names` on MessageView; `:len / :at / :iter / :tolist` on
- ArrayView; `:get / :has / :keys / :iter / :totable` on MapView.
- Mutation via `:set` is supported and propagates sub-view edits
- transparently (sub-MessageViews tracked on a flat array for
- JIT-stable `is_dirty` — see lazy.lua's `_sub_msg_views`).
- Re-encode is passthrough: untouched views return their original
- bytes verbatim; partially-dirty views walk fields in id order,
- splicing clean segments and re-emitting dirty ones. WKT
- descriptors (those with `desc.decode`) are eager-wrapped so the
- API stays uniform.
-
- Conformance: every interop fixture round-trips byte-equal through
- `decode_lazy(b):encode()`. Trace stability: gated by `make
- jit-trace` — index pass, sparse `:get` x2, and passthrough
- `:encode` all compile with no fatal aborts.
-
- Workload characteristics (from `tarantool bench/lazy_bench.lua`,
- Person at 1KB / 10KB / 100KB, after the SoA index refactor):
- - **Passthrough re-encode** is the headline win: **1.6–1.9×**
- faster than eager decode→encode across all sizes and both modes.
- Untouched views never re-walk the wire.
- - **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
- of eager**: break-even at small sizes, slight win at 100KB
- (especially in runtime mode where eager pays more dispatch
- cost). Earlier 0.60–0.77× regression came from one Lua table
- per wire entry; replacing with parallel int arrays closed the
- allocation gap.
- - **Mutate-then-reencode** is **1.09–1.26× of eager**; the
- per-field splice path now consistently beats full re-encode.
- The honest framing: lazy is a *byte-passthrough* optimization
- that also handles sparse reads at parity. Best fit: proxy /
- router shapes that decode, touch a few fields, and re-encode.
-
-### M7 — Developer ergonomics *(done)*
-
-- [x] Generated EmmyLua / lua-language-server type annotations so
- `t:Person_encode({name=...})` autocompletes in editors.
- Codegen emits `---@class <full.Name>` per message (with one
- `---@field` per field), `---@alias <full.Name> integer` per
- enum, and `---@param` / `---@return` on every `_new`, `_encode`,
- `_decode`, `_decode_lazy`, `_has_*`, `_clear_*` wrapper. Class
- identifiers use proto full names verbatim so cross-file
- references resolve. Lazy view types (`pb.MessageView`,
- `pb.ArrayView`, `pb.MapView`) are declared inline in
- `runtime/pb/lazy.lua` so the LSP sees them. Pure comment
- addition — no runtime impact, 300/300 luatest + 19/19
- jit-trace gate stay green.
-- [x] `pb.from_pb(file_descriptor_set)` — accepts binary `FileDescriptorSet`
- bytes (output of `protoc --descriptor_set_out=...`) and returns
- `{files = {[name] = module}, order, lookup}`. Each per-file module
- has the same surface as `pb.parse` output (statically-generated
- runtime mode). Translation pipeline: hand-built `descriptor.proto`
- descriptors decode the wire bytes via `pb.codec`, then a translator
- converts each `FileDescriptorProto` to the AST shape `pb.parser`
- emits, which `pb.dynamic.build` consumes. Map fields are
- reconstructed from synthetic entry messages (skipped from
- `nested_messages`); `proto3_optional` is rehydrated as
- `optional=true` instead of being modelled as a synthetic oneof.
-- [x] JSON encoding per the [proto3 JSON spec][proto3json]
- (`pb.json.encode` / `pb.json.decode`).
-- [x] Text-format printer. `pb.text.encode(desc, t, opts)` returns the
- `protoc --decode` form (one field per line, 2-space indent, octal
- byte escapes); `opts.single_line=true` collapses to a space-separated
- one-liner for log lines and inline goldens. Codegen emits
- `M.<Type>_text(t, opts)` in both modes. WKT types know their
- idiomatic Lua shapes — `Timestamp`/`Duration` accept datetime cdata
- or `{seconds,nanos}`, wrappers print their unwrapped scalar as
- `value: ...`, `Struct`/`Value`/`ListValue` walk the tagged-table
- form, `FieldMask` prints `paths: ...` per entry, `Any` stays opaque.
-- [x] Text-format parser. `pb.text.decode(desc, text, opts)` is the
- recursive-descent counterpart: handles every grammar bucket the
- proto3 conformance suite exercises — decimal/hex/octal int literals,
- float specials (`inf`/`infinity`/`nan` any case, oversize exponents
- saturating to ±inf, underflow to ±0), C-style + `\u`/`\U` string
- escapes with adjacent-literal concat and surrogate rejection,
- aggregate `{}` / `<>` bodies, repeated short-form `[a, b, c]`,
- `key: K value: V` map entries, the `[type.googleapis.com/...]`
- inline Any form, enum-by-name-or-number, reserved-name drop, and
- numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
- duplicate singular fields, and threads through the conformance
- runner — `cmd/conformance/core.lua` no longer skips `text_payload`.
- Plugin gained a small `reserved_names` emitter so the parser can
- match mainline TextFormat::Parser's "silently drop reserved" rule.
- Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 416
- ✓ / 18 skipped / 0 failures** (the 18 are the proto2 message-type
- bucket; everything in scope passes).
-- [x] JSON strict-validation pass. Six classes of relaxation that the
- proto3 JSON conformance corpus flagged are now enforced — together
- with the -0 codec fix this empties `known_failures.txt`:
- 1. Duplicate JSON keys (`{"foo":1,"foo":2}`) rejected via a
- byte-walking pre-scan that runs before `json.decode`.
- 2. camelCase / snake_case aliases of the same proto field
- rejected via a per-message `field_seen` set.
- 3. JSON `null` inside repeated arrays and map values rejected.
- 4. Unknown enum *names* rejected by default; the
- `ignore_unknown_fields=true` opt silently drops them (and
- the conformance dispatch forwards this flag when
- `req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST`).
- 5. `google.protobuf.NullValue` JSON canonical form: literal
- `null`, not the string `"NULL_VALUE"`. Null on a
- NullValue-typed oneof member marks the oneof active.
- 6. Strict FieldMask round-trip (see M3 entry).
-- [x] **Preserve descriptor options.** Every populated `*Options` message
- surfaces on the generated descriptor as a plain Lua sub-table named
- `options` (or `oneof_options` / `value_options` for oneof / enum value).
- Standard fields use their proto name; extensions use their
- fully-qualified name (bracket-quoted). The walker is generic —
- consumers pull `(google.api.http)`, `(versionpb.etcd_version_*)`,
- `[deprecated = true]`, and any in-house extension without pb knowing
- about them. Option-free protos emit no `options` keys (byte-identical
- output to before). Resolver re-links *Options* messages so in-file
- extensions surface via the protoreflect walker (protogen builds
- `f.Desc` before in-file extensions are registered; we rebuild
- manually). See [docs/codegen.md](docs/codegen.md) for the per-descriptor
- shape and the README's "Descriptor options" section for examples.
-- [x] `protoc-gen-tarantool-doc`: sibling Go plugin under
- `cmd/protoc-gen-tarantool-doc/` that emits one Markdown file per
- input `.proto`. Sections: header (package + imports), messages
- (per-message description + field table with `# | Field | Type |
- Label | Description`), enums (value table), services (method table
- with `unary` / `client` / `server` / `bidi` streaming label).
- Field type cells render scalar names, full type names for
- message/enum references, and `map<K, V>` for maps; synthetic
- map-entry messages are skipped. Leading comments are preserved via
- SourceCodeInfo (squashed to a single line inside table cells).
- Build with `just build-doc`; generate sample docs into
- `examples/docs/` with `just gen-docs`.
-
-[proto3json]: https://protobuf.dev/programming-guides/proto3/#json
-
-### M8 — Release engineering
-
-- [ ] Sourcecraft.dev project + CI pipeline (matrix: Tarantool 2.11 / 3.x
- EE + CE, Linux + macOS).
-- [ ] Tagged releases; rockspec for the Lua runtime; pre-built binaries
- for the Go plugin.
-- [ ] Migration guide from Tarantool's built-in `require('protobuf')` to
- `require('pb')`.
-- [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via
- protobuf-encoded events on a queue.
-
-### M9 — Proto2 baseline *(done; 100% conformance on the patched proto2 suite)*
-
-- [x] Plugin accepts `syntax = "proto2"`. Field descriptors now carry
- `required = true`, `default_value = …`, and `optional = true` for
- every singular field. Existing `IsPacked()`-based packing logic
- already produced the correct proto2 unpacked-by-default.
-- [x] Runtime parser captures `required`, `[default = X]`, and threads
- them through `pb.dynamic`. The dynamic builder now calls
- `codec.compile_writers/compile_readers` so source-parsed schemas
- get the same per-field specializations build-time codegen does.
-- [x] Codec: `required` writer errors on missing-on-encode in both
- inline and runtime modes.
-- [x] JSON + text codecs: presence-tracked elision (`required`,
- `optional`, oneof) bypasses the proto3 implicit-zero rules.
-- [x] **`group` fields** end-to-end. Wire layer carries
- `WIRE_SGROUP=3` / `WIRE_EGROUP=4`; the codec's
- `decode_group(desc, buf, pos, stop_id)` mirrors decode_message
- but stops on EGROUP. Inline + runtime codegen both emit start/end
- tag pairs around the body (no LEN prefix). Text format renders
- groups under the capitalized submessage name. Runtime parser
- desugars `optional|required|repeated group Name = id { … }` into
- a nested message plus a `kind='group'` field whose lowercased
- name is `name` — matches build-time codegen for the dynamic path.
-- [x] **`extend` blocks / `extensions` ranges.** Plugin iterates
- `file.Extensions` + per-message `m.Extensions`, emitting
- `pb.register_extension(extendee_desc, {…})` calls. The codec
- stores set extensions under `data._extensions[full_name]` and
- walks them after the regular field loop on encode; decode routes
- tags not in `field_by_id` through `decode_extension` when
- `extensions_by_id` matches. Text + JSON encoders/decoders speak
- the bracketed `[pkg.ext_name]` syntax. Meta-extensions on
- `google.protobuf.*` descriptors (file/message/field options) are
- skipped at codegen time — the WKT module doesn't surface those
- descriptors at runtime.
-- [x] **Proto2 closed enums.** Plugin emits `closed = true` on every
- proto2 enum descriptor (driven by protoreflect's `IsClosed()`).
- Text format's `parse_enum_value` rejects integer literals that
- don't map to a declared value for closed enums; proto3 stays
- open to preserve forward-compatibility on the wire.
-- [x] 50+ luatest cases (`proto2_basic.*` groups) — full, runtime, and
- dynamic-from-source byte parity, plus group wire-byte pinning
- and JSON/text presence rules.
-- [ ] **`MessageSet`** wire format. `protobuf-go`'s protoreflect
- refuses to load a FileDescriptor declaring
- `option message_set_wire_format = true;`, calling it a removed
- proto1 feature. We strip the four MessageSet-flavored nested
- messages from our vendored
- `test/conformance/proto/test_messages_proto2.proto` so the rest
- of the schema compiles. Real support would require forking
- protoreflect; deferred until a real Tarantool consumer asks.
-- [x] **Conformance numbers.** Full suite, strict mode (protobuf v34.1):
- Binary + JSON 2806 ✓ / 0 failures (was 1493 / 1313 skipped);
- Text-format 434 ✓ / 0 failures (was 416 / 18 skipped).
-
-## 4. Per-feature design notes
-
-### 4.1 Inline (full) codegen — wire bytes
-
-For each message we emit one `_encode` and one `_decode` function. Tag
-bytes are precomputed string literals; field-presence checks are inlined.
-The runtime is reduced to wire-format primitives.
-
-```lua
-function M.Address_encode(t)
- local out, n = {}, 0
- local v
- v = t.street
- if v ~= nil and v ~= '' then
- n = n + 1; out[n] = '\x0a' -- tag(1, LEN)
- n = n + 1; out[n] = wire.encode_string(v)
- end
- -- ...
- return table.concat(out)
-end
-```
-
-Decoder uses an `if-elseif` chain on field id (LuaJIT compiles this
-well for small chain lengths; for >16 fields we may want a numeric jump
-table or tag-byte switching).
-
-### 4.2 Map fields
-
-`map<K,V>` is wire-format-equivalent to:
-
-```proto
-message FooEntry { K key = 1; V value = 2; }
-repeated FooEntry foos = N;
-```
-
-Codegen synthesizes the entry message internally but exposes the field as a
-Lua table (hash, not array). Encode iterates with `pairs`; decode merges
-on duplicate keys (last wins, per spec).
-
-### 4.3 Oneofs
-
-Descriptor gains `oneofs = {[oneof_name] = {field_names...}}`. Encode walks
-fields in declaration order and emits the first non-nil branch. Decode
-clears sibling fields on assignment so callers see exactly one set.
-
-In inline mode, the encode walk becomes an explicit `if`-chain; the decode
-clears are emitted alongside each `elseif id == N then` arm.
-
-### 4.4 64-bit integer ergonomics
-
-Locked: `int64`/`uint64`/`fixed64`/`sfixed64`/`sint64` are LuaJIT
-`int64_t` / `uint64_t` cdata. Reasons:
-- Lossless beyond 2^53.
-- Same convention as Tarantool `msgpackffi`, `net.box`, `box.tuple`.
-- Compares cleanly against `0` (numeric coercion in LuaJIT).
-
-Two coercion helpers — `pb.to_uint64(v)` and `pb.to_int64(v)` (re-exported
-from `pb.wire`) — accept Lua number, cdata, or a numeric string and return
-the matching 64-bit cdata. Use them at the boundary when values come from
-JSON, text format, or net.box arguments where the type isn't already cdata.
-
-### 4.5 Unknown fields *(implemented)*
-
-`t._unknown_fields` is a single Lua string holding the verbatim
-concatenation of tag+value bytes for fields the decoder didn't recognize.
-Captured during decode in encounter order; re-emitted at the tail of
-`_encode`. Mirrors Tarantool's built-in `protobuf` module convention.
-
-Map entries don't preserve unknowns (per spec — synthetic Entry messages).
-WKT types bypass this too, since they have custom `desc.encode/decode`.
-
-Both codegen modes implement it: runtime mode in `pb.codec`, full (inline)
-mode emits per-message capture/re-emit blocks. Tests live in
-`test/unknown_test.lua` and run against both modes.
-
-### 4.6 Protobuf editions *(not implemented; tracking)*
-
-We advertise `CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL` only, not
-`FEATURE_SUPPORTS_EDITIONS`. If protoc is invoked against an
-`edition = "2023";` file with our plugin, it errors out — mainline
-refuses to call an editions-unaware plugin.
-
-**What editions changes.** Editions replaces the proto2-vs-proto3 split
-with one language whose behavior is controlled by per-file / per-field
-`FeatureSet` annotations. The knobs that matter for us:
-
-- `field_presence = EXPLICIT | IMPLICIT | LEGACY_REQUIRED` — presence
- becomes an opt-in/opt-out per field instead of a syntax-wide default.
-- `repeated_field_encoding = PACKED | EXPANDED` — per-field opt-out
- from packed encoding for scalar `repeated` fields.
-- `enum_type = OPEN | CLOSED` — closed enums route unknown values to
- unknown fields (proto2 semantics) instead of round-tripping the int.
-- `utf8_validation = VERIFY | NONE` — per-field strictness on `string`.
-- `json_format = ALLOW | LEGACY_BEST_EFFORT` — JSON behavior overrides.
-
-**What we'd actually gain.** Honestly modest, ranked by user-visible
-value:
-
-1. **Per-field packed opt-out** — the only knob proto3 doesn't expose
- today. Real interop scenarios (talking to legacy proto2 services,
- certain gRPC gateways) sometimes need expanded encoding on a
- specific field. Right now our users have no escape hatch.
-2. **Migration path off proto3.** Proto3 syntax is being phased out in
- favor of edition 2023. This is a forcing function, not a feature
- win — at some point users will write `edition = "2023";` and we'll
- need to read it.
-3. **Explicit presence by default** without the `optional` keyword (and
- the synthetic-oneof wart it creates today).
-4. **Per-field UTF-8 toggle** for users carrying not-quite-UTF-8 bytes
- in `string` fields for legacy compat.
-5. **Closed enums** for users who want proto2-style unknown-value
- routing.
-6. **`LEGACY_REQUIRED` field presence** — niche; only way to express
- "required" without writing proto2.
-
-**What it would cost.** Setting the bit is one line; the work is:
-
-- Plugin (`cmd/protoc-gen-tarantool/internal/gen`) needs to read each
- field's resolved `FeatureSet` and drive codegen off it (packed,
- presence, UTF-8 strictness, enum closedness) instead of the proto3
- defaults baked in.
-- Runtime parser (`runtime/pb/parser.lua`) needs to accept
- `edition = "2023";` and the `features = { ... }` option syntax.
-- `runtime/pb/dynamic.lua` and `runtime/pb/fileset.lua` need to
- propagate resolved features onto the descriptor. The descriptor
- contract in `docs/codegen.md` would gain a `features` field per
- message/field/enum.
-- `CodeGeneratorResponse` needs `minimum_edition` / `maximum_edition`
- set, else protoc rejects an editions-supporting plugin.
-- The conformance suite has editions-specific buckets we'd start
- encountering.
-
-**Current verdict.** Deferred. No user pull today, proto3 will keep
-working for years, and the practical-win list above is mostly small
-quality-of-life items rather than blocking gaps. Revisit when (a)
-someone needs per-field packed control, (b) the conformance suite
-starts gating on editions, or (c) protoc deprecates proto3 syntax
-hard enough that real users hit the error.
-
-### 4.7 gRPC service descriptors
-
-```lua
-M.Greeter_service = {
- name = 'hello.Greeter',
- methods = {
- SayHello = {
- full_name = '/hello.Greeter/SayHello',
- input = M.HelloRequest_descriptor,
- output = M.HelloReply_descriptor,
- client_streaming = false,
- server_streaming = false,
- },
- -- ...
- },
-}
-```
-
-Client: `M.Greeter_client(transport)` returns
-`{SayHello = function(req) ... end, ...}`.
-
-Server: `M.Greeter_server(impl)` returns a table compatible with the
-`server:register(svc)` interface.
-
-## 5. Testing strategy
-
-Tests live as a flat `test/*.lua` set (15 files, 39 luatest groups, 639
-assertions as of 2026-05-16). Each behavior file is parametrized over
-both codegen modes via the pattern:
-
-```lua
-for _, mode in ipairs({'full', 'runtime'}) do
- local g = t.group(name .. '.' .. mode)
- local hello = require(mode .. '.hello.hello_pb')
- g.test_x = function() ... end
-end
-```
-
-Categories — see file names under `test/`:
-
-- **Wire + codec primitives** — exercised indirectly through `protobuf_test`
- (scalars, repeated, nested, optional, oneof) and `interop_test`
- (the byte-for-byte fixture corpus under `test/interop/fixtures/`,
- 10 `.txtpb`/`.bin` pairs from mainline `protoc --encode`).
-- **Composites** — `protobuf_test` for map/oneof/explicit-optional.
-- **WKT** — `struct_value_test`, `any_fieldmask_test`.
-- **Lazy** — `lazy_test` covers MessageView/ArrayView/MapView + the
- byte-splice re-encode path.
-- **Codec dialects** — `json_test`, `text_test`, `text_decode_test`,
- `unknown_test`.
-- **Dynamic descriptors** — `dynamic_test` (.proto source via
- `pb.parse`), `fileset_test` (FileDescriptorSet via `pb.from_pb`),
- with parity assertions against the generated modules.
-- **Codegen surface** — `codegen_doc_comments_test`,
- `doc_test` (the `protoc-gen-tarantool-doc` plugin), `conformance_test`
- (self-test of the runner against crafted requests).
-- **gRPC** — `grpc_streaming_test` against the loopback transport.
-
-Cross-mode parity is pinned by `parity.full_vs_runtime.*` groups
-inside each behavior file.
-
-### Conformance and interop
-
-The Google protobuf conformance suite is driven by
-`cmd/conformance-runner.lua` (stdin/stdout framing) and run via
-`just conformance` (Docker-bundled `conformance_test_runner`). Known
-failures live in `test/conformance/known_failures.txt` (binary+JSON)
-and `test/conformance/known_failures_text.txt` (text-format) — both
-empty for proto3 as of 2026-05-16.
-
-Interop fixtures (`test/interop/fixtures/*.{txtpb,bin}`) are produced
-by mainline `protoc --encode` via `just goldens` and asserted in
-`interop_test.lua` for round-trip equality across both codegen modes.
-Map fixtures use a single key to lock byte-for-byte equality (Lua
-`pairs` ordering won't match protoc's text-proto-order output);
-multi-key map behavior is covered via decode-then-compare-Lua-table
-rather than byte equality.
-
-### Performance + JIT-trace stability
-
-`bench/baseline.json` tracks allocation per op (a deterministic
-metric; throughput swings 30%+ across machines). `just bench-compare`
-fails on >5% alloc regression. `just jit-trace` (`bench/jit_trace.lua`)
-asserts the hot encode/decode paths stay on the JIT trace — a
-companion safety net for the "no `pairs()` on hot paths" rule.
-
-### Not (yet) automated
-
-- Fuzz harnesses (malformed-input + random-valid) — out of band.
-- Go-side `go test`. Codegen correctness is asserted end-to-end via
- Lua tests against committed expected outputs in
- `examples/expected/{full,runtime}/`.
-- CI pipeline. Local-only today; sourcecraft.dev wire-up is M8.
-
-## 6. Tooling roadmap
-
-- **Makefile + Justfile** — both shipped. Makefile covers `build`, `gen`,
- `test`, `goldens`, `bench`, `bench-baseline`, `bench-compare`,
- `jit-trace`, `clean`. Justfile adds `just conformance` (Docker-bundled
- conformance runner).
-- **golangci-lint** for the Go side; **luacheck** for the Lua side.
-- **gofumpt** + **stylua** for formatting.
-- **Coverage**: `go test -cover` for plugin; `luacov` for runtime.
-- **Doc generation**: `pkgsite` for Go; manually maintained Markdown for
- Lua until a tool emerges.
-
-## 7. Open questions / future decisions
-
-| Question | Notes |
-|---|---|
-| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. |
-| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
-| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. |
-| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. |
-| How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. |
-| Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. |
-| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
-| Integration with Tarantool 3.x declarative config? | A `pb.types.<name>` config role would be nice; defer until use case appears. |
-
-## 8. Non-goals (for now)
-
-- **proto2 `extend` / `extensions` / `group` / `MessageSet`** — the
- conformance-only edges. Core proto2 (required, optional, custom
- defaults, packed semantics) is in; see M9.
-- **Protobuf editions** beyond what proto3 enables — see §4.6 for
- the per-feature analysis (what we'd gain, what it would cost, the
- conditions that would force a revisit).
-- **HTTP/2 termination** for gRPC. Push it to Envoy; see
- [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md).
- Connect-JSON over HTTP/1.1 is the default external transport.
-- **Reflection service** (gRPC server reflection) — implement after M5.
-- **gRPC-Web** — covered by the same Connect conformance harness if we
- add it, but not on the v1 path.
-
-## 9. How to update this plan
-
-Edit this file directly. After any milestone closes:
-1. Move the milestone section under a `## Done` heading.
-2. Cross-link the closing PR.
-3. Update `README.md` status table.
-4. Cut a release on sourcecraft.dev.