# tarantool-protobuf — Roadmap & Test Plan

This document tracks the long-form design + testing strategy for
`protoc-gen-tarantool` and the `pb` Lua runtime. Short-term task tracking
lives in the in-session task list; this file is the slow-changing record.

## 1. Vision

A first-class Protocol Buffers + gRPC stack for Tarantool that:

- Generates idiomatic Lua from `.proto` files with **two modes** (full inline
  vs descriptor-driven runtime) so users can pick speed-vs-flexibility.
- Interoperates byte-for-byte with mainline protoc implementations (Go,
  Python, C++) — passes Google's protobuf conformance suite for proto3.
- Ships with a small, FFI-aware runtime that respects Tarantool conventions
  (LuaJIT cdata for 64-bit ints, no global state, no monkey-patching).
- Is comfortable to use from real Tarantool apps: clean integration with
  `net.box`, `fiber`, `box.session`, and the 3.x config framework.

## 2. Current state (post-M7)

See `README.md` for the user-visible summary. Internally:

- **Plugin**: Go, two emission modes (`mode=full` inline + `mode=runtime`
  descriptor-delegating wrappers). Sibling `protoc-gen-tarantool-doc`
  emits Markdown reference per `.proto`.
- **Runtime**: pure Lua + LuaJIT FFI. `wire.lua` + `codec.lua`
  + `lazy.lua` (zero-copy view) + `text.lua` (encode + decode) +
  `json.lua` (strict proto3 JSON) + `wkt.lua` (all 9 well-known types) +
  `grpc.lua` (transport + loopback) + `parser.lua` / `dynamic.lua` /
  `fileset.lua` (three runtime descriptor producers — `.proto` source,
  AST, FileDescriptorSet bytes).
- **Tests**: 639 luatest tests across 15 test files (39 groups), parametrized over
  both codegen modes where applicable.
- **Conformance**: proto3 binary+JSON suite **1493 ✓ / 0 failures**;
  proto3 text-format suite **416 ✓ / 0 failures** (Google's
  `conformance_test_runner` v34.1).
- **Bench**: `bench/baseline.json` tracks allocation/op (regression
  gate at 5%); `just jit-trace` pins LuaJIT trace stability.

The remaining unfinished bullets in section 3 are mostly M8 (release
engineering) and a handful of optional perf items in M6.

## 3. Phased roadmap

Each milestone ends with a green CI run, an updated README, and a tagged
release on sourcecraft.dev.

### M1 — Two codegen modes + luatest harness  *(done)*

- [x] Promote scalar encode/decode to typed helpers in `pb.wire`.
      `wire.encode_int32(v)`, `wire.decode_string(buf, pos)`, etc., for all
      15 scalar proto types. Both modes consume the same primitives.
- [x] **Full (inline) codegen**: emit per-message `_encode` / `_decode`
      functions with no descriptor lookup. Tag bytes precomputed at gen
      time as Lua string literals. This is the JIT-friendly hot path.
- [x] **Runtime codegen**: keep current behavior. Useful for introspection,
      schema registries, and forward-compat with descriptor-only consumers.
- [x] Plugin parameter `mode=full|runtime` (default: `full`).
- [x] Migrate tests to luatest groups. Every behavior tested against **both**
      generated modules to prove parity.
- [x] Generate side-by-side outputs in `examples/expected/{full,runtime}/`
      for visual diffing.

### M2 — Composite types  *(done)*

- [x] **`map<K,V>`**: emit as repeated synthetic `*Entry` messages with
      `key`/`value` fields per spec. Decode merges into a Lua table; encode
      iterates with `pairs`. Key types limited to scalars + string per spec.
- [x] **`oneof`**: descriptor includes `oneof_index` per field. Encode
      emits at most one branch (last assignment wins). Decode clears prior
      oneof siblings on assignment. Hot-path lookup goes through
      `desc.oneofs_list` (flat array) to keep the trace JIT-stable.
- [x] **proto3 explicit `optional`**: respect presence — emit field even
      when value equals scalar default. Generated descriptor exposes
      `has_<name>(t)` / `clear_<name>(t)` helpers.

### M3 — Well-known types  *(done)*

- [x] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module
      (epoch + nsec mapping). Out-of-spec inputs (negative nanos,
      year > 9999) keep the raw `{seconds, nanos}` table so JSON
      serialization can reject them with `serialize_error` instead of
      crashing on decode.
- [x] `google.protobuf.Duration` ↔ `{seconds, nanos}` table (interval
      proved a poor fit — it carries months/days that don't map
      cleanly).
- [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar:
      pass plain Lua value → auto-wrap; decode → auto-unwrap.
- [x] FieldMask: `repeated string`. Strict round-trip validation —
      snake_case paths must use only `[a-z0-9_]`, no leading/trailing
      `_`, no `__`, and `_` must precede a lowercase letter (not a
      digit). JSON form rejects any `_` (must be lowerCamelCase).
- [x] `Empty`.
- [x] `Any`: opaque `{type_url, value}` form by default; `pb.register(desc)`
      + `pb.any.pack(desc, t)` / `pb.any.unpack(any_t)` for typed
      round-trips. JSON canonical mapping emits the flat `{"@type": ...}`
      object when the type is registered, falls back to base64 opaque form
      otherwise.
- [x] `Struct`, `Value`, `ListValue` ↔ idiomatic Lua tables.
      `box.NULL` is the null_value sentinel (re-exported as `pb.NULL`).
      `pb.wkt.struct(t)` / `pb.wkt.list(t)` tag tables when the auto-detect
      heuristic (`t[1] ~= nil` → list, else struct) needs to be overridden,
      and decode preserves the tag for byte-stable round trips.

**Done when**: WKT-using protos round-trip and integrate visibly with
`datetime` (e.g. `os.date(...)`-comparable timestamps).

### M4 — gRPC services (transport-agnostic)  *(done)*

- [x] Per-service descriptor: methods with full path `/pkg.Svc/Method`,
      input/output type refs, `client_streaming` / `server_streaming` flags.
- [x] **Client stub**: `MyService_client(transport)` returns a table with
      one function per method. Transport interface, all served by the
      shipped loopback + multiplex implementations:
      ```lua
      transport:unary(path, req_bytes, ctx)         -> resp_bytes
      transport:server_stream(path, req_bytes, ctx) -> stream
      transport:client_stream(path, ctx)            -> stream
      transport:bidi(path, ctx)                     -> stream
      ```
      Streaming returns a `stream` object with `send` / `close_send` /
      `recv() -> msg, err` / `cancel`. End-of-stream is `(nil, nil)`;
      handler errors surface as `(nil, err_string)`.
- [x] **Server side**: `MyService_server(impl)` returns
      `{service, methods, streams}` consumable by the loopback /
      multiplex transports. Handler signatures by RPC kind:
      ```lua
      unary:          function(req, ctx) -> resp
      server_stream:  function(req, stream, ctx)
      client_stream:  function(stream, ctx) -> resp
      bidi:           function(stream, ctx)
      ```
- [ ] Reference network transports (separate projects, deferred). Full
      design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md):
  - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via
    `tarantool/http`. Default external transport; works with browsers
    and `curl` without an HTTP/2 proxy.
  - `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls.
    First-class in-cluster path.
  - `pb.grpc.transport.http_client_unary` — outbound, unary only,
    via `http_client`.
  - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in
    front; the transport contract is HTTP/2-shaped so the same
    generated code works behind it.
  - Conformance anchor: `connectrpc/conformance` (same framed-runner
    shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect
    from one harness).

**Done.** The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.

### M5 — Conformance + interop  *(proto3 closed; CI wire-up + proto2 deferred)*

- [x] Wire up Google's [protobuf conformance test runner][conformance].
      `cmd/conformance-runner.lua` is the testee: reads length-prefixed
      `ConformanceRequest` on stdin, runs it through our codec / JSON /
      text codecs, writes a length-prefixed `ConformanceResponse` on
      stdout. Loops to EOF. Core dispatch lives in
      `cmd/conformance/core.lua` and is exercised directly by
      `test/conformance_test.lua` so the inner dev loop doesn't need
      Docker.
- [x] Run the canonical `conformance_test_runner` binary locally and
      track the pass-rate as a numeric metric (regression gate).
      `docker/conformance.Dockerfile` builds `conformance_test_runner`
      from upstream protobuf v34.1 source (matching the host's
      `libprotoc 34.1`) and bundles Tarantool 3 from the official deb;
      `just conformance` regenerates Lua then runs the harness against
      `cmd/conformance-runner.lua` with the repo mounted as a volume.
      Watchlists at `test/conformance/known_failures.txt` (binary +
      JSON suite) and `test/conformance/known_failures_text.txt`
      (text-format suite); **both are empty for the proto3 suites as
      of 2026-05-16.**

      Current baseline:
        - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
        - Text-format suite:  **416 ✓ /   18 skipped / 0 failures**

      The 1313 + 18 skipped all target
      `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
      is a separate slice — see [docs/codegen.md](../docs/codegen.md)
      for what it would take.

      Strict-validation closures landed across three commits on the
      `text-conformance-output` branch:
        - `pb.text.decode` — full grammar coverage (recursive-descent
          parser, ~580 LOC).
        - `codec` -0.0 preservation — float/double `is_default_scalar`
          and the inline-codegen elision both gained a sign-bit guard
          (`1/v == math.huge`).
        - `pb.json` strict-validation pass — duplicate-key rejection
          (literal + camel/snake alias detection via a byte-walking
          pre-scan), null-in-container rejection, unknown-enum-name
          rejection (with `ignore_unknown_fields` opt for the
          `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
          `google.protobuf.NullValue` round-trip as JSON `null`,
          strict FieldMask round-trip validation.

      The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
      short-circuit JSON output if a future encoder bug starts
      crashing jsoncpp.
- [ ] CI wire-up. The Docker image build is the long pole (~10–15 min
      on a clean cache); a registry push from a scheduled job would
      let CI runs reuse a warm cache.
- [x] Cross-impl interop: 18-fixture corpus in `test/interop/fixtures/`
      produced by mainline `protoc --encode`; tests assert byte-for-byte
      equality.

[conformance]: https://github.com/protocolbuffers/protobuf/tree/main/conformance

### M6 — Performance + production polish  *(delayed; bench harness shipped, further perf work parked)*

- [x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for
      messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped
      as `bench/bench.lua`, run via `just bench`. Throughput is stderr-only
      (varies with CPU load); JSON document on stdout.
- [x] Allocation profiling — bytes per encode/decode op, measured via GC
      delta with `collectgarbage('stop')` framing. Committed as
      `bench/baseline.json`. Regression gate: `just bench-compare` exits
      non-zero if alloc/op grows >5% vs baseline. Allocations are
      deterministic to ~10 bytes regardless of hardware.
- [x] Trace stability — `just jit-trace` (`bench/jit_trace.lua`)
      attaches a `jit.attach('trace')` listener over the hot
      encode/decode paths and asserts no aborts in our source files
      fall into the fatal set (NYI bytecode, blacklisting, persistent
      type instability). Pass: 13/13 scenarios. Two fixes shipped:
      decode_varint grew a 1-byte fast path so callers no longer drag
      an inner loop into the root trace; `pb.finalize_message` now
      precomputes `desc.oneofs_list` so runtime-mode oneof encoding
      uses `ipairs` instead of `pairs` (the latter compiles to bytecode
      ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
      interpreter-bridge counts as a benchmark-quality metric (decoders
      have 0–4 per run depending on JIT timing — caused by side traces
      returning from inlined `decode_varint` calls, which LuaJIT 2.1
      can't stitch back cleanly; small per-call overhead, structural
      to the engine). Scope caveat: map fields still encode via
      `pairs()` and remain off-trace — pinned by the gate's last
      scenario so we notice if upstream lifts the restriction.
- [ ] Optional output: ibuf-based encoder that writes into a caller-owned
      `ffi.cdata` byte buffer instead of building a string list. Targets
      hot RPC paths where allocation cost dominates.
- [x] Decoder fast path that returns an `msgpack.object`-like lazy view
      for nested messages; only materializes touched fields.
      Shipped as `runtime/pb/lazy.lua` + `M.<Type>_decode_lazy` codegen
      stubs in both modes. Surface: `:get / :has / :which / :iter /
      :names` on MessageView; `:len / :at / :iter / :tolist` on
      ArrayView; `:get / :has / :keys / :iter / :totable` on MapView.
      Mutation via `:set` is supported and propagates sub-view edits
      transparently (sub-MessageViews tracked on a flat array for
      JIT-stable `is_dirty` — see lazy.lua's `_sub_msg_views`).
      Re-encode is passthrough: untouched views return their original
      bytes verbatim; partially-dirty views walk fields in id order,
      splicing clean segments and re-emitting dirty ones. WKT
      descriptors (those with `desc.decode`) are eager-wrapped so the
      API stays uniform.

      Conformance: every interop fixture round-trips byte-equal through
      `decode_lazy(b):encode()`. Trace stability: gated by `make
      jit-trace` — index pass, sparse `:get` x2, and passthrough
      `:encode` all compile with no fatal aborts.

      Workload characteristics (from `tarantool bench/lazy_bench.lua`,
      Person at 1KB / 10KB / 100KB, after the SoA index refactor):
       - **Passthrough re-encode** is the headline win: **1.6–1.9×**
         faster than eager decode→encode across all sizes and both modes.
         Untouched views never re-walk the wire.
       - **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
         of eager**: break-even at small sizes, slight win at 100KB
         (especially in runtime mode where eager pays more dispatch
         cost). Earlier 0.60–0.77× regression came from one Lua table
         per wire entry; replacing with parallel int arrays closed the
         allocation gap.
       - **Mutate-then-reencode** is **1.09–1.26× of eager**; the
         per-field splice path now consistently beats full re-encode.
      The honest framing: lazy is a *byte-passthrough* optimization
      that also handles sparse reads at parity. Best fit: proxy /
      router shapes that decode, touch a few fields, and re-encode.

### M7 — Developer ergonomics  *(done)*

- [x] Generated EmmyLua / lua-language-server type annotations so
      `t:Person_encode({name=...})` autocompletes in editors.
      Codegen emits `---@class <full.Name>` per message (with one
      `---@field` per field), `---@alias <full.Name> integer` per
      enum, and `---@param` / `---@return` on every `_new`, `_encode`,
      `_decode`, `_decode_lazy`, `_has_*`, `_clear_*` wrapper. Class
      identifiers use proto full names verbatim so cross-file
      references resolve. Lazy view types (`pb.MessageView`,
      `pb.ArrayView`, `pb.MapView`) are declared inline in
      `runtime/pb/lazy.lua` so the LSP sees them. Pure comment
      addition — no runtime impact, 300/300 luatest + 19/19
      jit-trace gate stay green.
- [x] `pb.from_pb(file_descriptor_set)` — accepts binary `FileDescriptorSet`
      bytes (output of `protoc --descriptor_set_out=...`) and returns
      `{files = {[name] = module}, order, lookup}`. Each per-file module
      has the same surface as `pb.parse` output (statically-generated
      runtime mode). Translation pipeline: hand-built `descriptor.proto`
      descriptors decode the wire bytes via `pb.codec`, then a translator
      converts each `FileDescriptorProto` to the AST shape `pb.parser`
      emits, which `pb.dynamic.build` consumes. Map fields are
      reconstructed from synthetic entry messages (skipped from
      `nested_messages`); `proto3_optional` is rehydrated as
      `optional=true` instead of being modelled as a synthetic oneof.
- [x] JSON encoding per the [proto3 JSON spec][proto3json]
      (`pb.json.encode` / `pb.json.decode`).
- [x] Text-format printer. `pb.text.encode(desc, t, opts)` returns the
      `protoc --decode` form (one field per line, 2-space indent, octal
      byte escapes); `opts.single_line=true` collapses to a space-separated
      one-liner for log lines and inline goldens. Codegen emits
      `M.<Type>_text(t, opts)` in both modes. WKT types know their
      idiomatic Lua shapes — `Timestamp`/`Duration` accept datetime cdata
      or `{seconds,nanos}`, wrappers print their unwrapped scalar as
      `value: ...`, `Struct`/`Value`/`ListValue` walk the tagged-table
      form, `FieldMask` prints `paths: ...` per entry, `Any` stays opaque.
- [x] Text-format parser. `pb.text.decode(desc, text, opts)` is the
      recursive-descent counterpart: handles every grammar bucket the
      proto3 conformance suite exercises — decimal/hex/octal int literals,
      float specials (`inf`/`infinity`/`nan` any case, oversize exponents
      saturating to ±inf, underflow to ±0), C-style + `\u`/`\U` string
      escapes with adjacent-literal concat and surrogate rejection,
      aggregate `{}` / `<>` bodies, repeated short-form `[a, b, c]`,
      `key: K value: V` map entries, the `[type.googleapis.com/...]`
      inline Any form, enum-by-name-or-number, reserved-name drop, and
      numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
      duplicate singular fields, and threads through the conformance
      runner — `cmd/conformance/core.lua` no longer skips `text_payload`.
      Plugin gained a small `reserved_names` emitter so the parser can
      match mainline TextFormat::Parser's "silently drop reserved" rule.
      Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 416
      ✓ / 18 skipped / 0 failures** (the 18 are the proto2 message-type
      bucket; everything in scope passes).
- [x] JSON strict-validation pass. Six classes of relaxation that the
      proto3 JSON conformance corpus flagged are now enforced — together
      with the -0 codec fix this empties `known_failures.txt`:
        1. Duplicate JSON keys (`{"foo":1,"foo":2}`) rejected via a
           byte-walking pre-scan that runs before `json.decode`.
        2. camelCase / snake_case aliases of the same proto field
           rejected via a per-message `field_seen` set.
        3. JSON `null` inside repeated arrays and map values rejected.
        4. Unknown enum *names* rejected by default; the
           `ignore_unknown_fields=true` opt silently drops them (and
           the conformance dispatch forwards this flag when
           `req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST`).
        5. `google.protobuf.NullValue` JSON canonical form: literal
           `null`, not the string `"NULL_VALUE"`. Null on a
           NullValue-typed oneof member marks the oneof active.
        6. Strict FieldMask round-trip (see M3 entry).
- [x] `protoc-gen-tarantool-doc`: sibling Go plugin under
      `cmd/protoc-gen-tarantool-doc/` that emits one Markdown file per
      input `.proto`. Sections: header (package + imports), messages
      (per-message description + field table with `# | Field | Type |
      Label | Description`), enums (value table), services (method table
      with `unary` / `client` / `server` / `bidi` streaming label).
      Field type cells render scalar names, full type names for
      message/enum references, and `map<K, V>` for maps; synthetic
      map-entry messages are skipped. Leading comments are preserved via
      SourceCodeInfo (squashed to a single line inside table cells).
      Build with `just build-doc`; generate sample docs into
      `examples/docs/` with `just gen-docs`.

[proto3json]: https://protobuf.dev/programming-guides/proto3/#json

### M8 — Release engineering

- [ ] Sourcecraft.dev project + CI pipeline (matrix: Tarantool 2.11 / 3.x
      EE + CE, Linux + macOS).
- [ ] Tagged releases; rockspec for the Lua runtime; pre-built binaries
      for the Go plugin.
- [ ] Migration guide from Tarantool's built-in `require('protobuf')` to
      `require('pb')`.
- [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via
      protobuf-encoded events on a queue.

## 4. Per-feature design notes

### 4.1 Inline (full) codegen — wire bytes

For each message we emit one `_encode` and one `_decode` function. Tag
bytes are precomputed string literals; field-presence checks are inlined.
The runtime is reduced to wire-format primitives.

```lua
function M.Address_encode(t)
    local out, n = {}, 0
    local v
    v = t.street
    if v ~= nil and v ~= '' then
        n = n + 1; out[n] = '\x0a'  -- tag(1, LEN)
        n = n + 1; out[n] = wire.encode_string(v)
    end
    -- ...
    return table.concat(out)
end
```

Decoder uses an `if-elseif` chain on field id (LuaJIT compiles this
well for small chain lengths; for >16 fields we may want a numeric jump
table or tag-byte switching).

### 4.2 Map fields

`map<K,V>` is wire-format-equivalent to:

```proto
message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;
```

Codegen synthesizes the entry message internally but exposes the field as a
Lua table (hash, not array). Encode iterates with `pairs`; decode merges
on duplicate keys (last wins, per spec).

### 4.3 Oneofs

Descriptor gains `oneofs = {[oneof_name] = {field_names...}}`. Encode walks
fields in declaration order and emits the first non-nil branch. Decode
clears sibling fields on assignment so callers see exactly one set.

In inline mode, the encode walk becomes an explicit `if`-chain; the decode
clears are emitted alongside each `elseif id == N then` arm.

### 4.4 64-bit integer ergonomics

Locked: `int64`/`uint64`/`fixed64`/`sfixed64`/`sint64` are LuaJIT
`int64_t` / `uint64_t` cdata. Reasons:
- Lossless beyond 2^53.
- Same convention as Tarantool `msgpackffi`, `net.box`, `box.tuple`.
- Compares cleanly against `0` (numeric coercion in LuaJIT).

Two coercion helpers — `pb.to_uint64(v)` and `pb.to_int64(v)` (re-exported
from `pb.wire`) — accept Lua number, cdata, or a numeric string and return
the matching 64-bit cdata. Use them at the boundary when values come from
JSON, text format, or net.box arguments where the type isn't already cdata.

### 4.5 Unknown fields  *(implemented)*

`t._unknown_fields` is a single Lua string holding the verbatim
concatenation of tag+value bytes for fields the decoder didn't recognize.
Captured during decode in encounter order; re-emitted at the tail of
`_encode`. Mirrors Tarantool's built-in `protobuf` module convention.

Map entries don't preserve unknowns (per spec — synthetic Entry messages).
WKT types bypass this too, since they have custom `desc.encode/decode`.

Both codegen modes implement it: runtime mode in `pb.codec`, full (inline)
mode emits per-message capture/re-emit blocks. Tests live in
`test/unknown_test.lua` and run against both modes.

### 4.6 Protobuf editions  *(not implemented; tracking)*

We advertise `CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL` only, not
`FEATURE_SUPPORTS_EDITIONS`. If protoc is invoked against an
`edition = "2023";` file with our plugin, it errors out — mainline
refuses to call an editions-unaware plugin.

**What editions changes.** Editions replaces the proto2-vs-proto3 split
with one language whose behavior is controlled by per-file / per-field
`FeatureSet` annotations. The knobs that matter for us:

- `field_presence = EXPLICIT | IMPLICIT | LEGACY_REQUIRED` — presence
  becomes an opt-in/opt-out per field instead of a syntax-wide default.
- `repeated_field_encoding = PACKED | EXPANDED` — per-field opt-out
  from packed encoding for scalar `repeated` fields.
- `enum_type = OPEN | CLOSED` — closed enums route unknown values to
  unknown fields (proto2 semantics) instead of round-tripping the int.
- `utf8_validation = VERIFY | NONE` — per-field strictness on `string`.
- `json_format = ALLOW | LEGACY_BEST_EFFORT` — JSON behavior overrides.

**What we'd actually gain.** Honestly modest, ranked by user-visible
value:

1. **Per-field packed opt-out** — the only knob proto3 doesn't expose
   today. Real interop scenarios (talking to legacy proto2 services,
   certain gRPC gateways) sometimes need expanded encoding on a
   specific field. Right now our users have no escape hatch.
2. **Migration path off proto3.** Proto3 syntax is being phased out in
   favor of edition 2023. This is a forcing function, not a feature
   win — at some point users will write `edition = "2023";` and we'll
   need to read it.
3. **Explicit presence by default** without the `optional` keyword (and
   the synthetic-oneof wart it creates today).
4. **Per-field UTF-8 toggle** for users carrying not-quite-UTF-8 bytes
   in `string` fields for legacy compat.
5. **Closed enums** for users who want proto2-style unknown-value
   routing.
6. **`LEGACY_REQUIRED` field presence** — niche; only way to express
   "required" without writing proto2.

**What it would cost.** Setting the bit is one line; the work is:

- Plugin (`cmd/protoc-gen-tarantool/internal/gen`) needs to read each
  field's resolved `FeatureSet` and drive codegen off it (packed,
  presence, UTF-8 strictness, enum closedness) instead of the proto3
  defaults baked in.
- Runtime parser (`runtime/pb/parser.lua`) needs to accept
  `edition = "2023";` and the `features = { ... }` option syntax.
- `runtime/pb/dynamic.lua` and `runtime/pb/fileset.lua` need to
  propagate resolved features onto the descriptor. The descriptor
  contract in `docs/codegen.md` would gain a `features` field per
  message/field/enum.
- `CodeGeneratorResponse` needs `minimum_edition` / `maximum_edition`
  set, else protoc rejects an editions-supporting plugin.
- The conformance suite has editions-specific buckets we'd start
  encountering.

**Current verdict.** Deferred. No user pull today, proto3 will keep
working for years, and the practical-win list above is mostly small
quality-of-life items rather than blocking gaps. Revisit when (a)
someone needs per-field packed control, (b) the conformance suite
starts gating on editions, or (c) protoc deprecates proto3 syntax
hard enough that real users hit the error.

### 4.7 gRPC service descriptors

```lua
M.Greeter_service = {
    name = 'hello.Greeter',
    methods = {
        SayHello = {
            full_name = '/hello.Greeter/SayHello',
            input = M.HelloRequest_descriptor,
            output = M.HelloReply_descriptor,
            client_streaming = false,
            server_streaming = false,
        },
        -- ...
    },
}
```

Client: `M.Greeter_client(transport)` returns
`{SayHello = function(req) ... end, ...}`.

Server: `M.Greeter_server(impl)` returns a table compatible with the
`server:register(svc)` interface.

## 5. Testing strategy

Tests live as a flat `test/*.lua` set (15 files, 39 luatest groups, 639
assertions as of 2026-05-16). Each behavior file is parametrized over
both codegen modes via the pattern:

```lua
for _, mode in ipairs({'full', 'runtime'}) do
    local g = t.group(name .. '.' .. mode)
    local hello = require(mode .. '.hello.hello_pb')
    g.test_x = function() ... end
end
```

Categories — see file names under `test/`:

- **Wire + codec primitives** — exercised indirectly through `protobuf_test`
  (scalars, repeated, nested, optional, oneof) and `interop_test`
  (the byte-for-byte fixture corpus under `test/interop/fixtures/`,
  10 `.txtpb`/`.bin` pairs from mainline `protoc --encode`).
- **Composites** — `protobuf_test` for map/oneof/explicit-optional.
- **WKT** — `struct_value_test`, `any_fieldmask_test`.
- **Lazy** — `lazy_test` covers MessageView/ArrayView/MapView + the
  byte-splice re-encode path.
- **Codec dialects** — `json_test`, `text_test`, `text_decode_test`,
  `unknown_test`.
- **Dynamic descriptors** — `dynamic_test` (.proto source via
  `pb.parse`), `fileset_test` (FileDescriptorSet via `pb.from_pb`),
  with parity assertions against the generated modules.
- **Codegen surface** — `codegen_doc_comments_test`,
  `doc_test` (the `protoc-gen-tarantool-doc` plugin), `conformance_test`
  (self-test of the runner against crafted requests).
- **gRPC** — `grpc_streaming_test` against the loopback transport.

Cross-mode parity is pinned by `parity.full_vs_runtime.*` groups
inside each behavior file.

### Conformance and interop

The Google protobuf conformance suite is driven by
`cmd/conformance-runner.lua` (stdin/stdout framing) and run via
`just conformance` (Docker-bundled `conformance_test_runner`). Known
failures live in `test/conformance/known_failures.txt` (binary+JSON)
and `test/conformance/known_failures_text.txt` (text-format) — both
empty for proto3 as of 2026-05-16.

Interop fixtures (`test/interop/fixtures/*.{txtpb,bin}`) are produced
by mainline `protoc --encode` via `just goldens` and asserted in
`interop_test.lua` for round-trip equality across both codegen modes.
Map fixtures use a single key to lock byte-for-byte equality (Lua
`pairs` ordering won't match protoc's text-proto-order output);
multi-key map behavior is covered via decode-then-compare-Lua-table
rather than byte equality.

### Performance + JIT-trace stability

`bench/baseline.json` tracks allocation per op (a deterministic
metric; throughput swings 30%+ across machines). `just bench-compare`
fails on >5% alloc regression. `just jit-trace` (`bench/jit_trace.lua`)
asserts the hot encode/decode paths stay on the JIT trace — a
companion safety net for the "no `pairs()` on hot paths" rule.

### Not (yet) automated

- Fuzz harnesses (malformed-input + random-valid) — out of band.
- Go-side `go test`. Codegen correctness is asserted end-to-end via
  Lua tests against committed expected outputs in
  `examples/expected/{full,runtime}/`.
- CI pipeline. Local-only today; sourcecraft.dev wire-up is M8.

## 6. Tooling roadmap

- **Makefile + Justfile** — both shipped. Makefile covers `build`, `gen`,
  `test`, `goldens`, `bench`, `bench-baseline`, `bench-compare`,
  `jit-trace`, `clean`. Justfile adds `just conformance` (Docker-bundled
  conformance runner).
- **golangci-lint** for the Go side; **luacheck** for the Lua side.
- **gofumpt** + **stylua** for formatting.
- **Coverage**: `go test -cover` for plugin; `luacov` for runtime.
- **Doc generation**: `pkgsite` for Go; manually maintained Markdown for
  Lua until a tool emerges.

## 7. Open questions / future decisions

| Question | Notes |
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. |
| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. |
| How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. |
| Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
| Integration with Tarantool 3.x declarative config? | A `pb.types.<name>` config role would be nice; defer until use case appears. |

## 8. Non-goals (for now)

- **proto2** syntax — spec is more complex (required, default values,
  groups), real demand is rare on Tarantool.
- **Protobuf editions** beyond what proto3 enables — see §4.6 for
  the per-feature analysis (what we'd gain, what it would cost, the
  conditions that would force a revisit).
- **HTTP/2 termination** for gRPC. Push it to Envoy; see
  [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md).
  Connect-JSON over HTTP/1.1 is the default external transport.
- **Reflection service** (gRPC server reflection) — implement after M5.
- **gRPC-Web** — covered by the same Connect conformance harness if we
  add it, but not on the v1 path.

## 9. How to update this plan

Edit this file directly. After any milestone closes:
1. Move the milestone section under a `## Done` heading.
2. Cross-link the closing PR.
3. Update `README.md` status table.
4. Cut a release on sourcecraft.dev.