# tarantool-protobuf — Roadmap & Test Plan

This document tracks the long-form design + testing strategy for
`protoc-gen-tarantool` and the `pb` Lua runtime. Short-term task tracking
lives in the in-session task list; this file is the slow-changing record.

## 1. Vision

A first-class Protocol Buffers + gRPC stack for Tarantool that:

- Generates idiomatic Lua from `.proto` files with **two modes** (full inline
  vs descriptor-driven runtime) so users can pick speed-vs-flexibility.
- Interoperates byte-for-byte with mainline protoc implementations (Go,
  Python, C++) — passes Google's protobuf conformance suite for proto3.
- Ships with a small, FFI-aware runtime that respects Tarantool conventions
  (LuaJIT cdata for 64-bit ints, no global state, no monkey-patching).
- Is comfortable to use from real Tarantool apps: clean integration with
  `net.box`, `fiber`, `box.session`, and the 3.x config framework.

## 2. Current state (post-M7)

See `README.md` for the user-visible summary. Internally:

- **Plugin**: Go, two emission modes (`mode=full` inline + `mode=runtime`
  descriptor-delegating wrappers). Sibling `protoc-gen-tarantool-doc`
  emits Markdown reference per `.proto`.
- **Runtime**: pure Lua + LuaJIT FFI. `wire.lua` (~660 LOC) + `codec.lua`
  + `lazy.lua` (zero-copy view) + `text.lua` (encode + decode) +
  `json.lua` (strict proto3 JSON) + `wkt.lua` (all 9 well-known types) +
  `grpc.lua` (transport + loopback) + `parser.lua` / `dynamic.lua` /
  `fileset.lua` (three runtime descriptor producers — `.proto` source,
  AST, FileDescriptorSet bytes).
- **Tests**: 613 luatest assertions across 19 groups, parametrized over
  both codegen modes where applicable.
- **Conformance**: proto3 binary+JSON suite **1493 ✓ / 0 failures**;
  proto3 text-format suite **416 ✓ / 0 failures** (Google's
  `conformance_test_runner` v34.1).
- **Bench**: `bench/baseline.json` tracks allocation/op (regression
  gate at 5%); `make jit-trace` pins LuaJIT trace stability.

The remaining unfinished bullets in section 3 are mostly M8 (release
engineering) and a handful of optional perf items in M6.

## 3. Phased roadmap

Each milestone ends with a green CI run, an updated README, and a tagged
release on sourcecraft.dev.

### M1 — Two codegen modes + luatest harness  *(done)*

- [x] Promote scalar encode/decode to typed helpers in `pb.wire`.
      `wire.encode_int32(v)`, `wire.decode_string(buf, pos)`, etc., for all
      15 scalar proto types. Both modes consume the same primitives.
- [x] **Full (inline) codegen**: emit per-message `_encode` / `_decode`
      functions with no descriptor lookup. Tag bytes precomputed at gen
      time as Lua string literals. This is the JIT-friendly hot path.
- [x] **Runtime codegen**: keep current behavior. Useful for introspection,
      schema registries, and forward-compat with descriptor-only consumers.
- [x] Plugin parameter `mode=full|runtime` (default: `full`).
- [x] Migrate tests to luatest groups. Every behavior tested against **both**
      generated modules to prove parity.
- [x] Generate side-by-side outputs in `examples/expected/{full,runtime}/`
      for visual diffing.

### M2 — Composite types  *(done)*

- [x] **`map<K,V>`**: emit as repeated synthetic `*Entry` messages with
      `key`/`value` fields per spec. Decode merges into a Lua table; encode
      iterates with `pairs`. Key types limited to scalars + string per spec.
- [x] **`oneof`**: descriptor includes `oneof_index` per field. Encode
      emits at most one branch (last assignment wins). Decode clears prior
      oneof siblings on assignment. Hot-path lookup goes through
      `desc.oneofs_list` (flat array) to keep the trace JIT-stable.
- [x] **proto3 explicit `optional`**: respect presence — emit field even
      when value equals scalar default. Generated descriptor exposes
      `has_<name>(t)` / `clear_<name>(t)` helpers.

### M3 — Well-known types  *(done)*

- [x] `google.protobuf.Timestamp` ↔ Tarantool `datetime` module
      (epoch + nsec mapping). Out-of-spec inputs (negative nanos,
      year > 9999) keep the raw `{seconds, nanos}` table so JSON
      serialization can reject them with `serialize_error` instead of
      crashing on decode.
- [x] `google.protobuf.Duration` ↔ `{seconds, nanos}` table (interval
      proved a poor fit — it carries months/days that don't map
      cleanly).
- [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar:
      pass plain Lua value → auto-wrap; decode → auto-unwrap.
- [x] FieldMask: `repeated string`. Strict round-trip validation —
      snake_case paths must use only `[a-z0-9_]`, no leading/trailing
      `_`, no `__`, and `_` must precede a lowercase letter (not a
      digit). JSON form rejects any `_` (must be lowerCamelCase).
- [x] `Empty`.
- [x] `Any`: opaque `{type_url, value}` form by default; `pb.register(desc)`
      + `pb.any.pack(desc, t)` / `pb.any.unpack(any_t)` for typed
      round-trips. JSON canonical mapping emits the flat `{"@type": ...}`
      object when the type is registered, falls back to base64 opaque form
      otherwise.
- [x] `Struct`, `Value`, `ListValue` ↔ idiomatic Lua tables.
      `box.NULL` is the null_value sentinel (re-exported as `pb.NULL`).
      `pb.wkt.struct(t)` / `pb.wkt.list(t)` tag tables when the auto-detect
      heuristic (`t[1] ~= nil` → list, else struct) needs to be overridden,
      and decode preserves the tag for byte-stable round trips.

**Done when**: WKT-using protos round-trip and integrate visibly with
`datetime` (e.g. `os.date(...)`-comparable timestamps).

### M4 — gRPC services (transport-agnostic)  *(done)*

- [x] Per-service descriptor: methods with full path `/pkg.Svc/Method`,
      input/output type refs, `client_streaming` / `server_streaming` flags.
- [x] **Client stub**: `MyService_client(transport)` returns a table with
      one function per method. Transport interface, all served by the
      shipped loopback + multiplex implementations:
      ```lua
      transport:unary(path, req_bytes, ctx)         -> resp_bytes
      transport:server_stream(path, req_bytes, ctx) -> stream
      transport:client_stream(path, ctx)            -> stream
      transport:bidi(path, ctx)                     -> stream
      ```
      Streaming returns a `stream` object with `send` / `close_send` /
      `recv() -> msg, err` / `cancel`. End-of-stream is `(nil, nil)`;
      handler errors surface as `(nil, err_string)`.
- [x] **Server side**: `MyService_server(impl)` returns
      `{service, methods, streams}` consumable by the loopback /
      multiplex transports. Handler signatures by RPC kind:
      ```lua
      unary:          function(req, ctx) -> resp
      server_stream:  function(req, stream, ctx)
      client_stream:  function(stream, ctx) -> resp
      bidi:           function(stream, ctx)
      ```
- [ ] Reference network transports (separate projects, deferred). Full
      design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md):
  - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via
    `tarantool/http`. Default external transport; works with browsers
    and `curl` without an HTTP/2 proxy.
  - `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls.
    First-class in-cluster path.
  - `pb.grpc.transport.http_client_unary` — outbound, unary only,
    via `http_client`.
  - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in
    front; the transport contract is HTTP/2-shaped so the same
    generated code works behind it.
  - Conformance anchor: `connectrpc/conformance` (same framed-runner
    shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect
    from one harness).

**Done.** The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.

### M5 — Conformance + interop  *(proto3 closed; CI wire-up + proto2 deferred)*

- [x] Wire up Google's [protobuf conformance test runner][conformance].
      `cmd/conformance-runner.lua` is the testee: reads length-prefixed
      `ConformanceRequest` on stdin, runs it through our codec / JSON /
      text codecs, writes a length-prefixed `ConformanceResponse` on
      stdout. Loops to EOF. Core dispatch lives in
      `cmd/conformance/core.lua` and is exercised directly by
      `test/conformance_test.lua` so the inner dev loop doesn't need
      Docker.
- [x] Run the canonical `conformance_test_runner` binary locally and
      track the pass-rate as a numeric metric (regression gate).
      `docker/conformance.Dockerfile` builds `conformance_test_runner`
      from upstream protobuf v34.1 source (matching the host's
      `libprotoc 34.1`) and bundles Tarantool 3 from the official deb;
      `just conformance` regenerates Lua then runs the harness against
      `cmd/conformance-runner.lua` with the repo mounted as a volume.
      Watchlists at `test/conformance/known_failures.txt` (binary +
      JSON suite) and `test/conformance/known_failures_text.txt`
      (text-format suite); **both are empty for the proto3 suites as
      of 2026-05-16.**

      Current baseline:
        - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
        - Text-format suite:  **416 ✓ /   18 skipped / 0 failures**

      The 1313 + 18 skipped all target
      `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
      is a separate slice — see [docs/codegen.md](../docs/codegen.md)
      for what it would take.

      Strict-validation closures landed across three commits on the
      `text-conformance-output` branch:
        - `pb.text.decode` — full grammar coverage (recursive-descent
          parser, ~580 LOC).
        - `codec` -0.0 preservation — float/double `is_default_scalar`
          and the inline-codegen elision both gained a sign-bit guard
          (`1/v == math.huge`).
        - `pb.json` strict-validation pass — duplicate-key rejection
          (literal + camel/snake alias detection via a byte-walking
          pre-scan), null-in-container rejection, unknown-enum-name
          rejection (with `ignore_unknown_fields` opt for the
          `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
          `google.protobuf.NullValue` round-trip as JSON `null`,
          strict FieldMask round-trip validation.

      The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
      short-circuit JSON output if a future encoder bug starts
      crashing jsoncpp.
- [ ] CI wire-up. The Docker image build is the long pole (~10–15 min
      on a clean cache); a registry push from a scheduled job would
      let CI runs reuse a warm cache.
- [x] Cross-impl interop: 18-fixture corpus in `test/interop/fixtures/`
      produced by mainline `protoc --encode`; tests assert byte-for-byte
      equality.

[conformance]: https://github.com/protocolbuffers/protobuf/tree/main/conformance

### M6 — Performance + production polish  *(delayed; bench harness shipped, further perf work parked)*

- [x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for
      messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped
      as `bench/bench.lua`, run via `make bench`. Throughput is stderr-only
      (varies with CPU load); JSON document on stdout.
- [x] Allocation profiling — bytes per encode/decode op, measured via GC
      delta with `collectgarbage('stop')` framing. Committed as
      `bench/baseline.json`. Regression gate: `make bench-compare` exits
      non-zero if alloc/op grows >5% vs baseline. Allocations are
      deterministic to ~10 bytes regardless of hardware.
- [x] Trace stability — `make jit-trace` (`bench/jit_trace.lua`)
      attaches a `jit.attach('trace')` listener over the hot
      encode/decode paths and asserts no aborts in our source files
      fall into the fatal set (NYI bytecode, blacklisting, persistent
      type instability). Pass: 13/13 scenarios. Two fixes shipped:
      decode_varint grew a 1-byte fast path so callers no longer drag
      an inner loop into the root trace; `pb.finalize_message` now
      precomputes `desc.oneofs_list` so runtime-mode oneof encoding
      uses `ipairs` instead of `pairs` (the latter compiles to bytecode
      ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
      interpreter-bridge counts as a benchmark-quality metric (decoders
      have 0–4 per run depending on JIT timing — caused by side traces
      returning from inlined `decode_varint` calls, which LuaJIT 2.1
      can't stitch back cleanly; small per-call overhead, structural
      to the engine). Scope caveat: map fields still encode via
      `pairs()` and remain off-trace — pinned by the gate's last
      scenario so we notice if upstream lifts the restriction.
- [ ] Optional output: ibuf-based encoder that writes into a caller-owned
      `ffi.cdata` byte buffer instead of building a string list. Targets
      hot RPC paths where allocation cost dominates.
- [x] Decoder fast path that returns an `msgpack.object`-like lazy view
      for nested messages; only materializes touched fields.
      Shipped as `runtime/pb/lazy.lua` + `M.<Type>_decode_lazy` codegen
      stubs in both modes. Surface: `:get / :has / :which / :iter /
      :names` on MessageView; `:len / :at / :iter / :tolist` on
      ArrayView; `:get / :has / :keys / :iter / :totable` on MapView.
      Mutation via `:set` is supported and propagates sub-view edits
      transparently (sub-MessageViews tracked on a flat array for
      JIT-stable `is_dirty` — see lazy.lua's `_sub_msg_views`).
      Re-encode is passthrough: untouched views return their original
      bytes verbatim; partially-dirty views walk fields in id order,
      splicing clean segments and re-emitting dirty ones. WKT
      descriptors (those with `desc.decode`) are eager-wrapped so the
      API stays uniform.

      Conformance: every interop fixture round-trips byte-equal through
      `decode_lazy(b):encode()`. Trace stability: gated by `make
      jit-trace` — index pass, sparse `:get` x2, and passthrough
      `:encode` all compile with no fatal aborts.

      Workload characteristics (from `tarantool bench/lazy_bench.lua`,
      Person at 1KB / 10KB / 100KB, after the SoA index refactor):
       - **Passthrough re-encode** is the headline win: **1.6–1.9×**
         faster than eager decode→encode across all sizes and both modes.
         Untouched views never re-walk the wire.
       - **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
         of eager**: break-even at small sizes, slight win at 100KB
         (especially in runtime mode where eager pays more dispatch
         cost). Earlier 0.60–0.77× regression came from one Lua table
         per wire entry; replacing with parallel int arrays closed the
         allocation gap.
       - **Mutate-then-reencode** is **1.09–1.26× of eager**; the
         per-field splice path now consistently beats full re-encode.
      The honest framing: lazy is a *byte-passthrough* optimization
      that also handles sparse reads at parity. Best fit: proxy /
      router shapes that decode, touch a few fields, and re-encode.

### M7 — Developer ergonomics

- [x] Generated EmmyLua / lua-language-server type annotations so
      `t:Person_encode({name=...})` autocompletes in editors.
      Codegen emits `---@class <full.Name>` per message (with one
      `---@field` per field), `---@alias <full.Name> integer` per
      enum, and `---@param` / `---@return` on every `_new`, `_encode`,
      `_decode`, `_decode_lazy`, `_has_*`, `_clear_*` wrapper. Class
      identifiers use proto full names verbatim so cross-file
      references resolve. Lazy view types (`pb.MessageView`,
      `pb.ArrayView`, `pb.MapView`) are declared inline in
      `runtime/pb/lazy.lua` so the LSP sees them. Pure comment
      addition — no runtime impact, 300/300 luatest + 19/19
      jit-trace gate stay green.
- [x] `pb.from_pb(file_descriptor_set)` — accepts binary `FileDescriptorSet`
      bytes (output of `protoc --descriptor_set_out=...`) and returns
      `{files = {[name] = module}, order, lookup}`. Each per-file module
      has the same surface as `pb.parse` output (statically-generated
      runtime mode). Translation pipeline: hand-built `descriptor.proto`
      descriptors decode the wire bytes via `pb.codec`, then a translator
      converts each `FileDescriptorProto` to the AST shape `pb.parser`
      emits, which `pb.dynamic.build` consumes. Map fields are
      reconstructed from synthetic entry messages (skipped from
      `nested_messages`); `proto3_optional` is rehydrated as
      `optional=true` instead of being modelled as a synthetic oneof.
- [x] JSON encoding per the [proto3 JSON spec][proto3json]
      (`pb.json.encode` / `pb.json.decode`).
- [x] Text-format printer. `pb.text.encode(desc, t, opts)` returns the
      `protoc --decode` form (one field per line, 2-space indent, octal
      byte escapes); `opts.single_line=true` collapses to a space-separated
      one-liner for log lines and inline goldens. Codegen emits
      `M.<Type>_text(t, opts)` in both modes. WKT types know their
      idiomatic Lua shapes — `Timestamp`/`Duration` accept datetime cdata
      or `{seconds,nanos}`, wrappers print their unwrapped scalar as
      `value: ...`, `Struct`/`Value`/`ListValue` walk the tagged-table
      form, `FieldMask` prints `paths: ...` per entry, `Any` stays opaque.
- [x] Text-format parser. `pb.text.decode(desc, text, opts)` is the
      recursive-descent counterpart: handles every grammar bucket the
      proto3 conformance suite exercises — decimal/hex/octal int literals,
      float specials (`inf`/`infinity`/`nan` any case, oversize exponents
      saturating to ±inf, underflow to ±0), C-style + `\u`/`\U` string
      escapes with adjacent-literal concat and surrogate rejection,
      aggregate `{}` / `<>` bodies, repeated short-form `[a, b, c]`,
      `key: K value: V` map entries, the `[type.googleapis.com/...]`
      inline Any form, enum-by-name-or-number, reserved-name drop, and
      numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
      duplicate singular fields, and threads through the conformance
      runner — `cmd/conformance/core.lua` no longer skips `text_payload`.
      Plugin gained a small `reserved_names` emitter so the parser can
      match mainline TextFormat::Parser's "silently drop reserved" rule.
      Proto3 text-format conformance suite: **8 ✓ / 426 skipped → 416
      ✓ / 18 skipped / 0 failures** (the 18 are the proto2 message-type
      bucket; everything in scope passes).
- [x] JSON strict-validation pass. Six classes of relaxation that the
      proto3 JSON conformance corpus flagged are now enforced — together
      with the -0 codec fix this empties `known_failures.txt`:
        1. Duplicate JSON keys (`{"foo":1,"foo":2}`) rejected via a
           byte-walking pre-scan that runs before `json.decode`.
        2. camelCase / snake_case aliases of the same proto field
           rejected via a per-message `field_seen` set.
        3. JSON `null` inside repeated arrays and map values rejected.
        4. Unknown enum *names* rejected by default; the
           `ignore_unknown_fields=true` opt silently drops them (and
           the conformance dispatch forwards this flag when
           `req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST`).
        5. `google.protobuf.NullValue` JSON canonical form: literal
           `null`, not the string `"NULL_VALUE"`. Null on a
           NullValue-typed oneof member marks the oneof active.
        6. Strict FieldMask round-trip (see M3 entry).
- [x] `protoc-gen-tarantool-doc`: sibling Go plugin under
      `cmd/protoc-gen-tarantool-doc/` that emits one Markdown file per
      input `.proto`. Sections: header (package + imports), messages
      (per-message description + field table with `# | Field | Type |
      Label | Description`), enums (value table), services (method table
      with `unary` / `client` / `server` / `bidi` streaming label).
      Field type cells render scalar names, full type names for
      message/enum references, and `map<K, V>` for maps; synthetic
      map-entry messages are skipped. Leading comments are preserved via
      SourceCodeInfo (squashed to a single line inside table cells).
      Build with `make build-doc`; generate sample docs into
      `examples/docs/` with `make gen-docs`.

[proto3json]: https://protobuf.dev/programming-guides/proto3/#json

### M8 — Release engineering

- [ ] Sourcecraft.dev project + CI pipeline (matrix: Tarantool 2.11 / 3.x
      EE + CE, Linux + macOS).
- [ ] Tagged releases; rockspec for the Lua runtime; pre-built binaries
      for the Go plugin.
- [ ] Migration guide from Tarantool's built-in `require('protobuf')` to
      `require('pb')`.
- [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via
      protobuf-encoded events on a queue.

## 4. Per-feature design notes

### 4.1 Inline (full) codegen — wire bytes

For each message we emit one `_encode` and one `_decode` function. Tag
bytes are precomputed string literals; field-presence checks are inlined.
The runtime is reduced to wire-format primitives.

```lua
function M.Address_encode(t)
    local out, n = {}, 0
    local v
    v = t.street
    if v ~= nil and v ~= '' then
        n = n + 1; out[n] = '\x0a'  -- tag(1, LEN)
        n = n + 1; out[n] = wire.encode_string(v)
    end
    -- ...
    return table.concat(out)
end
```

Decoder uses an `if-elseif` chain on field id (LuaJIT compiles this
well for small chain lengths; for >16 fields we may want a numeric jump
table or tag-byte switching).

### 4.2 Map fields

`map<K,V>` is wire-format-equivalent to:

```proto
message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;
```

Codegen synthesizes the entry message internally but exposes the field as a
Lua table (hash, not array). Encode iterates with `pairs`; decode merges
on duplicate keys (last wins, per spec).

### 4.3 Oneofs

Descriptor gains `oneofs = {[oneof_name] = {field_names...}}`. Encode walks
fields in declaration order and emits the first non-nil branch. Decode
clears sibling fields on assignment so callers see exactly one set.

In inline mode, the encode walk becomes an explicit `if`-chain; the decode
clears are emitted alongside each `elseif id == N then` arm.

### 4.4 64-bit integer ergonomics

Locked: `int64`/`uint64`/`fixed64`/`sfixed64`/`sint64` are LuaJIT
`int64_t` / `uint64_t` cdata. Reasons:
- Lossless beyond 2^53.
- Same convention as Tarantool `msgpackffi`, `net.box`, `box.tuple`.
- Compares cleanly against `0` (numeric coercion in LuaJIT).

We will document a `wire.from_string(s)` helper for users who get hex/dec
strings (e.g. from JSON) and need to feed them into encode.

### 4.5 Unknown fields  *(implemented)*

`t._unknown_fields` is a single Lua string holding the verbatim
concatenation of tag+value bytes for fields the decoder didn't recognize.
Captured during decode in encounter order; re-emitted at the tail of
`_encode`. Mirrors Tarantool's built-in `protobuf` module convention.

Map entries don't preserve unknowns (per spec — synthetic Entry messages).
WKT types bypass this too, since they have custom `desc.encode/decode`.

Both codegen modes implement it: runtime mode in `pb.codec`, full (inline)
mode emits per-message capture/re-emit blocks. Tests live in
`test/unknown_test.lua` and run against both modes.

### 4.6 gRPC service descriptors

```lua
M.Greeter_service = {
    name = 'hello.Greeter',
    methods = {
        SayHello = {
            full_name = '/hello.Greeter/SayHello',
            input = M.HelloRequest_descriptor,
            output = M.HelloReply_descriptor,
            client_streaming = false,
            server_streaming = false,
        },
        -- ...
    },
}
```

Client: `M.Greeter_client(transport)` returns
`{SayHello = function(req) ... end, ...}`.

Server: `M.Greeter_server(impl)` returns a table compatible with the
`server:register(svc)` interface.

## 5. Testing strategy

Tests live in `test/`. Layout:

```
test/
  unit/           -- pb.wire and pb.codec unit tests (no codegen)
    wire_test.lua
    codec_test.lua
  roundtrip/      -- generated-module round-trip, both modes
    scalars_test.lua
    repeated_test.lua
    nested_test.lua
    map_test.lua            -- M2
    oneof_test.lua          -- M2
    wkt_test.lua            -- M3
  conformance/    -- M5: Google conformance harness
  interop/        -- M5: cross-impl byte-for-byte equality
    fixtures/     -- pre-encoded payloads from Go/Python
  bench/          -- M6: microbenchmarks
  fuzz/           -- malformed-input + random-input harness
```

### 5.1 Unit tests (M1 onwards)

For each `wire.encode_<type>` / `wire.decode_<type>` pair:
- Round-trip 0, min, max, edge cases (1, -1, NaN, Inf, empty string,
  254/255/256-byte string for varint length-byte boundaries).
- Truncated input → controlled error.
- Spec-conformant byte sequences from the [protobuf encoding doc][encoding].

[encoding]: https://protobuf.dev/programming-guides/encoding/

### 5.2 Round-trip parity tests (M1+)

Every test runs against **both** generated modules (full and runtime mode)
using a parameterized luatest group:

```lua
for _, mode in ipairs({'full', 'runtime'}) do
    local g = t.group('roundtrip.' .. mode)
    local hello = require('hello.' .. mode .. '.hello_pb')
    g.test_address = function() ... end
    -- ...
end
```

Coverage targets:
- All 15 scalar types, singular and repeated.
- Packed vs non-packed repeated (default + explicit).
- Nested messages, self-references, mutually recursive cycles.
- Cross-file imports (2-file fixture).
- Maps with all valid key types (M2).
- Oneofs with all 3 wire-type families (M2).
- Default-value elision (proto3 semantics).
- Explicit-optional presence (M2).
- Empty messages, single-field messages, 100-field messages.
- Field IDs straddling varint boundaries: 1, 15, 16, 2047, 2048, 2^29 - 1.

### 5.3 Conformance suite (M5)

Build a small Lua binary `cmd/conformance-runner.lua` that speaks the
Google protobuf conformance protocol on stdin/stdout. Checked into the
repo so CI runs it against the canonical test corpus. Track pass-rate as
a CI metric.

### 5.4 Cross-impl interop (M5)

For each of N reference protos:
1. `protoc --encode=msg < input.txt > golden.bin` (using mainline protoc).
2. Our test asserts `pb.decode(msg, read('golden.bin'))` produces the
   expected Lua table.
3. Asserts `pb.encode(msg, expected)` produces bytes that, when decoded
   by mainline protoc, yield the same logical message (allow field
   reordering since neither order is canonical).

Generate goldens once via a `make goldens` target; commit them to the
repo. CI just verifies them, never regenerates.

### 5.5 Fuzz tests (M5/M6)

- **Malformed-input fuzz**: feed random bytes to every `_decode` function;
  assert no crashes, only controlled `error()` calls.
- **Round-trip fuzz**: generate random-but-valid messages (size-bounded),
  assert `decode(encode(t)) == t` (value-equal under our equality helper).
- Use Tarantool's `math.random` with a seeded PRNG for reproducibility.

### 5.6 Performance tests (M6)

Microbenchmarks measured:
- Encode + decode throughput (msgs/sec, bytes/sec).
- Allocation rate (tables/sec, strings/sec) via `misc.memprof`.
- JIT trace count (no side traces in hot loops).

Run on a fixed corpus across 5 message sizes; track results in
`bench/baseline.json` and fail PRs that regress >5%.

### 5.7 Test infrastructure

- **luatest** as the framework. Already installed via
  `tt rocks install luatest` to `.rocks/`.
- Run via `make test` → `.rocks/bin/luatest -v test/`.
- CI: sourcecraft.dev native pipelines (TBD), matrix on Tarantool versions.
- Go side: `go test ./...` for any pure-Go logic in the plugin.
- Go integration test: spawn `protoc` on a fixture, diff generated Lua
  against committed expected output.

## 6. Tooling roadmap

- **Justfile** (or extend Makefile) with targets: `build`, `gen`,
  `test`, `bench`, `lint`, `goldens`, `conformance`, `clean`, `release`.
- **golangci-lint** for the Go side; **luacheck** for the Lua side.
- **gofumpt** + **stylua** for formatting.
- **Coverage**: `go test -cover` for plugin; `luacov` for runtime.
- **Doc generation**: `pkgsite` for Go; manually maintained Markdown for
  Lua until a tool emerges.

## 7. Open questions / future decisions

| Question | Notes |
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. |
| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. |
| How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. |
| Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
| Integration with Tarantool 3.x declarative config? | A `pb.types.<name>` config role would be nice; defer until use case appears. |

## 8. Non-goals (for now)

- **proto2** syntax — spec is more complex (required, default values,
  groups), real demand is rare on Tarantool.
- **Protobuf editions** beyond what proto3 enables.
- **HTTP/2 termination** for gRPC. Push it to Envoy; see
  [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md).
  Connect-JSON over HTTP/1.1 is the default external transport.
- **Reflection service** (gRPC server reflection) — implement after M5.
- **gRPC-Web** — covered by the same Connect conformance harness if we
  add it, but not on the v1 path.

## 9. How to update this plan

Edit this file directly. After any milestone closes:
1. Move the milestone section under a `## Done` heading.
2. Cross-link the closing PR.
3. Update `README.md` status table.
4. Cut a release on sourcecraft.dev.