~bigbes/tarantool

tarantool-protobuf

ref: 4db0366a5008e874de1e26f690f4bff499de51bf tarantool-protobuf/PLAN.md -rw-r--r-- 33.6 KiB
4db0366a — Eugene Blikh runtime: parser + dynamic accept proto2 sources 3 months ago

#tarantool-protobuf — Roadmap & Test Plan

This document tracks the long-form design + testing strategy for protoc-gen-tarantool and the pb Lua runtime. Short-term task tracking lives in the in-session task list; this file is the slow-changing record.

#1. Vision

A first-class Protocol Buffers + gRPC stack for Tarantool that:

  • Generates idiomatic Lua from .proto files with two modes (full inline vs descriptor-driven runtime) so users can pick speed-vs-flexibility.
  • Interoperates byte-for-byte with mainline protoc implementations (Go, Python, C++) — passes Google's protobuf conformance suite for proto3.
  • Ships with a small, FFI-aware runtime that respects Tarantool conventions (LuaJIT cdata for 64-bit ints, no global state, no monkey-patching).
  • Is comfortable to use from real Tarantool apps: clean integration with net.box, fiber, box.session, and the 3.x config framework.

#2. Current state (post-M7)

See README.md for the user-visible summary. Internally:

  • Plugin: Go, two emission modes (mode=full inline + mode=runtime descriptor-delegating wrappers). Sibling protoc-gen-tarantool-doc emits Markdown reference per .proto.
  • Runtime: pure Lua + LuaJIT FFI. wire.lua + codec.lua
    • lazy.lua (zero-copy view) + text.lua (encode + decode) + json.lua (strict proto3 JSON) + wkt.lua (all 9 well-known types) + grpc.lua (transport + loopback) + parser.lua / dynamic.lua / fileset.lua (three runtime descriptor producers — .proto source, AST, FileDescriptorSet bytes).
  • Tests: 639 luatest tests across 15 test files (39 groups), parametrized over both codegen modes where applicable.
  • Conformance: proto3 binary+JSON suite 1493 ✓ / 0 failures; proto3 text-format suite 416 ✓ / 0 failures (Google's conformance_test_runner v34.1).
  • Bench: bench/baseline.json tracks allocation/op (regression gate at 5%); just jit-trace pins LuaJIT trace stability.

The remaining unfinished bullets in section 3 are mostly M8 (release engineering) and a handful of optional perf items in M6.

#3. Phased roadmap

Each milestone ends with a green CI run, an updated README, and a tagged release on sourcecraft.dev.

#M1 — Two codegen modes + luatest harness (done)

  • [x] Promote scalar encode/decode to typed helpers in pb.wire. wire.encode_int32(v), wire.decode_string(buf, pos), etc., for all 15 scalar proto types. Both modes consume the same primitives.
  • [x] Full (inline) codegen: emit per-message _encode / _decode functions with no descriptor lookup. Tag bytes precomputed at gen time as Lua string literals. This is the JIT-friendly hot path.
  • [x] Runtime codegen: keep current behavior. Useful for introspection, schema registries, and forward-compat with descriptor-only consumers.
  • [x] Plugin parameter mode=full|runtime (default: full).
  • [x] Migrate tests to luatest groups. Every behavior tested against both generated modules to prove parity.
  • [x] Generate side-by-side outputs in examples/expected/{full,runtime}/ for visual diffing.

#M2 — Composite types (done)

  • [x] map<K,V>: emit as repeated synthetic *Entry messages with key/value fields per spec. Decode merges into a Lua table; encode iterates with pairs. Key types limited to scalars + string per spec.
  • [x] oneof: descriptor includes oneof_index per field. Encode emits at most one branch (last assignment wins). Decode clears prior oneof siblings on assignment. Hot-path lookup goes through desc.oneofs_list (flat array) to keep the trace JIT-stable.
  • [x] proto3 explicit optional: respect presence — emit field even when value equals scalar default. Generated descriptor exposes has_<name>(t) / clear_<name>(t) helpers.

#M3 — Well-known types (done)

  • [x] google.protobuf.Timestamp ↔ Tarantool datetime module (epoch + nsec mapping). Out-of-spec inputs (negative nanos, year > 9999) keep the raw {seconds, nanos} table so JSON serialization can reject them with serialize_error instead of crashing on decode.
  • [x] google.protobuf.Duration{seconds, nanos} table (interval proved a poor fit — it carries months/days that don't map cleanly).
  • [x] Wrappers (Int32Value, StringValue, BoolValue, …) with sugar: pass plain Lua value → auto-wrap; decode → auto-unwrap.
  • [x] FieldMask: repeated string. Strict round-trip validation — snake_case paths must use only [a-z0-9_], no leading/trailing _, no __, and _ must precede a lowercase letter (not a digit). JSON form rejects any _ (must be lowerCamelCase).
  • [x] Empty.
  • [x] Any: opaque {type_url, value} form by default; pb.register(desc) + pb.any.pack(desc, t) / pb.any.unpack(any_t) for typed round-trips. JSON canonical mapping emits the flat {"@type": ...} object when the type is registered, falls back to base64 opaque form otherwise.
  • [x] Struct, Value, ListValue ↔ idiomatic Lua tables. box.NULL is the null_value sentinel (re-exported as pb.NULL). pb.wkt.struct(t) / pb.wkt.list(t) tag tables when the auto-detect heuristic (t[1] ~= nil → list, else struct) needs to be overridden, and decode preserves the tag for byte-stable round trips.

Done when: WKT-using protos round-trip and integrate visibly with datetime (e.g. os.date(...)-comparable timestamps).

#M4 — gRPC services (transport-agnostic) (done)

  • [x] Per-service descriptor: methods with full path /pkg.Svc/Method, input/output type refs, client_streaming / server_streaming flags.
  • [x] Client stub: MyService_client(transport) returns a table with one function per method. Transport interface, all served by the shipped loopback + multiplex implementations: lua transport:unary(path, req_bytes, ctx) -> resp_bytes transport:server_stream(path, req_bytes, ctx) -> stream transport:client_stream(path, ctx) -> stream transport:bidi(path, ctx) -> stream Streaming returns a stream object with send / close_send / recv() -> msg, err / cancel. End-of-stream is (nil, nil); handler errors surface as (nil, err_string).
  • [x] Server side: MyService_server(impl) returns {service, methods, streams} consumable by the loopback / multiplex transports. Handler signatures by RPC kind: lua unary: function(req, ctx) -> resp server_stream: function(req, stream, ctx) client_stream: function(stream, ctx) -> resp bidi: function(stream, ctx)
  • [ ] Reference network transports (separate projects, deferred). Full design in docs/specs/grpc_transports.md:
    • pb.grpc.transport.http_server — Connect-JSON over HTTP/1.1 via tarantool/http. Default external transport; works with browsers and curl without an HTTP/2 proxy.
    • pb.grpc.transport.netbox — gRPC tunneled over net.box calls. First-class in-cluster path.
    • pb.grpc.transport.http_client_unary — outbound, unary only, via http_client.
    • HTTP/2 termination — explicitly not shipped. Recommend Envoy in front; the transport contract is HTTP/2-shaped so the same generated code works behind it.
    • Conformance anchor: connectrpc/conformance (same framed-runner shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect from one harness).

Done. The loopback transport runs each streaming handler on its own fiber and bridges client ↔ handler via fiber.channel. All four flavors (unary + 3 streaming) are exercised by parameterized luatest groups.

#M5 — Conformance + interop (proto3 closed; CI wire-up + proto2 deferred)

  • [x] Wire up Google's protobuf conformance test runner. cmd/conformance-runner.lua is the testee: reads length-prefixed ConformanceRequest on stdin, runs it through our codec / JSON / text codecs, writes a length-prefixed ConformanceResponse on stdout. Loops to EOF. Core dispatch lives in cmd/conformance/core.lua and is exercised directly by test/conformance_test.lua so the inner dev loop doesn't need Docker.

  • [x] Run the canonical conformance_test_runner binary locally and track the pass-rate as a numeric metric (regression gate). docker/conformance.Dockerfile builds conformance_test_runner from upstream protobuf v34.1 source (matching the host's libprotoc 34.1) and bundles Tarantool 3 from the official deb; just conformance regenerates Lua then runs the harness against cmd/conformance-runner.lua with the repo mounted as a volume. Watchlists at test/conformance/known_failures.txt (binary + JSON suite) and test/conformance/known_failures_text.txt (text-format suite); both are empty for the proto3 suites as of 2026-05-16.

    Current baseline:
      - Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
      - Text-format suite:  **416 ✓ /   18 skipped / 0 failures**
    
    The 1313 + 18 skipped all target
    `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
    is a separate slice — see [docs/codegen.md](../docs/codegen.md)
    for what it would take.
    
    Strict-validation closures landed across three commits on the
    `text-conformance-output` branch:
      - `pb.text.decode` — full grammar coverage (recursive-descent
        parser, ~580 LOC).
      - `codec` -0.0 preservation — float/double `is_default_scalar`
        and the inline-codegen elision both gained a sign-bit guard
        (`1/v == math.huge`).
      - `pb.json` strict-validation pass — duplicate-key rejection
        (literal + camel/snake alias detection via a byte-walking
        pre-scan), null-in-container rejection, unknown-enum-name
        rejection (with `ignore_unknown_fields` opt for the
        `JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
        `google.protobuf.NullValue` round-trip as JSON `null`,
        strict FieldMask round-trip validation.
    
    The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
    short-circuit JSON output if a future encoder bug starts
    crashing jsoncpp.
    
  • [ ] CI wire-up. The Docker image build is the long pole (~10–15 min on a clean cache); a registry push from a scheduled job would let CI runs reuse a warm cache.

  • [x] Cross-impl interop: 18-fixture corpus in test/interop/fixtures/ produced by mainline protoc --encode; tests assert byte-for-byte equality.

#M6 — Performance + production polish (delayed; bench harness shipped, further perf work parked)

  • [x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped as bench/bench.lua, run via just bench. Throughput is stderr-only (varies with CPU load); JSON document on stdout.

  • [x] Allocation profiling — bytes per encode/decode op, measured via GC delta with collectgarbage('stop') framing. Committed as bench/baseline.json. Regression gate: just bench-compare exits non-zero if alloc/op grows >5% vs baseline. Allocations are deterministic to ~10 bytes regardless of hardware.

  • [x] Trace stability — just jit-trace (bench/jit_trace.lua) attaches a jit.attach('trace') listener over the hot encode/decode paths and asserts no aborts in our source files fall into the fatal set (NYI bytecode, blacklisting, persistent type instability). Pass: 13/13 scenarios. Two fixes shipped: decode_varint grew a 1-byte fast path so callers no longer drag an inner loop into the root trace; pb.finalize_message now precomputes desc.oneofs_list so runtime-mode oneof encoding uses ipairs instead of pairs (the latter compiles to bytecode ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports interpreter-bridge counts as a benchmark-quality metric (decoders have 0–4 per run depending on JIT timing — caused by side traces returning from inlined decode_varint calls, which LuaJIT 2.1 can't stitch back cleanly; small per-call overhead, structural to the engine). Scope caveat: map fields still encode via pairs() and remain off-trace — pinned by the gate's last scenario so we notice if upstream lifts the restriction.

  • [ ] Optional output: ibuf-based encoder that writes into a caller-owned ffi.cdata byte buffer instead of building a string list. Targets hot RPC paths where allocation cost dominates.

  • [x] Decoder fast path that returns an msgpack.object-like lazy view for nested messages; only materializes touched fields. Shipped as runtime/pb/lazy.lua + M.<Type>_decode_lazy codegen stubs in both modes. Surface: :get / :has / :which / :iter / :names on MessageView; :len / :at / :iter / :tolist on ArrayView; :get / :has / :keys / :iter / :totable on MapView. Mutation via :set is supported and propagates sub-view edits transparently (sub-MessageViews tracked on a flat array for JIT-stable is_dirty — see lazy.lua's _sub_msg_views). Re-encode is passthrough: untouched views return their original bytes verbatim; partially-dirty views walk fields in id order, splicing clean segments and re-emitting dirty ones. WKT descriptors (those with desc.decode) are eager-wrapped so the API stays uniform.

    Conformance: every interop fixture round-trips byte-equal through
    `decode_lazy(b):encode()`. Trace stability: gated by `make
    jit-trace` — index pass, sparse `:get` x2, and passthrough
    `:encode` all compile with no fatal aborts.
    
    Workload characteristics (from `tarantool bench/lazy_bench.lua`,
    Person at 1KB / 10KB / 100KB, after the SoA index refactor):
     - **Passthrough re-encode** is the headline win: **1.6–1.9×**
       faster than eager decode→encode across all sizes and both modes.
       Untouched views never re-walk the wire.
     - **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
       of eager**: break-even at small sizes, slight win at 100KB
       (especially in runtime mode where eager pays more dispatch
       cost). Earlier 0.60–0.77× regression came from one Lua table
       per wire entry; replacing with parallel int arrays closed the
       allocation gap.
     - **Mutate-then-reencode** is **1.09–1.26× of eager**; the
       per-field splice path now consistently beats full re-encode.
    The honest framing: lazy is a *byte-passthrough* optimization
    that also handles sparse reads at parity. Best fit: proxy /
    router shapes that decode, touch a few fields, and re-encode.
    

#M7 — Developer ergonomics (done)

  • [x] Generated EmmyLua / lua-language-server type annotations so t:Person_encode({name=...}) autocompletes in editors. Codegen emits ---@class <full.Name> per message (with one ---@field per field), ---@alias <full.Name> integer per enum, and ---@param / ---@return on every _new, _encode, _decode, _decode_lazy, _has_*, _clear_* wrapper. Class identifiers use proto full names verbatim so cross-file references resolve. Lazy view types (pb.MessageView, pb.ArrayView, pb.MapView) are declared inline in runtime/pb/lazy.lua so the LSP sees them. Pure comment addition — no runtime impact, 300/300 luatest + 19/19 jit-trace gate stay green.
  • [x] pb.from_pb(file_descriptor_set) — accepts binary FileDescriptorSet bytes (output of protoc --descriptor_set_out=...) and returns {files = {[name] = module}, order, lookup}. Each per-file module has the same surface as pb.parse output (statically-generated runtime mode). Translation pipeline: hand-built descriptor.proto descriptors decode the wire bytes via pb.codec, then a translator converts each FileDescriptorProto to the AST shape pb.parser emits, which pb.dynamic.build consumes. Map fields are reconstructed from synthetic entry messages (skipped from nested_messages); proto3_optional is rehydrated as optional=true instead of being modelled as a synthetic oneof.
  • [x] JSON encoding per the proto3 JSON spec (pb.json.encode / pb.json.decode).
  • [x] Text-format printer. pb.text.encode(desc, t, opts) returns the protoc --decode form (one field per line, 2-space indent, octal byte escapes); opts.single_line=true collapses to a space-separated one-liner for log lines and inline goldens. Codegen emits M.<Type>_text(t, opts) in both modes. WKT types know their idiomatic Lua shapes — Timestamp/Duration accept datetime cdata or {seconds,nanos}, wrappers print their unwrapped scalar as value: ..., Struct/Value/ListValue walk the tagged-table form, FieldMask prints paths: ... per entry, Any stays opaque.
  • [x] Text-format parser. pb.text.decode(desc, text, opts) is the recursive-descent counterpart: handles every grammar bucket the proto3 conformance suite exercises — decimal/hex/octal int literals, float specials (inf/infinity/nan any case, oversize exponents saturating to ±inf, underflow to ±0), C-style + \u/\U string escapes with adjacent-literal concat and surrogate rejection, aggregate {} / <> bodies, repeated short-form [a, b, c], key: K value: V map entries, the [type.googleapis.com/...] inline Any form, enum-by-name-or-number, reserved-name drop, and numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects duplicate singular fields, and threads through the conformance runner — cmd/conformance/core.lua no longer skips text_payload. Plugin gained a small reserved_names emitter so the parser can match mainline TextFormat::Parser's "silently drop reserved" rule. Proto3 text-format conformance suite: 8 ✓ / 426 skipped → 416 ✓ / 18 skipped / 0 failures (the 18 are the proto2 message-type bucket; everything in scope passes).
  • [x] JSON strict-validation pass. Six classes of relaxation that the proto3 JSON conformance corpus flagged are now enforced — together with the -0 codec fix this empties known_failures.txt: 1. Duplicate JSON keys ({"foo":1,"foo":2}) rejected via a byte-walking pre-scan that runs before json.decode. 2. camelCase / snake_case aliases of the same proto field rejected via a per-message field_seen set. 3. JSON null inside repeated arrays and map values rejected. 4. Unknown enum names rejected by default; the ignore_unknown_fields=true opt silently drops them (and the conformance dispatch forwards this flag when req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST). 5. google.protobuf.NullValue JSON canonical form: literal null, not the string "NULL_VALUE". Null on a NullValue-typed oneof member marks the oneof active. 6. Strict FieldMask round-trip (see M3 entry).
  • [x] Preserve descriptor options. Every populated *Options message surfaces on the generated descriptor as a plain Lua sub-table named options (or oneof_options / value_options for oneof / enum value). Standard fields use their proto name; extensions use their fully-qualified name (bracket-quoted). The walker is generic — consumers pull (google.api.http), (versionpb.etcd_version_*), [deprecated = true], and any in-house extension without pb knowing about them. Option-free protos emit no options keys (byte-identical output to before). Resolver re-links Options messages so in-file extensions surface via the protoreflect walker (protogen builds f.Desc before in-file extensions are registered; we rebuild manually). See docs/codegen.md for the per-descriptor shape and the README's "Descriptor options" section for examples.
  • [x] protoc-gen-tarantool-doc: sibling Go plugin under cmd/protoc-gen-tarantool-doc/ that emits one Markdown file per input .proto. Sections: header (package + imports), messages (per-message description + field table with # | Field | Type | Label | Description), enums (value table), services (method table with unary / client / server / bidi streaming label). Field type cells render scalar names, full type names for message/enum references, and map<K, V> for maps; synthetic map-entry messages are skipped. Leading comments are preserved via SourceCodeInfo (squashed to a single line inside table cells). Build with just build-doc; generate sample docs into examples/docs/ with just gen-docs.

#M8 — Release engineering

  • [ ] Sourcecraft.dev project + CI pipeline (matrix: Tarantool 2.11 / 3.x EE + CE, Linux + macOS).
  • [ ] Tagged releases; rockspec for the Lua runtime; pre-built binaries for the Go plugin.
  • [ ] Migration guide from Tarantool's built-in require('protobuf') to require('pb').
  • [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via protobuf-encoded events on a queue.

#4. Per-feature design notes

#4.1 Inline (full) codegen — wire bytes

For each message we emit one _encode and one _decode function. Tag bytes are precomputed string literals; field-presence checks are inlined. The runtime is reduced to wire-format primitives.

function M.Address_encode(t)
    local out, n = {}, 0
    local v
    v = t.street
    if v ~= nil and v ~= '' then
        n = n + 1; out[n] = '\x0a'  -- tag(1, LEN)
        n = n + 1; out[n] = wire.encode_string(v)
    end
    -- ...
    return table.concat(out)
end

Decoder uses an if-elseif chain on field id (LuaJIT compiles this well for small chain lengths; for >16 fields we may want a numeric jump table or tag-byte switching).

#4.2 Map fields

map<K,V> is wire-format-equivalent to:

message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;

Codegen synthesizes the entry message internally but exposes the field as a Lua table (hash, not array). Encode iterates with pairs; decode merges on duplicate keys (last wins, per spec).

#4.3 Oneofs

Descriptor gains oneofs = {[oneof_name] = {field_names...}}. Encode walks fields in declaration order and emits the first non-nil branch. Decode clears sibling fields on assignment so callers see exactly one set.

In inline mode, the encode walk becomes an explicit if-chain; the decode clears are emitted alongside each elseif id == N then arm.

#4.4 64-bit integer ergonomics

Locked: int64/uint64/fixed64/sfixed64/sint64 are LuaJIT int64_t / uint64_t cdata. Reasons:

  • Lossless beyond 2^53.
  • Same convention as Tarantool msgpackffi, net.box, box.tuple.
  • Compares cleanly against 0 (numeric coercion in LuaJIT).

Two coercion helpers — pb.to_uint64(v) and pb.to_int64(v) (re-exported from pb.wire) — accept Lua number, cdata, or a numeric string and return the matching 64-bit cdata. Use them at the boundary when values come from JSON, text format, or net.box arguments where the type isn't already cdata.

#4.5 Unknown fields (implemented)

t._unknown_fields is a single Lua string holding the verbatim concatenation of tag+value bytes for fields the decoder didn't recognize. Captured during decode in encounter order; re-emitted at the tail of _encode. Mirrors Tarantool's built-in protobuf module convention.

Map entries don't preserve unknowns (per spec — synthetic Entry messages). WKT types bypass this too, since they have custom desc.encode/decode.

Both codegen modes implement it: runtime mode in pb.codec, full (inline) mode emits per-message capture/re-emit blocks. Tests live in test/unknown_test.lua and run against both modes.

#4.6 Protobuf editions (not implemented; tracking)

We advertise CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL only, not FEATURE_SUPPORTS_EDITIONS. If protoc is invoked against an edition = "2023"; file with our plugin, it errors out — mainline refuses to call an editions-unaware plugin.

What editions changes. Editions replaces the proto2-vs-proto3 split with one language whose behavior is controlled by per-file / per-field FeatureSet annotations. The knobs that matter for us:

  • field_presence = EXPLICIT | IMPLICIT | LEGACY_REQUIRED — presence becomes an opt-in/opt-out per field instead of a syntax-wide default.
  • repeated_field_encoding = PACKED | EXPANDED — per-field opt-out from packed encoding for scalar repeated fields.
  • enum_type = OPEN | CLOSED — closed enums route unknown values to unknown fields (proto2 semantics) instead of round-tripping the int.
  • utf8_validation = VERIFY | NONE — per-field strictness on string.
  • json_format = ALLOW | LEGACY_BEST_EFFORT — JSON behavior overrides.

What we'd actually gain. Honestly modest, ranked by user-visible value:

  1. Per-field packed opt-out — the only knob proto3 doesn't expose today. Real interop scenarios (talking to legacy proto2 services, certain gRPC gateways) sometimes need expanded encoding on a specific field. Right now our users have no escape hatch.
  2. Migration path off proto3. Proto3 syntax is being phased out in favor of edition 2023. This is a forcing function, not a feature win — at some point users will write edition = "2023"; and we'll need to read it.
  3. Explicit presence by default without the optional keyword (and the synthetic-oneof wart it creates today).
  4. Per-field UTF-8 toggle for users carrying not-quite-UTF-8 bytes in string fields for legacy compat.
  5. Closed enums for users who want proto2-style unknown-value routing.
  6. LEGACY_REQUIRED field presence — niche; only way to express "required" without writing proto2.

What it would cost. Setting the bit is one line; the work is:

  • Plugin (cmd/protoc-gen-tarantool/internal/gen) needs to read each field's resolved FeatureSet and drive codegen off it (packed, presence, UTF-8 strictness, enum closedness) instead of the proto3 defaults baked in.
  • Runtime parser (runtime/pb/parser.lua) needs to accept edition = "2023"; and the features = { ... } option syntax.
  • runtime/pb/dynamic.lua and runtime/pb/fileset.lua need to propagate resolved features onto the descriptor. The descriptor contract in docs/codegen.md would gain a features field per message/field/enum.
  • CodeGeneratorResponse needs minimum_edition / maximum_edition set, else protoc rejects an editions-supporting plugin.
  • The conformance suite has editions-specific buckets we'd start encountering.

Current verdict. Deferred. No user pull today, proto3 will keep working for years, and the practical-win list above is mostly small quality-of-life items rather than blocking gaps. Revisit when (a) someone needs per-field packed control, (b) the conformance suite starts gating on editions, or (c) protoc deprecates proto3 syntax hard enough that real users hit the error.

#4.7 gRPC service descriptors

M.Greeter_service = {
    name = 'hello.Greeter',
    methods = {
        SayHello = {
            full_name = '/hello.Greeter/SayHello',
            input = M.HelloRequest_descriptor,
            output = M.HelloReply_descriptor,
            client_streaming = false,
            server_streaming = false,
        },
        -- ...
    },
}

Client: M.Greeter_client(transport) returns {SayHello = function(req) ... end, ...}.

Server: M.Greeter_server(impl) returns a table compatible with the server:register(svc) interface.

#5. Testing strategy

Tests live as a flat test/*.lua set (15 files, 39 luatest groups, 639 assertions as of 2026-05-16). Each behavior file is parametrized over both codegen modes via the pattern:

for _, mode in ipairs({'full', 'runtime'}) do
    local g = t.group(name .. '.' .. mode)
    local hello = require(mode .. '.hello.hello_pb')
    g.test_x = function() ... end
end

Categories — see file names under test/:

  • Wire + codec primitives — exercised indirectly through protobuf_test (scalars, repeated, nested, optional, oneof) and interop_test (the byte-for-byte fixture corpus under test/interop/fixtures/, 10 .txtpb/.bin pairs from mainline protoc --encode).
  • Compositesprotobuf_test for map/oneof/explicit-optional.
  • WKTstruct_value_test, any_fieldmask_test.
  • Lazylazy_test covers MessageView/ArrayView/MapView + the byte-splice re-encode path.
  • Codec dialectsjson_test, text_test, text_decode_test, unknown_test.
  • Dynamic descriptorsdynamic_test (.proto source via pb.parse), fileset_test (FileDescriptorSet via pb.from_pb), with parity assertions against the generated modules.
  • Codegen surfacecodegen_doc_comments_test, doc_test (the protoc-gen-tarantool-doc plugin), conformance_test (self-test of the runner against crafted requests).
  • gRPCgrpc_streaming_test against the loopback transport.

Cross-mode parity is pinned by parity.full_vs_runtime.* groups inside each behavior file.

#Conformance and interop

The Google protobuf conformance suite is driven by cmd/conformance-runner.lua (stdin/stdout framing) and run via just conformance (Docker-bundled conformance_test_runner). Known failures live in test/conformance/known_failures.txt (binary+JSON) and test/conformance/known_failures_text.txt (text-format) — both empty for proto3 as of 2026-05-16.

Interop fixtures (test/interop/fixtures/*.{txtpb,bin}) are produced by mainline protoc --encode via just goldens and asserted in interop_test.lua for round-trip equality across both codegen modes. Map fixtures use a single key to lock byte-for-byte equality (Lua pairs ordering won't match protoc's text-proto-order output); multi-key map behavior is covered via decode-then-compare-Lua-table rather than byte equality.

#Performance + JIT-trace stability

bench/baseline.json tracks allocation per op (a deterministic metric; throughput swings 30%+ across machines). just bench-compare fails on >5% alloc regression. just jit-trace (bench/jit_trace.lua) asserts the hot encode/decode paths stay on the JIT trace — a companion safety net for the "no pairs() on hot paths" rule.

#Not (yet) automated

  • Fuzz harnesses (malformed-input + random-valid) — out of band.
  • Go-side go test. Codegen correctness is asserted end-to-end via Lua tests against committed expected outputs in examples/expected/{full,runtime}/.
  • CI pipeline. Local-only today; sourcecraft.dev wire-up is M8.

#6. Tooling roadmap

  • Makefile + Justfile — both shipped. Makefile covers build, gen, test, goldens, bench, bench-baseline, bench-compare, jit-trace, clean. Justfile adds just conformance (Docker-bundled conformance runner).
  • golangci-lint for the Go side; luacheck for the Lua side.
  • gofumpt + stylua for formatting.
  • Coverage: go test -cover for plugin; luacov for runtime.
  • Doc generation: pkgsite for Go; manually maintained Markdown for Lua until a tool emerges.

#7. Open questions / future decisions

Question Notes
Should map encoder be deterministic (sorted by key)? Spec says no; some users want yes. Add pb.encode_deterministic flag later.
Public C-FFI accelerator for varint? Defer until perf benchmarks show pure-Lua bottleneck.
Should we ship a stub HTTP/2 transport for gRPC? Answered in docs/specs/grpc_transports.md: no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport.
msgpack-flavored encoder for proto schemas? Design sketched in docs/specs/msgpack_encoding.md. Open: map-keyed-by-int (default) vs name; ARRAY layout for box.space feeders; MP_TUPLE ext opt-in.
How to handle the existing Tarantool-builtin require('protobuf')? Document migration; do not override the loader.
Lua module path mapping when no lua_package and no proto package? Currently uses bare filename; consider erroring out instead.
Schema upgrade support (v1 → v2 of a message)? Out of scope; protobuf is forward-compatible by design.
Integration with Tarantool 3.x declarative config? A pb.types.<name> config role would be nice; defer until use case appears.

#8. Non-goals (for now)

  • proto2 syntax — spec is more complex (required, default values, groups), real demand is rare on Tarantool.
  • Protobuf editions beyond what proto3 enables — see §4.6 for the per-feature analysis (what we'd gain, what it would cost, the conditions that would force a revisit).
  • HTTP/2 termination for gRPC. Push it to Envoy; see docs/specs/grpc_transports.md. Connect-JSON over HTTP/1.1 is the default external transport.
  • Reflection service (gRPC server reflection) — implement after M5.
  • gRPC-Web — covered by the same Connect conformance harness if we add it, but not on the v1 path.

#9. How to update this plan

Edit this file directly. After any milestone closes:

  1. Move the milestone section under a ## Done heading.
  2. Cross-link the closing PR.
  3. Update README.md status table.
  4. Cut a release on sourcecraft.dev.