#tarantool-protobuf

A protoc plugin and pure-Lua runtime for using Protocol Buffers (proto3) and gRPC service stubs from Tarantool.

Tarantool ships an in-tree require('protobuf') module, but it is encode-only and has no support for map, oneof, services, or any decode path. This project fills those gaps with:

• protoc-gen-tarantool — a protoc plugin (Go) that turns .proto files into Lua modules.
• runtime/pb — a pure Lua + LuaJIT-FFI runtime the generated code uses for the wire format. Named pb rather than protobuf to avoid colliding with Tarantool's built-in module.

#Status

Proto3 conformance is closed: every Required.* and Recommended.* test in both the binary+JSON and text-format suites passes.

#Quick start

# 1. Build the plugin and generate the example.
make gen

# 2. Run the round-trip test in Tarantool.
make test

The plugin emits one .lua file per .proto. By default the output path mirrors the proto package (package foo.bar; baz.proto → foo/bar/baz_pb.lua), required as foo.bar.baz_pb. Override the Lua module path with a file option:

import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";

#Generated API

For each message Foo the plugin emits:

local M = require('myapp.proto.foo')

M.Foo_descriptor       -- the descriptor table consumed by the runtime
M.Foo_new(t)           -- returns t (or {}); placeholder for future validation
M.Foo_encode(t)        -- table -> wire bytes (string)
M.Foo_decode(b)        -- wire bytes (string) -> table
M.Foo_decode_lazy(b)   -- wire bytes -> MessageView (zero-copy view)
M.Foo_text(t, opts)    -- table -> protoc-style text format (debug printer)
M.Foo_has_<field>(t)   -- only emitted for proto3 explicit-`optional` fields
M.Foo_clear_<field>(t) -- same

Text-format decoding is exposed on the runtime as pb.text.decode(desc, text, opts) (no per-message wrapper — it's used from a few places, like the conformance runner, and didn't warrant codegen surface).

For each enum Color:

M.Color_descriptor          -- { name, by_name, by_value }
M.Color                     -- alias for by_name: M.Color.RED -> 0

Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers (int64, uint64, fixed64, sfixed64, sint64) are LuaJIT int64_t / uint64_t cdata — lossless and the same convention used by Tarantool's net.box, msgpack, and built-in protobuf modules.

Three API modes live side-by-side. Same descriptor, three call shapes — the inline (full) generated API is the default, the descriptor-driven runtime API is for dynamic schemas, and the lazy API is a zero-copy view for sparse reads and proxy / router workloads. See docs/api-modes.md for when to pick which, with measured trade-offs.

#Layout

cmd/
  protoc-gen-tarantool/      Go plugin (the codegen)
    main.go                  reads CodeGeneratorRequest, hands off to gen
    internal/gen/            per-message emission for both modes
  protoc-gen-tarantool-doc/  separate Go plugin that emits Markdown docs
  conformance/               Lua conformance dispatch (loaded by the runner)
  conformance-runner.lua     stdin/stdout framing for `conformance_test_runner`

runtime/pb/                  pure-Lua runtime (`require('pb')`)
  init.lua                   public surface
  wire.lua                   varint / zigzag / fixed / float / LEN primitives
  codec.lua                  descriptor-driven encode/decode
  lazy.lua                   zero-copy MessageView / ArrayView / MapView
  text.lua                   text format encode + decode (proto3)
  json.lua                   proto3 JSON encode + decode (strict)
  wkt.lua                    Timestamp / Duration / Empty / Wrappers /
                             Struct / Value / ListValue / Any / FieldMask
  grpc.lua                   transport interface + loopback / multiplex
  parser.lua                 pure-Lua proto3 schema parser (.proto → AST)
  dynamic.lua                AST → descriptor module
  fileset.lua                FileDescriptorSet bytes → descriptor module
  descriptor_pb.lua          hand-built descriptors of descriptor.proto

options/tarantool/           custom proto file options
  tarantool.proto            (tarantool.lua_package) — Lua require path override

examples/proto/              demo .proto inputs
examples/expected/           generated output for full + runtime modes
                             (both committed for inspection + parametrized tests)
test/                        luatest groups, conformance regressions, fixtures
docs/                        codegen notes, API mode comparison, design briefs
bench/                       per-helper bench + JIT-trace gate + alloc baseline

#Documentation

• docs/api-modes.md — when to use Foo_encode (full), pb.encode(desc, t) (runtime / reflect), or pb.decode_lazy(desc, b) (lazy view). Measured allocation + throughput trade-offs.
• docs/codegen.md — plugin internals, descriptor shape contract, how to add a new wire type or scalar, the LuaJIT hot-path rules the generated code observes.
• docs/text_format_parser_brief.md — retrospective brief on the text-format parser slice (pb.text.decode).
• PLAN.md — phased roadmap and per-feature design notes.
• CLAUDE.md — invariants and conventions enforced across the codebase (no pairs() on hot paths, 64-bit ints as cdata, SoA over AoS for large index structures, etc.).

#Conformance

cmd/conformance-runner.lua speaks the Google protobuf conformance protocol on stdin/stdout. Drive it with the canonical conformance_test_runner binary like so:

make gen
conformance_test_runner --enforce_recommended \
    tarantool cmd/conformance-runner.lua

Homebrew's protobuf package does not ship conformance_test_runner, so a Dockerfile under docker/conformance.Dockerfile builds it from upstream protobuf source and bundles Tarantool. Run the full suite locally with:

just conformance

(Mounts the repo into the container — generated Lua from make gen on the host is what gets tested.) Known failures live in test/conformance/known_failures.txt (binary + JSON suite) and test/conformance/known_failures_text.txt (text-format suite); both are empty for the proto3 suites as of 2026-05-16.

Current baseline (2026-05-16, protobuf v34.1):

The 1313 + 18 skipped tests all target protobuf_test_messages.proto2.TestAllTypesProto2, which we don't generate Lua for. Proto2 support is a separate slice — see docs/codegen.md#proto2-deferral for what it would take.

The runner supports protobuf_test_messages.proto3.TestAllTypesProto3 in binary, JSON, and text-format input/output, including the JSON_IGNORE_UNKNOWN_PARSING_TEST category (forwarded as ignore_unknown_fields=true to pb.json.decode). The self-test in test/conformance_test.lua exercises the runner with crafted requests on every make test run.

#Benchmarks

make bench           # print throughput + alloc per op (5 sizes × 2 modes)
make bench-baseline  # overwrite bench/baseline.json (run on a quiet machine)
make bench-compare   # exit 1 if any alloc-per-op regressed >5% vs baseline

The committed bench/baseline.json tracks only allocation per op — that's reproducible across machines because it counts bytes, not time. Throughput in stderr is informational; it swings 30%+ on a contended CPU.

Current baseline (LuaJIT 2.1, hello.Person):

Lazy decode trades a much higher index-build cost (one MessageView table + four SoA arrays) for near-zero allocation on subsequent field reads — best when you touch a small fraction of fields, or when you re-encode mostly-unchanged messages (proxy / router workloads). See docs/api-modes.md for the bench numbers and the cross-over point.

#Why named pb instead of protobuf?

Tarantool's loader prefers the built-in require('protobuf') over any filesystem module of the same name. Trying to override it would break code that uses the built-in's encode API. pb is short, unambiguous, and lives alongside the built-in.

#License

BSD 2-Clause. See LICENSE.