# tarantool-protobuf A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto3) and gRPC service stubs from [Tarantool](https://www.tarantool.io/). Tarantool ships an in-tree `require('protobuf')` module, but it is **encode-only** and has no support for `map`, `oneof`, services, or any decode path. This project fills those gaps with: - **`protoc-gen-tarantool`** — a protoc plugin (Go) that turns `.proto` files into Lua modules. - **`runtime/pb`** — a pure Lua + LuaJIT-FFI runtime the generated code uses for the wire format. Named `pb` rather than `protobuf` to avoid colliding with Tarantool's built-in module. ## Status Proto3 conformance is closed: every `Required.*` and `Recommended.*` test in both the binary+JSON and text-format suites passes. | Feature | State | |----------------------------------|--------------| | proto3 scalars (all 15 types) | ✅ | | Repeated, packed by default | ✅ | | Nested messages, self-reference | ✅ | | Cross-file imports | ✅ | | Enums (open semantics) | ✅ | | 64-bit integers as LuaJIT cdata | ✅ | | Two codegen modes (full + runtime) | ✅ | | Zero-copy lazy decode views | ✅ | | `map` (scalar/message values) | ✅ | | `oneof` | ✅ | | proto3 explicit `optional` + `has_*`/`clear_*` | ✅ | | `-0.0` preserved for float/double | ✅ | | gRPC service stubs (unary) | ✅ | | gRPC streaming (server / client / bidi) | ✅ | | Loopback / multiplex transport | ✅ | | WKT: Timestamp ↔ `datetime` | ✅ | | WKT: Duration, Empty, wrappers | ✅ | | WKT: Struct, Value, ListValue | ✅ | | WKT: Any (opaque + registry pack/unpack) | ✅ | | WKT: FieldMask (strict round-trip) | ✅ | | Byte-for-byte interop with `protoc` (18 fixtures) | ✅ | | **Google conformance suite — proto3 binary+JSON** | **1493 ✓ / 0 failures** | | **Google conformance suite — proto3 text format** | **416 ✓ / 0 failures** | | Runtime `.proto` parsing (`pb.parse`) | ✅ | | Runtime `FileDescriptorSet` ingest (`pb.from_pb`) | ✅ | | Markdown doc generator (`protoc-gen-tarantool-doc`) | ✅ | | proto3 JSON (`pb.json.encode`/`.decode`) | ✅ | | Text format (`pb.text.encode` / `pb.text.decode`) | ✅ | | Unknown-field passthrough (`_unknown_fields`) | ✅ | | Microbenchmark + alloc regression gate (`make bench`) | ✅ | | proto2 / editions | ❌ deferred (separate slice; see [docs/codegen.md](docs/codegen.md)) | ## Quick start ```bash # 1. Build the plugin and generate the example. make gen # 2. Run the round-trip test in Tarantool. make test ``` The plugin emits one `.lua` file per `.proto`. By default the output path mirrors the proto package (`package foo.bar; baz.proto` → `foo/bar/baz_pb.lua`), required as `foo.bar.baz_pb`. Override the Lua module path with a file option: ```proto import "tarantool/tarantool.proto"; option (tarantool.lua_package) = "myapp.proto.foo"; ``` ## Generated API For each message `Foo` the plugin emits: ```lua local M = require('myapp.proto.foo') M.Foo_descriptor -- the descriptor table consumed by the runtime M.Foo_new(t) -- returns t (or {}); placeholder for future validation M.Foo_encode(t) -- table -> wire bytes (string) M.Foo_decode(b) -- wire bytes (string) -> table M.Foo_decode_lazy(b) -- wire bytes -> MessageView (zero-copy view) M.Foo_text(t, opts) -- table -> protoc-style text format (debug printer) M.Foo_has_(t) -- only emitted for proto3 explicit-`optional` fields M.Foo_clear_(t) -- same ``` Text-format **decoding** is exposed on the runtime as `pb.text.decode(desc, text, opts)` (no per-message wrapper — it's used from a few places, like the conformance runner, and didn't warrant codegen surface). For each enum `Color`: ```lua M.Color_descriptor -- { name, by_name, by_value } M.Color -- alias for by_name: M.Color.RED -> 0 ``` Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers (`int64`, `uint64`, `fixed64`, `sfixed64`, `sint64`) are LuaJIT `int64_t` / `uint64_t` cdata — lossless and the same convention used by Tarantool's `net.box`, `msgpack`, and built-in `protobuf` modules. **Three API modes live side-by-side.** Same descriptor, three call shapes — the inline (full) generated API is the default, the descriptor-driven runtime API is for dynamic schemas, and the lazy API is a zero-copy view for sparse reads and proxy / router workloads. See [docs/api-modes.md](docs/api-modes.md) for when to pick which, with measured trade-offs. ## Layout ``` cmd/ protoc-gen-tarantool/ Go plugin (the codegen) main.go reads CodeGeneratorRequest, hands off to gen internal/gen/ per-message emission for both modes protoc-gen-tarantool-doc/ separate Go plugin that emits Markdown docs conformance/ Lua conformance dispatch (loaded by the runner) conformance-runner.lua stdin/stdout framing for `conformance_test_runner` runtime/pb/ pure-Lua runtime (`require('pb')`) init.lua public surface wire.lua varint / zigzag / fixed / float / LEN primitives codec.lua descriptor-driven encode/decode lazy.lua zero-copy MessageView / ArrayView / MapView text.lua text format encode + decode (proto3) json.lua proto3 JSON encode + decode (strict) wkt.lua Timestamp / Duration / Empty / Wrappers / Struct / Value / ListValue / Any / FieldMask grpc.lua transport interface + loopback / multiplex parser.lua pure-Lua proto3 schema parser (.proto → AST) dynamic.lua AST → descriptor module fileset.lua FileDescriptorSet bytes → descriptor module descriptor_pb.lua hand-built descriptors of descriptor.proto options/tarantool/ custom proto file options tarantool.proto (tarantool.lua_package) — Lua require path override examples/proto/ demo .proto inputs examples/expected/ generated output for full + runtime modes (both committed for inspection + parametrized tests) test/ luatest groups, conformance regressions, fixtures docs/ codegen notes, API mode comparison, design briefs bench/ per-helper bench + JIT-trace gate + alloc baseline ``` ## Documentation - **[docs/api-modes.md](docs/api-modes.md)** — when to use `Foo_encode` (full), `pb.encode(desc, t)` (runtime / reflect), or `pb.decode_lazy(desc, b)` (lazy view). Measured allocation + throughput trade-offs. - **[docs/codegen.md](docs/codegen.md)** — plugin internals, descriptor shape contract, how to add a new wire type or scalar, the LuaJIT hot-path rules the generated code observes. - **[docs/text_format_parser_brief.md](docs/text_format_parser_brief.md)** — retrospective brief on the text-format parser slice (`pb.text.decode`). - **PLAN.md** — phased roadmap and per-feature design notes. - **CLAUDE.md** — invariants and conventions enforced across the codebase (no `pairs()` on hot paths, 64-bit ints as cdata, SoA over AoS for large index structures, etc.). ## Conformance `cmd/conformance-runner.lua` speaks the [Google protobuf conformance protocol][gconf] on stdin/stdout. Drive it with the canonical `conformance_test_runner` binary like so: ```bash make gen conformance_test_runner --enforce_recommended \ tarantool cmd/conformance-runner.lua ``` Homebrew's `protobuf` package does not ship `conformance_test_runner`, so a Dockerfile under `docker/conformance.Dockerfile` builds it from upstream protobuf source and bundles Tarantool. Run the full suite locally with: ```bash just conformance ``` (Mounts the repo into the container — generated Lua from `make gen` on the host is what gets tested.) Known failures live in `test/conformance/known_failures.txt` (binary + JSON suite) and `test/conformance/known_failures_text.txt` (text-format suite); both are empty for the proto3 suites as of 2026-05-16. Current baseline (2026-05-16, protobuf v34.1): | Suite | Successes | Skipped | Expected failures | Unexpected | |-------|-----------|---------|-------------------|------------| | Binary + JSON | **1493** | 1313 | 0 | 0 | | Text-format | **416** | 18 | 0 | 0 | The 1313 + 18 skipped tests all target `protobuf_test_messages.proto2.TestAllTypesProto2`, which we don't generate Lua for. Proto2 support is a separate slice — see [docs/codegen.md#proto2-deferral](docs/codegen.md) for what it would take. The runner supports `protobuf_test_messages.proto3.TestAllTypesProto3` in binary, JSON, and text-format input/output, including the `JSON_IGNORE_UNKNOWN_PARSING_TEST` category (forwarded as `ignore_unknown_fields=true` to `pb.json.decode`). The self-test in `test/conformance_test.lua` exercises the runner with crafted requests on every `make test` run. [gconf]: https://github.com/protocolbuffers/protobuf/tree/main/conformance ## Benchmarks ```bash make bench # print throughput + alloc per op (5 sizes × 2 modes) make bench-baseline # overwrite bench/baseline.json (run on a quiet machine) make bench-compare # exit 1 if any alloc-per-op regressed >5% vs baseline ``` The committed `bench/baseline.json` tracks only allocation per op — that's reproducible across machines because it counts bytes, not time. Throughput in stderr is informational; it swings 30%+ on a contended CPU. Current baseline (LuaJIT 2.1, hello.Person): | Payload | encode alloc (full / runtime) | decode alloc (full / runtime) | |---------|-------------------------------|-------------------------------| | 10 B | 0.37 / 0.63 KB | 0.51 / 0.51 KB | | 100 B | 0.37 / 0.63 KB | 0.51 / 0.51 KB | | 1 KB | 5.98 / 7.04 KB | 7.75 / 7.88 KB | | 10 KB | 47.5 / 48.6 KB | 59.5 / 59.6 KB | | 100 KB | 444 / 445 KB | 568 / 568 KB | Lazy decode trades a much higher *index-build* cost (one MessageView table + four SoA arrays) for near-zero allocation on subsequent field reads — best when you touch a small fraction of fields, or when you re-encode mostly-unchanged messages (proxy / router workloads). See [docs/api-modes.md](docs/api-modes.md) for the bench numbers and the cross-over point. ## Why named `pb` instead of `protobuf`? Tarantool's loader prefers the built-in `require('protobuf')` over any filesystem module of the same name. Trying to override it would break code that uses the built-in's encode API. `pb` is short, unambiguous, and lives alongside the built-in. ## License BSD 2-Clause. See [LICENSE](LICENSE).