#tarantool-protobuf

A protoc plugin and pure-Lua runtime for using Protocol Buffers (proto3) and gRPC service stubs from Tarantool.

Tarantool ships an in-tree require('protobuf') module, but it is encode-only and has no support for map, oneof, services, or any decode path. This project fills those gaps with:

• protoc-gen-tarantool — a protoc plugin (Go) that turns .proto files into Lua modules.
• runtime/pb — a pure Lua + LuaJIT-FFI runtime the generated code uses for the wire format. Named pb rather than protobuf to avoid colliding with Tarantool's built-in module.

#Status

MVP — proto3 messages and enums, end-to-end round-trip verified.

#Quick start

# 1. Build the plugin and generate the example.
make gen

# 2. Run the round-trip test in Tarantool.
make test

The plugin emits one .lua file per .proto. By default the output path mirrors the proto package (package foo.bar; baz.proto → foo/bar/baz_pb.lua), required as foo.bar.baz_pb. Override the Lua module path with a file option:

import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";

#Generated API

For each message Foo the plugin emits:

local M = require('myapp.proto.foo')

M.Foo_descriptor   -- the descriptor table consumed by the runtime
M.Foo_new(t)       -- returns t (or {}); placeholder for future validation
M.Foo_encode(t)    -- table -> wire bytes (string)
M.Foo_decode(b)    -- wire bytes (string) -> table

For each enum Color:

M.Color_descriptor          -- { name, by_name, by_value }
M.Color                     -- alias for by_name: M.Color.RED -> 0

Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers (int64, uint64, fixed64, sfixed64, sint64) are LuaJIT int64_t / uint64_t cdata — lossless and the same convention used by Tarantool's net.box, msgpack, and built-in protobuf modules.

#Layout

cmd/protoc-gen-tarantool/   # Go plugin
  main.go                   # reads CodeGeneratorRequest, hands off to gen
  internal/gen/             # codegen package
runtime/pb/                 # Lua runtime (require('pb'))
  init.lua                  # public surface
  wire.lua                  # varint / zigzag / fixed / float / LEN
  codec.lua                 # descriptor-driven encode/decode
options/tarantool/          # custom proto options
  tarantool.proto           # (tarantool.lua_package) file option
examples/proto/             # demo .proto inputs
examples/expected/          # generated output (committed for inspection)
test/                       # roundtrip test

#Conformance

cmd/conformance-runner.lua speaks the Google protobuf conformance protocol on stdin/stdout. Drive it with the canonical conformance_test_runner binary like so:

make gen
conformance_test_runner --enforce_recommended \
    tarantool cmd/conformance-runner.lua

The runner currently supports protobuf_test_messages.proto3.TestAllTypesProto3 in both protobuf and JSON formats; proto2 / editions / JSPB / text-format cases return skipped. The self-test in test/conformance_test.lua exercises the runner with crafted requests on every make test run.

#Benchmarks

make bench           # print throughput + alloc per op (5 sizes × 2 modes)
make bench-baseline  # overwrite bench/baseline.json (run on a quiet machine)
make bench-compare   # exit 1 if any alloc-per-op regressed >5% vs baseline

The committed bench/baseline.json tracks only allocation per op — that's reproducible across machines because it counts bytes, not time. Throughput in stderr is informational; it swings 30%+ on a contended CPU.

Current baseline (LuaJIT 2.1, hello.Person):

#Why named pb instead of protobuf?

Tarantool's loader prefers the built-in require('protobuf') over any filesystem module of the same name. Trying to override it would break code that uses the built-in's encode API. pb is short, unambiguous, and lives alongside the built-in.

#License

TBD.