#tarantool-protobuf

A protoc plugin and pure-Lua runtime for using Protocol Buffers (proto3) and gRPC service stubs from Tarantool.

Tarantool ships an in-tree require('protobuf') module, but it is encode-only and has no support for map, oneof, services, or any decode path. This project fills those gaps with:

• protoc-gen-tarantool — a protoc plugin (Go) that turns .proto files into Lua modules.
• runtime/pb — a pure Lua + LuaJIT-FFI runtime the generated code uses for the wire format. Named pb rather than protobuf to avoid colliding with Tarantool's built-in module.

#Status

Proto3 conformance is closed: every Required.* and Recommended.* test in both the binary+JSON and text-format suites passes.

#Install

Until the repository is published on its canonical remote, the source.url in tarantool-protobuf-scm-1.rockspec is aspirational and tt rocks install <url> will fail. Install from a local checkout instead:

git clone <this-repo> && cd tarantool-protobuf
tt rocks make tarantool-protobuf-scm-1.rockspec

That puts the pb.* runtime modules under .rocks/share/tarantool/. The Go plugin still has to be built separately — see "Quick start" below.

#Quick start

# 1. Build the plugin and generate the example.
just gen

# 2. Run the round-trip test in Tarantool.
just test

The plugin emits one .lua file per .proto. By default the output path mirrors the proto package (package foo.bar; baz.proto → foo/bar/baz_pb.lua), required as foo.bar.baz_pb. Two ways to override:

// 1. Per-file, via a proto option:
import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";

# 2. Plugin-wide, via the `prefix=` plugin parameter. Every generated
#    module is prepended with this namespace and cross-file imports rewrite
#    to match. Equivalent to applying `option (tarantool.lua_package)` to
#    every input file, but without touching the .proto.
protoc --tarantool_out=out \
       --tarantool_opt=prefix=myapp.proto \
       file.proto
# -> out/myapp/proto/<pkg>/<file>_pb.lua, required as
#    "myapp.proto.<pkg>.<file>_pb"

prefix= and (tarantool.lua_package) compose: when both are set, the prefix is prepended to the option's value. See docs/reference/cli.md for the full mapping.

For a full walk-through that takes a fresh .proto to a Tarantool process encoding and decoding it, see docs/howto/01-first-message.md.

#Generated API

For each message Foo the plugin emits:

local M = require('myapp.proto.foo')

M.Foo_descriptor       -- the descriptor table consumed by the runtime
M.Foo_new(t)           -- returns t (or {}); placeholder for future validation
M.Foo_encode(t)        -- table -> wire bytes (string)
M.Foo_decode(b)        -- wire bytes (string) -> table
M.Foo_decode_lazy(b)   -- wire bytes -> MessageView (zero-copy view)
M.Foo_text(t, opts)    -- table -> protoc-style text format (debug printer)
M.Foo_has_<field>(t)   -- only emitted for proto3 explicit-`optional` fields
M.Foo_clear_<field>(t) -- same
M.Foo_fields           -- strict {field = "field", ...} for lazy-view callers
M.Foo_oneofs           -- strict {oneof_group = "oneof_group", ...}, when any

Lazy-view field-name arguments should be routed through M.Foo_fields / M.Foo_oneofs rather than passed as string literals — typos error at the read site instead of silently returning nil. See docs/api-modes.md.

Text-format decoding is exposed on the runtime as pb.text.decode(desc, text, opts) (no per-message wrapper — it's used from a few places, like the conformance runner, and didn't warrant codegen surface).

For each enum Color:

M.Color_descriptor          -- { name, by_name, by_value }
M.Color                     -- alias for by_name: M.Color.RED -> 0

Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers (int64, uint64, fixed64, sfixed64, sint64) are LuaJIT int64_t / uint64_t cdata — lossless and the same convention used by Tarantool's net.box, msgpack, and built-in protobuf modules.

#Migrating from a Lua proto library that auto-down-casts int64

If you're moving from a library that hands back Lua numbers (silently losing precision past 2^53), expect a sweep wherever a cdata value crosses into a primitive that doesn't accept it. The four patterns that catch out every migrator:

local id = msg.user_id  -- cdata: uint64_t

-- 1. log / printf format verbs: %d on cdata raises an error.
log.info('user %d signed in', tonumber(id))

-- 2. numeric for-loop bounds: `for i = 1, n` requires a Lua number.
for i = 1, tonumber(msg.row_count) do ... end

-- 3. string.format with %d / %x: same as log.
local hex = string.format('%016x', tonumber(id))

-- 4. table keys: cdata is hashed by identity, not value, so two
--    distinct cdata for the same number won't collide. Either convert
--    to number (if it fits) or use tostring(id) as the key.
cache[tonumber(id)] = row

box.tuple / net.box / msgpack / Tarantool's protobuf all accept cdata int64 directly — those paths don't need a tonumber(). The boundary is Lua primitives that expect a number. Past 2^53 (≈ 9e15), tonumber() silently truncates; if your IDs can be that large, keep them as cdata or stringify with tostring(id):gsub('U?LL$', '').

Three API modes live side-by-side. Same descriptor, three call shapes — the inline (full) generated API is the default, the descriptor-driven runtime API is for dynamic schemas, and the lazy API is a zero-copy view for sparse reads and proxy / router workloads. See docs/api-modes.md for when to pick which, with measured trade-offs.

#Layout

cmd/
  protoc-gen-tarantool/      Go plugin (the codegen)
    main.go                  reads CodeGeneratorRequest, hands off to gen
    internal/gen/            per-message emission for both modes
  protoc-gen-tarantool-doc/  separate Go plugin that emits Markdown docs
  conformance/               Lua conformance dispatch (loaded by the runner)
  conformance-runner.lua     stdin/stdout framing for `conformance_test_runner`

runtime/pb/                  pure-Lua runtime (`require('pb')`)
  init.lua                   public surface
  wire.lua                   varint / zigzag / fixed / float / LEN primitives
  codec.lua                  descriptor-driven encode/decode
  lazy.lua                   zero-copy MessageView / ArrayView / MapView
  text.lua                   text format encode + decode (proto3)
  json.lua                   proto3 JSON encode + decode (strict)
  wkt.lua                    Timestamp / Duration / Empty / Wrappers /
                             Struct / Value / ListValue / Any / FieldMask
  grpc.lua                   transport interface + loopback / multiplex
  parser.lua                 pure-Lua proto3 schema parser (.proto → AST)
  dynamic.lua                AST → descriptor module
  fileset.lua                FileDescriptorSet bytes → descriptor module
  descriptor_pb.lua          hand-built descriptors of descriptor.proto

options/tarantool/           custom proto file options
  tarantool.proto            (tarantool.lua_package) — Lua require path override

examples/proto/              demo .proto inputs
examples/expected/           generated output for full + runtime modes
                             (both committed for inspection + parametrized tests)
test/                        luatest groups, conformance regressions, fixtures
docs/                        codegen notes, API mode comparison, design briefs
bench/                       per-helper bench + JIT-trace gate + alloc baseline

#Documentation

Start at docs/index.md — the documentation map, grouped by what you're trying to do (getting started, reference, specs, internals). The two reference pages worth knowing by name:

• docs/api-modes.md — when to use Foo_encode (full), pb.encode(desc, t) (runtime / reflect), or pb.decode_lazy(desc, b) (lazy view). Measured allocation + throughput trade-offs.
• docs/codegen.md — plugin internals, descriptor shape contract, how to add a new wire type or scalar, the LuaJIT hot-path rules generated code observes.

Roadmap is in PLAN.md; cross-codebase invariants in CLAUDE.md.

#Conformance

cmd/conformance-runner.lua speaks the Google protobuf conformance protocol on stdin/stdout. Drive it with the canonical conformance_test_runner binary like so:

just gen
conformance_test_runner --enforce_recommended \
    tarantool cmd/conformance-runner.lua

Homebrew's protobuf package does not ship conformance_test_runner, so a Dockerfile under docker/conformance.Dockerfile builds it from upstream protobuf source and bundles Tarantool. Run the full suite locally with:

just conformance

(Mounts the repo into the container — generated Lua from just gen on the host is what gets tested.) Known failures live in test/conformance/known_failures.txt (binary + JSON suite) and test/conformance/known_failures_text.txt (text-format suite); both are empty for the proto3 suites as of 2026-05-16.

Current baseline (2026-05-16, protobuf v34.1):

The 1313 + 18 skipped tests all target protobuf_test_messages.proto2.TestAllTypesProto2, which we don't generate Lua for. Proto2 support is a separate slice — see docs/codegen.md#proto2-deferral for what it would take.

The runner supports protobuf_test_messages.proto3.TestAllTypesProto3 in binary, JSON, and text-format input/output, including the JSON_IGNORE_UNKNOWN_PARSING_TEST category (forwarded as ignore_unknown_fields=true to pb.json.decode). The self-test in test/conformance_test.lua exercises the runner with crafted requests on every just test run.

#Benchmarks

just bench           # print throughput + alloc per op (5 sizes × 2 modes)
just bench-baseline  # overwrite bench/baseline.json (run on a quiet machine)
just bench-compare   # exit 1 if any alloc-per-op regressed >5% vs baseline

The committed bench/baseline.json tracks only allocation per op — that's reproducible across machines because it counts bytes, not time. Throughput in stderr is informational; it swings 30%+ on a contended CPU.

Current baseline (LuaJIT 2.1, hello.Person):

Lazy decode trades a much higher index-build cost (one MessageView table + four SoA arrays) for near-zero allocation on subsequent field reads — best when you touch a small fraction of fields, or when you re-encode mostly-unchanged messages (proxy / router workloads). See docs/api-modes.md for the bench numbers and the cross-over point.

#Why named pb instead of protobuf?

Tarantool's loader prefers the built-in require('protobuf') over any filesystem module of the same name. Trying to override it would break code that uses the built-in's encode API. pb is short, unambiguous, and lives alongside the built-in.

#License

BSD 2-Clause. See LICENSE.