This document tracks the long-form design + testing strategy for
protoc-gen-tarantool and the pb Lua runtime. Short-term task tracking
lives in the in-session task list; this file is the slow-changing record.
A first-class Protocol Buffers + gRPC stack for Tarantool that:
.proto files with two modes (full inline
vs descriptor-driven runtime) so users can pick speed-vs-flexibility.net.box, fiber, box.session, and the 3.x config framework.See README.md for the user-visible summary. Internally:
This is the floor we're building on.
Each milestone ends with a green CI run, an updated README, and a tagged release on sourcecraft.dev.
pb.wire.
wire.encode_int32(v), wire.decode_string(buf, pos), etc., for all
15 scalar proto types. Both modes consume the same primitives._encode / _decode
functions with no descriptor lookup. Tag bytes precomputed at gen
time as Lua string literals. This is the JIT-friendly hot path.mode=full|runtime (default: full).examples/expected/{full,runtime}/
for visual diffing.Done when: same .proto produces two modules; identical wire bytes; one
luatest run covers both.
map<K,V>: emit as repeated synthetic *Entry messages with
key/value fields per spec. Decode merges into a Lua table; encode
iterates with pairs. Key types limited to scalars + string per spec.oneof: descriptor includes oneof_index per field. Encode
emits at most one branch (last assignment wins). Decode clears prior
oneof siblings on assignment.optional: respect presence — emit field even
when value equals scalar default. Generated descriptor exposes
has_<name>(t) / clear_<name>(t) helpers.Done when: a "composite" example proto with map<string,int32>, a oneof with 3 cases, and an explicit-optional bool round-trips through both modes with parity-against-protoc.
google.protobuf.Timestamp ↔ Tarantool datetime module
(epoch + nsec mapping).google.protobuf.Duration ↔ interval (or seconds+nanos table).Empty, FieldMask. FieldMask is a repeated string; JSON mapping
is the canonical comma-joined lowerCamelCase form.Any: opaque {type_url, value} form by default; pb.register(desc)
+ pb.any.pack(desc, t) / pb.any.unpack(any_t) for typed
round-trips. JSON canonical mapping emits the flat {"@type": ...}
object when the type is registered, falls back to base64 opaque form
otherwise.Struct, Value, ListValue ↔ idiomatic Lua tables.
box.NULL is the null_value sentinel (re-exported as pb.NULL).
pb.wkt.struct(t) / pb.wkt.list(t) tag tables when the auto-detect
heuristic (t[1] ~= nil → list, else struct) needs to be overridden,
and decode preserves the tag for byte-stable round trips.Done when: WKT-using protos round-trip and integrate visibly with
datetime (e.g. os.date(...)-comparable timestamps).
/pkg.Svc/Method,
input/output type refs, client_streaming / server_streaming flags.MyService_client(transport) returns a table with
one function per method. Transport interface, all served by the
shipped loopback + multiplex implementations:
lua transport:unary(path, req_bytes, ctx) -> resp_bytes transport:server_stream(path, req_bytes, ctx) -> stream transport:client_stream(path, ctx) -> stream transport:bidi(path, ctx) -> stream
Streaming returns a stream object with send / close_send /
recv() -> msg, err / cancel. End-of-stream is (nil, nil);
handler errors surface as (nil, err_string).MyService_server(impl) returns
{service, methods, streams} consumable by the loopback /
multiplex transports. Handler signatures by RPC kind:
lua unary: function(req, ctx) -> resp server_stream: function(req, stream, ctx) client_stream: function(stream, ctx) -> resp bidi: function(stream, ctx) pb.grpc.transport.netbox — gRPC tunneled over net.box calls.Done. The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via fiber.channel. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.
cmd/conformance-runner.lua is the testee: reads length-prefixed
ConformanceRequest on stdin, runs it through our codec / JSON,
writes a length-prefixed ConformanceResponse on stdout. Loops to
EOF. Core dispatch is factored into cmd/conformance/core.lua and
exercised directly by test/conformance_test.lua (10 dispatch
cases + 3 stdin/stdout framing cases).conformance_test_runner binary in CI and track
the pass-rate as a numeric metric (regression gate).test/interop/fixtures/
produced by mainline protoc --encode; tests assert byte-for-byte
equality.bench/bench.lua, run via make bench. Throughput is stderr-only
(varies with CPU load); JSON document on stdout.collectgarbage('stop') framing. Committed as
bench/baseline.json. Regression gate: make bench-compare exits
non-zero if alloc/op grows >5% vs baseline. Allocations are
deterministic to ~10 bytes regardless of hardware.make jit-trace (bench/jit_trace.lua)
attaches a jit.attach('trace') listener over the hot
encode/decode paths and asserts no aborts in our source files
fall into the fatal set (NYI bytecode, blacklisting, persistent
type instability). Pass: 13/13 scenarios. Two fixes shipped:
decode_varint grew a 1-byte fast path so callers no longer drag
an inner loop into the root trace; pb.finalize_message now
precomputes desc.oneofs_list so runtime-mode oneof encoding
uses ipairs instead of pairs (the latter compiles to bytecode
ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
interpreter-bridge counts as a benchmark-quality metric (decoders
have 0–4 per run depending on JIT timing — caused by side traces
returning from inlined decode_varint calls, which LuaJIT 2.1
can't stitch back cleanly; small per-call overhead, structural
to the engine). Scope caveat: map fields still encode via
pairs() and remain off-trace — pinned by the gate's last
scenario so we notice if upstream lifts the restriction.ffi.cdata byte buffer instead of building a string list. Targets
hot RPC paths where allocation cost dominates.msgpack.object-like lazy view
for nested messages; only materializes touched fields.t:Person_encode({name=...}) autocompletes in editors.pb.from_pb(file_descriptor_set) — runtime descriptor parser, lets
apps load schemas at runtime without protoc-time codegen.MyMsg.print(t)).protoc-gen-tarantool-doc: generates Markdown reference docs
from .proto files.require('protobuf') to
require('pb').For each message we emit one _encode and one _decode function. Tag
bytes are precomputed string literals; field-presence checks are inlined.
The runtime is reduced to wire-format primitives.
function M.Address_encode(t)
local out, n = {}, 0
local v
v = t.street
if v ~= nil and v ~= '' then
n = n + 1; out[n] = '\x0a' -- tag(1, LEN)
n = n + 1; out[n] = wire.encode_string(v)
end
-- ...
return table.concat(out)
end
Decoder uses an if-elseif chain on field id (LuaJIT compiles this
well for small chain lengths; for >16 fields we may want a numeric jump
table or tag-byte switching).
map<K,V> is wire-format-equivalent to:
message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;
Codegen synthesizes the entry message internally but exposes the field as a
Lua table (hash, not array). Encode iterates with pairs; decode merges
on duplicate keys (last wins, per spec).
Descriptor gains oneofs = {[oneof_name] = {field_names...}}. Encode walks
fields in declaration order and emits the first non-nil branch. Decode
clears sibling fields on assignment so callers see exactly one set.
In inline mode, the encode walk becomes an explicit if-chain; the decode
clears are emitted alongside each elseif id == N then arm.
Locked: int64/uint64/fixed64/sfixed64/sint64 are LuaJIT
int64_t / uint64_t cdata. Reasons:
msgpackffi, net.box, box.tuple.0 (numeric coercion in LuaJIT).We will document a wire.from_string(s) helper for users who get hex/dec
strings (e.g. from JSON) and need to feed them into encode.
t._unknown_fields is a single Lua string holding the verbatim
concatenation of tag+value bytes for fields the decoder didn't recognize.
Captured during decode in encounter order; re-emitted at the tail of
_encode. Mirrors Tarantool's built-in protobuf module convention.
Map entries don't preserve unknowns (per spec — synthetic Entry messages).
WKT types bypass this too, since they have custom desc.encode/decode.
Both codegen modes implement it: runtime mode in pb.codec, full (inline)
mode emits per-message capture/re-emit blocks. Tests live in
test/unknown_test.lua and run against both modes.
M.Greeter_service = {
name = 'hello.Greeter',
methods = {
SayHello = {
full_name = '/hello.Greeter/SayHello',
input = M.HelloRequest_descriptor,
output = M.HelloReply_descriptor,
client_streaming = false,
server_streaming = false,
},
-- ...
},
}
Client: M.Greeter_client(transport) returns
{SayHello = function(req) ... end, ...}.
Server: M.Greeter_server(impl) returns a table compatible with the
server:register(svc) interface.
Tests live in test/. Layout:
test/
unit/ -- pb.wire and pb.codec unit tests (no codegen)
wire_test.lua
codec_test.lua
roundtrip/ -- generated-module round-trip, both modes
scalars_test.lua
repeated_test.lua
nested_test.lua
map_test.lua -- M2
oneof_test.lua -- M2
wkt_test.lua -- M3
conformance/ -- M5: Google conformance harness
interop/ -- M5: cross-impl byte-for-byte equality
fixtures/ -- pre-encoded payloads from Go/Python
bench/ -- M6: microbenchmarks
fuzz/ -- malformed-input + random-input harness
For each wire.encode_<type> / wire.decode_<type> pair:
Every test runs against both generated modules (full and runtime mode) using a parameterized luatest group:
for _, mode in ipairs({'full', 'runtime'}) do
local g = t.group('roundtrip.' .. mode)
local hello = require('hello.' .. mode .. '.hello_pb')
g.test_address = function() ... end
-- ...
end
Coverage targets:
Build a small Lua binary cmd/conformance-runner.lua that speaks the
Google protobuf conformance protocol on stdin/stdout. Checked into the
repo so CI runs it against the canonical test corpus. Track pass-rate as
a CI metric.
For each of N reference protos:
protoc --encode=msg < input.txt > golden.bin (using mainline protoc).pb.decode(msg, read('golden.bin')) produces the
expected Lua table.pb.encode(msg, expected) produces bytes that, when decoded
by mainline protoc, yield the same logical message (allow field
reordering since neither order is canonical).Generate goldens once via a make goldens target; commit them to the
repo. CI just verifies them, never regenerates.
_decode function;
assert no crashes, only controlled error() calls.decode(encode(t)) == t (value-equal under our equality helper).math.random with a seeded PRNG for reproducibility.Microbenchmarks measured:
misc.memprof.Run on a fixed corpus across 5 message sizes; track results in
bench/baseline.json and fail PRs that regress >5%.
tt rocks install luatest to .rocks/.make test → .rocks/bin/luatest -v test/.go test ./... for any pure-Go logic in the plugin.protoc on a fixture, diff generated Lua
against committed expected output.build, gen,
test, bench, lint, goldens, conformance, clean, release.go test -cover for plugin; luacov for runtime.pkgsite for Go; manually maintained Markdown for
Lua until a tool emerges.| Question | Notes |
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add pb.encode_deterministic flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
| Should we ship a stub HTTP/2 transport for gRPC? | Probably no — large dependency. Recommend lua-http or write a focused project. |
How to handle the existing Tarantool-builtin require('protobuf')? |
Document migration; do not override the loader. |
Lua module path mapping when no lua_package and no proto package? |
Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
| Integration with Tarantool 3.x declarative config? | A pb.types.<name> config role would be nice; defer until use case appears. |
Edit this file directly. After any milestone closes:
## Done heading.README.md status table.