This document tracks the long-form design + testing strategy for
protoc-gen-tarantool and the pb Lua runtime. Short-term task tracking
lives in the in-session task list; this file is the slow-changing record.
A first-class Protocol Buffers + gRPC stack for Tarantool that:
.proto files with two modes (full inline
vs descriptor-driven runtime) so users can pick speed-vs-flexibility.net.box, fiber, box.session, and the 3.x config framework.See README.md for the user-visible summary. Internally:
mode=full inline + mode=runtime
descriptor-delegating wrappers). Sibling protoc-gen-tarantool-doc
emits Markdown reference per .proto.wire.lua (~660 LOC) + codec.lua
lazy.lua (zero-copy view) + text.lua (encode + decode) +
json.lua (strict proto3 JSON) + wkt.lua (all 9 well-known types) +
grpc.lua (transport + loopback) + parser.lua / dynamic.lua /
fileset.lua (three runtime descriptor producers — .proto source,
AST, FileDescriptorSet bytes).conformance_test_runner v34.1).bench/baseline.json tracks allocation/op (regression
gate at 5%); make jit-trace pins LuaJIT trace stability.The remaining unfinished bullets in section 3 are mostly M8 (release engineering) and a handful of optional perf items in M6.
Each milestone ends with a green CI run, an updated README, and a tagged release on sourcecraft.dev.
pb.wire.
wire.encode_int32(v), wire.decode_string(buf, pos), etc., for all
15 scalar proto types. Both modes consume the same primitives._encode / _decode
functions with no descriptor lookup. Tag bytes precomputed at gen
time as Lua string literals. This is the JIT-friendly hot path.mode=full|runtime (default: full).examples/expected/{full,runtime}/
for visual diffing.map<K,V>: emit as repeated synthetic *Entry messages with
key/value fields per spec. Decode merges into a Lua table; encode
iterates with pairs. Key types limited to scalars + string per spec.oneof: descriptor includes oneof_index per field. Encode
emits at most one branch (last assignment wins). Decode clears prior
oneof siblings on assignment. Hot-path lookup goes through
desc.oneofs_list (flat array) to keep the trace JIT-stable.optional: respect presence — emit field even
when value equals scalar default. Generated descriptor exposes
has_<name>(t) / clear_<name>(t) helpers.google.protobuf.Timestamp ↔ Tarantool datetime module
(epoch + nsec mapping). Out-of-spec inputs (negative nanos,
year > 9999) keep the raw {seconds, nanos} table so JSON
serialization can reject them with serialize_error instead of
crashing on decode.google.protobuf.Duration ↔ {seconds, nanos} table (interval
proved a poor fit — it carries months/days that don't map
cleanly).repeated string. Strict round-trip validation —
snake_case paths must use only [a-z0-9_], no leading/trailing
_, no __, and _ must precede a lowercase letter (not a
digit). JSON form rejects any _ (must be lowerCamelCase).Empty.Any: opaque {type_url, value} form by default; pb.register(desc)
+ pb.any.pack(desc, t) / pb.any.unpack(any_t) for typed
round-trips. JSON canonical mapping emits the flat {"@type": ...}
object when the type is registered, falls back to base64 opaque form
otherwise.Struct, Value, ListValue ↔ idiomatic Lua tables.
box.NULL is the null_value sentinel (re-exported as pb.NULL).
pb.wkt.struct(t) / pb.wkt.list(t) tag tables when the auto-detect
heuristic (t[1] ~= nil → list, else struct) needs to be overridden,
and decode preserves the tag for byte-stable round trips.Done when: WKT-using protos round-trip and integrate visibly with
datetime (e.g. os.date(...)-comparable timestamps).
/pkg.Svc/Method,
input/output type refs, client_streaming / server_streaming flags.MyService_client(transport) returns a table with
one function per method. Transport interface, all served by the
shipped loopback + multiplex implementations:
lua transport:unary(path, req_bytes, ctx) -> resp_bytes transport:server_stream(path, req_bytes, ctx) -> stream transport:client_stream(path, ctx) -> stream transport:bidi(path, ctx) -> stream
Streaming returns a stream object with send / close_send /
recv() -> msg, err / cancel. End-of-stream is (nil, nil);
handler errors surface as (nil, err_string).MyService_server(impl) returns
{service, methods, streams} consumable by the loopback /
multiplex transports. Handler signatures by RPC kind:
lua unary: function(req, ctx) -> resp server_stream: function(req, stream, ctx) client_stream: function(stream, ctx) -> resp bidi: function(stream, ctx) pb.grpc.transport.netbox — gRPC tunneled over net.box calls.Done. The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via fiber.channel. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.
[x] Wire up Google's protobuf conformance test runner.
cmd/conformance-runner.lua is the testee: reads length-prefixed
ConformanceRequest on stdin, runs it through our codec / JSON /
text codecs, writes a length-prefixed ConformanceResponse on
stdout. Loops to EOF. Core dispatch lives in
cmd/conformance/core.lua and is exercised directly by
test/conformance_test.lua so the inner dev loop doesn't need
Docker.
[x] Run the canonical conformance_test_runner binary locally and
track the pass-rate as a numeric metric (regression gate).
docker/conformance.Dockerfile builds conformance_test_runner
from upstream protobuf v34.1 source (matching the host's
libprotoc 34.1) and bundles Tarantool 3 from the official deb;
just conformance regenerates Lua then runs the harness against
cmd/conformance-runner.lua with the repo mounted as a volume.
Watchlists at test/conformance/known_failures.txt (binary +
JSON suite) and test/conformance/known_failures_text.txt
(text-format suite); both are empty for the proto3 suites as
of 2026-05-16.
Current baseline:
- Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
- Text-format suite: **416 ✓ / 18 skipped / 0 failures**
The 1313 + 18 skipped all target
`protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
is a separate slice — see [docs/codegen.md](../docs/codegen.md)
for what it would take.
Strict-validation closures landed across three commits on the
`text-conformance-output` branch:
- `pb.text.decode` — full grammar coverage (recursive-descent
parser, ~580 LOC; see
[docs/text_format_parser_brief.md](../docs/text_format_parser_brief.md)).
- `codec` -0.0 preservation — float/double `is_default_scalar`
and the inline-codegen elision both gained a sign-bit guard
(`1/v == math.huge`).
- `pb.json` strict-validation pass — duplicate-key rejection
(literal + camel/snake alias detection via a byte-walking
pre-scan), null-in-container rejection, unknown-enum-name
rejection (with `ignore_unknown_fields` opt for the
`JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
`google.protobuf.NullValue` round-trip as JSON `null`,
strict FieldMask round-trip validation.
The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
short-circuit JSON output if a future encoder bug starts
crashing jsoncpp.
[ ] CI wire-up. The Docker image build is the long pole (~10–15 min on a clean cache); a registry push from a scheduled job would let CI runs reuse a warm cache.
[x] Cross-impl interop: 18-fixture corpus in test/interop/fixtures/
produced by mainline protoc --encode; tests assert byte-for-byte
equality.
[x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for
messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped
as bench/bench.lua, run via make bench. Throughput is stderr-only
(varies with CPU load); JSON document on stdout.
[x] Allocation profiling — bytes per encode/decode op, measured via GC
delta with collectgarbage('stop') framing. Committed as
bench/baseline.json. Regression gate: make bench-compare exits
non-zero if alloc/op grows >5% vs baseline. Allocations are
deterministic to ~10 bytes regardless of hardware.
[x] Trace stability — make jit-trace (bench/jit_trace.lua)
attaches a jit.attach('trace') listener over the hot
encode/decode paths and asserts no aborts in our source files
fall into the fatal set (NYI bytecode, blacklisting, persistent
type instability). Pass: 13/13 scenarios. Two fixes shipped:
decode_varint grew a 1-byte fast path so callers no longer drag
an inner loop into the root trace; pb.finalize_message now
precomputes desc.oneofs_list so runtime-mode oneof encoding
uses ipairs instead of pairs (the latter compiles to bytecode
ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
interpreter-bridge counts as a benchmark-quality metric (decoders
have 0–4 per run depending on JIT timing — caused by side traces
returning from inlined decode_varint calls, which LuaJIT 2.1
can't stitch back cleanly; small per-call overhead, structural
to the engine). Scope caveat: map fields still encode via
pairs() and remain off-trace — pinned by the gate's last
scenario so we notice if upstream lifts the restriction.
[ ] Optional output: ibuf-based encoder that writes into a caller-owned
ffi.cdata byte buffer instead of building a string list. Targets
hot RPC paths where allocation cost dominates.
[x] Decoder fast path that returns an msgpack.object-like lazy view
for nested messages; only materializes touched fields.
Shipped as runtime/pb/lazy.lua + M.<Type>_decode_lazy codegen
stubs in both modes. Surface: :get / :has / :which / :iter / :names on MessageView; :len / :at / :iter / :tolist on
ArrayView; :get / :has / :keys / :iter / :totable on MapView.
Mutation via :set is supported and propagates sub-view edits
transparently (sub-MessageViews tracked on a flat array for
JIT-stable is_dirty — see lazy.lua's _sub_msg_views).
Re-encode is passthrough: untouched views return their original
bytes verbatim; partially-dirty views walk fields in id order,
splicing clean segments and re-emitting dirty ones. WKT
descriptors (those with desc.decode) are eager-wrapped so the
API stays uniform.
Conformance: every interop fixture round-trips byte-equal through
`decode_lazy(b):encode()`. Trace stability: gated by `make
jit-trace` — index pass, sparse `:get` x2, and passthrough
`:encode` all compile with no fatal aborts.
Workload characteristics (from `tarantool bench/lazy_bench.lua`,
Person at 1KB / 10KB / 100KB, after the SoA index refactor):
- **Passthrough re-encode** is the headline win: **1.6–1.9×**
faster than eager decode→encode across all sizes and both modes.
Untouched views never re-walk the wire.
- **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
of eager**: break-even at small sizes, slight win at 100KB
(especially in runtime mode where eager pays more dispatch
cost). Earlier 0.60–0.77× regression came from one Lua table
per wire entry; replacing with parallel int arrays closed the
allocation gap.
- **Mutate-then-reencode** is **1.09–1.26× of eager**; the
per-field splice path now consistently beats full re-encode.
The honest framing: lazy is a *byte-passthrough* optimization
that also handles sparse reads at parity. Best fit: proxy /
router shapes that decode, touch a few fields, and re-encode.
t:Person_encode({name=...}) autocompletes in editors.
Codegen emits ---@class <full.Name> per message (with one
---@field per field), ---@alias <full.Name> integer per
enum, and ---@param / ---@return on every _new, _encode,
_decode, _decode_lazy, _has_*, _clear_* wrapper. Class
identifiers use proto full names verbatim so cross-file
references resolve. Lazy view types (pb.MessageView,
pb.ArrayView, pb.MapView) are declared inline in
runtime/pb/lazy.lua so the LSP sees them. Pure comment
addition — no runtime impact, 300/300 luatest + 19/19
jit-trace gate stay green.pb.from_pb(file_descriptor_set) — accepts binary FileDescriptorSet
bytes (output of protoc --descriptor_set_out=...) and returns
{files = {[name] = module}, order, lookup}. Each per-file module
has the same surface as pb.parse output (statically-generated
runtime mode). Translation pipeline: hand-built descriptor.proto
descriptors decode the wire bytes via pb.codec, then a translator
converts each FileDescriptorProto to the AST shape pb.parser
emits, which pb.dynamic.build consumes. Map fields are
reconstructed from synthetic entry messages (skipped from
nested_messages); proto3_optional is rehydrated as
optional=true instead of being modelled as a synthetic oneof.pb.json.encode / pb.json.decode).pb.text.encode(desc, t, opts) returns the
protoc --decode form (one field per line, 2-space indent, octal
byte escapes); opts.single_line=true collapses to a space-separated
one-liner for log lines and inline goldens. Codegen emits
M.<Type>_text(t, opts) in both modes. WKT types know their
idiomatic Lua shapes — Timestamp/Duration accept datetime cdata
or {seconds,nanos}, wrappers print their unwrapped scalar as
value: ..., Struct/Value/ListValue walk the tagged-table
form, FieldMask prints paths: ... per entry, Any stays opaque.pb.text.decode(desc, text, opts) is the
recursive-descent counterpart: handles every grammar bucket the
proto3 conformance suite exercises — decimal/hex/octal int literals,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflow to ±0), C-style + \u/\U string
escapes with adjacent-literal concat and surrogate rejection,
aggregate {} / <> bodies, repeated short-form [a, b, c],
key: K value: V map entries, the [type.googleapis.com/...]
inline Any form, enum-by-name-or-number, reserved-name drop, and
numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
duplicate singular fields, and threads through the conformance
runner — cmd/conformance/core.lua no longer skips text_payload.
Plugin gained a small reserved_names emitter so the parser can
match mainline TextFormat::Parser's "silently drop reserved" rule.
Proto3 text-format conformance suite: 8 ✓ / 426 skipped → 416
✓ / 18 skipped / 0 failures (the 18 are the proto2 message-type
bucket; everything in scope passes).known_failures.txt:
1. Duplicate JSON keys ({"foo":1,"foo":2}) rejected via a
byte-walking pre-scan that runs before json.decode.
2. camelCase / snake_case aliases of the same proto field
rejected via a per-message field_seen set.
3. JSON null inside repeated arrays and map values rejected.
4. Unknown enum names rejected by default; the
ignore_unknown_fields=true opt silently drops them (and
the conformance dispatch forwards this flag when
req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST).
5. google.protobuf.NullValue JSON canonical form: literal
null, not the string "NULL_VALUE". Null on a
NullValue-typed oneof member marks the oneof active.
6. Strict FieldMask round-trip (see M3 entry).protoc-gen-tarantool-doc: sibling Go plugin under
cmd/protoc-gen-tarantool-doc/ that emits one Markdown file per
input .proto. Sections: header (package + imports), messages
(per-message description + field table with # | Field | Type | Label | Description), enums (value table), services (method table
with unary / client / server / bidi streaming label).
Field type cells render scalar names, full type names for
message/enum references, and map<K, V> for maps; synthetic
map-entry messages are skipped. Leading comments are preserved via
SourceCodeInfo (squashed to a single line inside table cells).
Build with make build-doc; generate sample docs into
examples/docs/ with make gen-docs.require('protobuf') to
require('pb').For each message we emit one _encode and one _decode function. Tag
bytes are precomputed string literals; field-presence checks are inlined.
The runtime is reduced to wire-format primitives.
function M.Address_encode(t)
local out, n = {}, 0
local v
v = t.street
if v ~= nil and v ~= '' then
n = n + 1; out[n] = '\x0a' -- tag(1, LEN)
n = n + 1; out[n] = wire.encode_string(v)
end
-- ...
return table.concat(out)
end
Decoder uses an if-elseif chain on field id (LuaJIT compiles this
well for small chain lengths; for >16 fields we may want a numeric jump
table or tag-byte switching).
map<K,V> is wire-format-equivalent to:
message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;
Codegen synthesizes the entry message internally but exposes the field as a
Lua table (hash, not array). Encode iterates with pairs; decode merges
on duplicate keys (last wins, per spec).
Descriptor gains oneofs = {[oneof_name] = {field_names...}}. Encode walks
fields in declaration order and emits the first non-nil branch. Decode
clears sibling fields on assignment so callers see exactly one set.
In inline mode, the encode walk becomes an explicit if-chain; the decode
clears are emitted alongside each elseif id == N then arm.
Locked: int64/uint64/fixed64/sfixed64/sint64 are LuaJIT
int64_t / uint64_t cdata. Reasons:
msgpackffi, net.box, box.tuple.0 (numeric coercion in LuaJIT).We will document a wire.from_string(s) helper for users who get hex/dec
strings (e.g. from JSON) and need to feed them into encode.
t._unknown_fields is a single Lua string holding the verbatim
concatenation of tag+value bytes for fields the decoder didn't recognize.
Captured during decode in encounter order; re-emitted at the tail of
_encode. Mirrors Tarantool's built-in protobuf module convention.
Map entries don't preserve unknowns (per spec — synthetic Entry messages).
WKT types bypass this too, since they have custom desc.encode/decode.
Both codegen modes implement it: runtime mode in pb.codec, full (inline)
mode emits per-message capture/re-emit blocks. Tests live in
test/unknown_test.lua and run against both modes.
M.Greeter_service = {
name = 'hello.Greeter',
methods = {
SayHello = {
full_name = '/hello.Greeter/SayHello',
input = M.HelloRequest_descriptor,
output = M.HelloReply_descriptor,
client_streaming = false,
server_streaming = false,
},
-- ...
},
}
Client: M.Greeter_client(transport) returns
{SayHello = function(req) ... end, ...}.
Server: M.Greeter_server(impl) returns a table compatible with the
server:register(svc) interface.
Tests live in test/. Layout:
test/
unit/ -- pb.wire and pb.codec unit tests (no codegen)
wire_test.lua
codec_test.lua
roundtrip/ -- generated-module round-trip, both modes
scalars_test.lua
repeated_test.lua
nested_test.lua
map_test.lua -- M2
oneof_test.lua -- M2
wkt_test.lua -- M3
conformance/ -- M5: Google conformance harness
interop/ -- M5: cross-impl byte-for-byte equality
fixtures/ -- pre-encoded payloads from Go/Python
bench/ -- M6: microbenchmarks
fuzz/ -- malformed-input + random-input harness
For each wire.encode_<type> / wire.decode_<type> pair:
Every test runs against both generated modules (full and runtime mode) using a parameterized luatest group:
for _, mode in ipairs({'full', 'runtime'}) do
local g = t.group('roundtrip.' .. mode)
local hello = require('hello.' .. mode .. '.hello_pb')
g.test_address = function() ... end
-- ...
end
Coverage targets:
Build a small Lua binary cmd/conformance-runner.lua that speaks the
Google protobuf conformance protocol on stdin/stdout. Checked into the
repo so CI runs it against the canonical test corpus. Track pass-rate as
a CI metric.
For each of N reference protos:
protoc --encode=msg < input.txt > golden.bin (using mainline protoc).pb.decode(msg, read('golden.bin')) produces the
expected Lua table.pb.encode(msg, expected) produces bytes that, when decoded
by mainline protoc, yield the same logical message (allow field
reordering since neither order is canonical).Generate goldens once via a make goldens target; commit them to the
repo. CI just verifies them, never regenerates.
_decode function;
assert no crashes, only controlled error() calls.decode(encode(t)) == t (value-equal under our equality helper).math.random with a seeded PRNG for reproducibility.Microbenchmarks measured:
misc.memprof.Run on a fixed corpus across 5 message sizes; track results in
bench/baseline.json and fail PRs that regress >5%.
tt rocks install luatest to .rocks/.make test → .rocks/bin/luatest -v test/.go test ./... for any pure-Go logic in the plugin.protoc on a fixture, diff generated Lua
against committed expected output.build, gen,
test, bench, lint, goldens, conformance, clean, release.go test -cover for plugin; luacov for runtime.pkgsite for Go; manually maintained Markdown for
Lua until a tool emerges.| Question | Notes |
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add pb.encode_deterministic flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
| Should we ship a stub HTTP/2 transport for gRPC? | Probably no — large dependency. Recommend lua-http or write a focused project. |
How to handle the existing Tarantool-builtin require('protobuf')? |
Document migration; do not override the loader. |
Lua module path mapping when no lua_package and no proto package? |
Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
| Integration with Tarantool 3.x declarative config? | A pb.types.<name> config role would be nice; defer until use case appears. |
Edit this file directly. After any milestone closes:
## Done heading.README.md status table.