This document tracks the long-form design + testing strategy for
protoc-gen-tarantool and the pb Lua runtime. Short-term task tracking
lives in the in-session task list; this file is the slow-changing record.
A first-class Protocol Buffers + gRPC stack for Tarantool that:
.proto files with two modes (full inline
vs descriptor-driven runtime) so users can pick speed-vs-flexibility.net.box, fiber, box.session, and the 3.x config framework.See README.md for the user-visible summary. Internally:
mode=full inline + mode=runtime
descriptor-delegating wrappers). Sibling protoc-gen-tarantool-doc
emits Markdown reference per .proto.wire.lua + codec.lua
lazy.lua (zero-copy view) + text.lua (encode + decode) +
json.lua (strict proto3 JSON) + wkt.lua (all 9 well-known types) +
grpc.lua (transport + loopback) + parser.lua / dynamic.lua /
fileset.lua (three runtime descriptor producers — .proto source,
AST, FileDescriptorSet bytes).conformance_test_runner v34.1).bench/baseline.json tracks allocation/op (regression
gate at 5%); just jit-trace pins LuaJIT trace stability.The remaining unfinished bullets in section 3 are mostly M8 (release engineering) and a handful of optional perf items in M6.
Each milestone ends with a green CI run, an updated README, and a tagged release on sourcecraft.dev.
pb.wire.
wire.encode_int32(v), wire.decode_string(buf, pos), etc., for all
15 scalar proto types. Both modes consume the same primitives._encode / _decode
functions with no descriptor lookup. Tag bytes precomputed at gen
time as Lua string literals. This is the JIT-friendly hot path.mode=full|runtime (default: full).examples/expected/{full,runtime}/
for visual diffing.map<K,V>: emit as repeated synthetic *Entry messages with
key/value fields per spec. Decode merges into a Lua table; encode
iterates with pairs. Key types limited to scalars + string per spec.oneof: descriptor includes oneof_index per field. Encode
emits at most one branch (last assignment wins). Decode clears prior
oneof siblings on assignment. Hot-path lookup goes through
desc.oneofs_list (flat array) to keep the trace JIT-stable.optional: respect presence — emit field even
when value equals scalar default. Generated descriptor exposes
has_<name>(t) / clear_<name>(t) helpers.google.protobuf.Timestamp ↔ Tarantool datetime module
(epoch + nsec mapping). Out-of-spec inputs (negative nanos,
year > 9999) keep the raw {seconds, nanos} table so JSON
serialization can reject them with serialize_error instead of
crashing on decode.google.protobuf.Duration ↔ {seconds, nanos} table (interval
proved a poor fit — it carries months/days that don't map
cleanly).repeated string. Strict round-trip validation —
snake_case paths must use only [a-z0-9_], no leading/trailing
_, no __, and _ must precede a lowercase letter (not a
digit). JSON form rejects any _ (must be lowerCamelCase).Empty.Any: opaque {type_url, value} form by default; pb.register(desc)
+ pb.any.pack(desc, t) / pb.any.unpack(any_t) for typed
round-trips. JSON canonical mapping emits the flat {"@type": ...}
object when the type is registered, falls back to base64 opaque form
otherwise.Struct, Value, ListValue ↔ idiomatic Lua tables.
box.NULL is the null_value sentinel (re-exported as pb.NULL).
pb.wkt.struct(t) / pb.wkt.list(t) tag tables when the auto-detect
heuristic (t[1] ~= nil → list, else struct) needs to be overridden,
and decode preserves the tag for byte-stable round trips.Done when: WKT-using protos round-trip and integrate visibly with
datetime (e.g. os.date(...)-comparable timestamps).
/pkg.Svc/Method,
input/output type refs, client_streaming / server_streaming flags.MyService_client(transport) returns a table with
one function per method. Transport interface, all served by the
shipped loopback + multiplex implementations:
lua transport:unary(path, req_bytes, ctx) -> resp_bytes transport:server_stream(path, req_bytes, ctx) -> stream transport:client_stream(path, ctx) -> stream transport:bidi(path, ctx) -> stream
Streaming returns a stream object with send / close_send /
recv() -> msg, err / cancel. End-of-stream is (nil, nil);
handler errors surface as (nil, err_string).MyService_server(impl) returns
{service, methods, streams} consumable by the loopback /
multiplex transports. Handler signatures by RPC kind:
lua unary: function(req, ctx) -> resp server_stream: function(req, stream, ctx) client_stream: function(stream, ctx) -> resp bidi: function(stream, ctx) docs/specs/grpc_transports.md:
pb.grpc.transport.http_server — Connect-JSON over HTTP/1.1 via
tarantool/http. Default external transport; works with browsers
and curl without an HTTP/2 proxy.pb.grpc.transport.netbox — gRPC tunneled over net.box calls.
First-class in-cluster path.pb.grpc.transport.http_client_unary — outbound, unary only,
via http_client.connectrpc/conformance (same framed-runner
shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect
from one harness).Done. The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via fiber.channel. All four flavors
(unary + 3 streaming) are exercised by parameterized luatest groups.
[x] Wire up Google's protobuf conformance test runner.
cmd/conformance-runner.lua is the testee: reads length-prefixed
ConformanceRequest on stdin, runs it through our codec / JSON /
text codecs, writes a length-prefixed ConformanceResponse on
stdout. Loops to EOF. Core dispatch lives in
cmd/conformance/core.lua and is exercised directly by
test/conformance_test.lua so the inner dev loop doesn't need
Docker.
[x] Run the canonical conformance_test_runner binary locally and
track the pass-rate as a numeric metric (regression gate).
docker/conformance.Dockerfile builds conformance_test_runner
from upstream protobuf v34.1 source (matching the host's
libprotoc 34.1) and bundles Tarantool 3 from the official deb;
just conformance regenerates Lua then runs the harness against
cmd/conformance-runner.lua with the repo mounted as a volume.
Watchlists at test/conformance/known_failures.txt (binary +
JSON suite) and test/conformance/known_failures_text.txt
(text-format suite); both are empty for the proto3 suites as
of 2026-05-16.
Current baseline:
- Binary+JSON suite: **1493 ✓ / 1313 skipped / 0 failures**
- Text-format suite: **416 ✓ / 18 skipped / 0 failures**
The 1313 + 18 skipped all target
`protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
is a separate slice — see [docs/codegen.md](../docs/codegen.md)
for what it would take.
Strict-validation closures landed across three commits on the
`text-conformance-output` branch:
- `pb.text.decode` — full grammar coverage (recursive-descent
parser, ~580 LOC).
- `codec` -0.0 preservation — float/double `is_default_scalar`
and the inline-codegen elision both gained a sign-bit guard
(`1/v == math.huge`).
- `pb.json` strict-validation pass — duplicate-key rejection
(literal + camel/snake alias detection via a byte-walking
pre-scan), null-in-container rejection, unknown-enum-name
rejection (with `ignore_unknown_fields` opt for the
`JSON_IGNORE_UNKNOWN_PARSING_TEST` conformance category),
`google.protobuf.NullValue` round-trip as JSON `null`,
strict FieldMask round-trip validation.
The `PB_CONFORMANCE_SKIP_JSON=1` env var still exists to
short-circuit JSON output if a future encoder bug starts
crashing jsoncpp.
[ ] CI wire-up. The Docker image build is the long pole (~10–15 min on a clean cache); a registry push from a scheduled job would let CI runs reuse a warm cache.
[x] Cross-impl interop: 18-fixture corpus in test/interop/fixtures/
produced by mainline protoc --encode; tests assert byte-for-byte
equality.
[x] Microbenchmarks: encode and decode throughput (MB/s, msgs/s) for
messages of 5 sizes (10 B / 100 B / 1 KB / 10 KB / 100 KB). Shipped
as bench/bench.lua, run via just bench. Throughput is stderr-only
(varies with CPU load); JSON document on stdout.
[x] Allocation profiling — bytes per encode/decode op, measured via GC
delta with collectgarbage('stop') framing. Committed as
bench/baseline.json. Regression gate: just bench-compare exits
non-zero if alloc/op grows >5% vs baseline. Allocations are
deterministic to ~10 bytes regardless of hardware.
[x] Trace stability — just jit-trace (bench/jit_trace.lua)
attaches a jit.attach('trace') listener over the hot
encode/decode paths and asserts no aborts in our source files
fall into the fatal set (NYI bytecode, blacklisting, persistent
type instability). Pass: 13/13 scenarios. Two fixes shipped:
decode_varint grew a 1-byte fast path so callers no longer drag
an inner loop into the root trace; pb.finalize_message now
precomputes desc.oneofs_list so runtime-mode oneof encoding
uses ipairs instead of pairs (the latter compiles to bytecode
ISNEXT, which is NYI in LuaJIT 2.1). The gate also reports
interpreter-bridge counts as a benchmark-quality metric (decoders
have 0–4 per run depending on JIT timing — caused by side traces
returning from inlined decode_varint calls, which LuaJIT 2.1
can't stitch back cleanly; small per-call overhead, structural
to the engine). Scope caveat: map fields still encode via
pairs() and remain off-trace — pinned by the gate's last
scenario so we notice if upstream lifts the restriction.
[ ] Optional output: ibuf-based encoder that writes into a caller-owned
ffi.cdata byte buffer instead of building a string list. Targets
hot RPC paths where allocation cost dominates.
[x] Decoder fast path that returns an msgpack.object-like lazy view
for nested messages; only materializes touched fields.
Shipped as runtime/pb/lazy.lua + M.<Type>_decode_lazy codegen
stubs in both modes. Surface: :get / :has / :which / :iter / :names on MessageView; :len / :at / :iter / :tolist on
ArrayView; :get / :has / :keys / :iter / :totable on MapView.
Mutation via :set is supported and propagates sub-view edits
transparently (sub-MessageViews tracked on a flat array for
JIT-stable is_dirty — see lazy.lua's _sub_msg_views).
Re-encode is passthrough: untouched views return their original
bytes verbatim; partially-dirty views walk fields in id order,
splicing clean segments and re-emitting dirty ones. WKT
descriptors (those with desc.decode) are eager-wrapped so the
API stays uniform.
Conformance: every interop fixture round-trips byte-equal through
`decode_lazy(b):encode()`. Trace stability: gated by `make
jit-trace` — index pass, sparse `:get` x2, and passthrough
`:encode` all compile with no fatal aborts.
Workload characteristics (from `tarantool bench/lazy_bench.lua`,
Person at 1KB / 10KB / 100KB, after the SoA index refactor):
- **Passthrough re-encode** is the headline win: **1.6–1.9×**
faster than eager decode→encode across all sizes and both modes.
Untouched views never re-walk the wire.
- **Sparse read** (`:get` two top-level fields) is **0.90–1.16×
of eager**: break-even at small sizes, slight win at 100KB
(especially in runtime mode where eager pays more dispatch
cost). Earlier 0.60–0.77× regression came from one Lua table
per wire entry; replacing with parallel int arrays closed the
allocation gap.
- **Mutate-then-reencode** is **1.09–1.26× of eager**; the
per-field splice path now consistently beats full re-encode.
The honest framing: lazy is a *byte-passthrough* optimization
that also handles sparse reads at parity. Best fit: proxy /
router shapes that decode, touch a few fields, and re-encode.
t:Person_encode({name=...}) autocompletes in editors.
Codegen emits ---@class <full.Name> per message (with one
---@field per field), ---@alias <full.Name> integer per
enum, and ---@param / ---@return on every _new, _encode,
_decode, _decode_lazy, _has_*, _clear_* wrapper. Class
identifiers use proto full names verbatim so cross-file
references resolve. Lazy view types (pb.MessageView,
pb.ArrayView, pb.MapView) are declared inline in
runtime/pb/lazy.lua so the LSP sees them. Pure comment
addition — no runtime impact, 300/300 luatest + 19/19
jit-trace gate stay green.pb.from_pb(file_descriptor_set) — accepts binary FileDescriptorSet
bytes (output of protoc --descriptor_set_out=...) and returns
{files = {[name] = module}, order, lookup}. Each per-file module
has the same surface as pb.parse output (statically-generated
runtime mode). Translation pipeline: hand-built descriptor.proto
descriptors decode the wire bytes via pb.codec, then a translator
converts each FileDescriptorProto to the AST shape pb.parser
emits, which pb.dynamic.build consumes. Map fields are
reconstructed from synthetic entry messages (skipped from
nested_messages); proto3_optional is rehydrated as
optional=true instead of being modelled as a synthetic oneof.pb.json.encode / pb.json.decode).pb.text.encode(desc, t, opts) returns the
protoc --decode form (one field per line, 2-space indent, octal
byte escapes); opts.single_line=true collapses to a space-separated
one-liner for log lines and inline goldens. Codegen emits
M.<Type>_text(t, opts) in both modes. WKT types know their
idiomatic Lua shapes — Timestamp/Duration accept datetime cdata
or {seconds,nanos}, wrappers print their unwrapped scalar as
value: ..., Struct/Value/ListValue walk the tagged-table
form, FieldMask prints paths: ... per entry, Any stays opaque.pb.text.decode(desc, text, opts) is the
recursive-descent counterpart: handles every grammar bucket the
proto3 conformance suite exercises — decimal/hex/octal int literals,
float specials (inf/infinity/nan any case, oversize exponents
saturating to ±inf, underflow to ±0), C-style + \u/\U string
escapes with adjacent-literal concat and surrogate rejection,
aggregate {} / <> bodies, repeated short-form [a, b, c],
key: K value: V map entries, the [type.googleapis.com/...]
inline Any form, enum-by-name-or-number, reserved-name drop, and
numeric-field-ID tolerance. Range-checks 32/64-bit ints, rejects
duplicate singular fields, and threads through the conformance
runner — cmd/conformance/core.lua no longer skips text_payload.
Plugin gained a small reserved_names emitter so the parser can
match mainline TextFormat::Parser's "silently drop reserved" rule.
Proto3 text-format conformance suite: 8 ✓ / 426 skipped → 416
✓ / 18 skipped / 0 failures (the 18 are the proto2 message-type
bucket; everything in scope passes).known_failures.txt:
1. Duplicate JSON keys ({"foo":1,"foo":2}) rejected via a
byte-walking pre-scan that runs before json.decode.
2. camelCase / snake_case aliases of the same proto field
rejected via a per-message field_seen set.
3. JSON null inside repeated arrays and map values rejected.
4. Unknown enum names rejected by default; the
ignore_unknown_fields=true opt silently drops them (and
the conformance dispatch forwards this flag when
req.test_category == JSON_IGNORE_UNKNOWN_PARSING_TEST).
5. google.protobuf.NullValue JSON canonical form: literal
null, not the string "NULL_VALUE". Null on a
NullValue-typed oneof member marks the oneof active.
6. Strict FieldMask round-trip (see M3 entry).protoc-gen-tarantool-doc: sibling Go plugin under
cmd/protoc-gen-tarantool-doc/ that emits one Markdown file per
input .proto. Sections: header (package + imports), messages
(per-message description + field table with # | Field | Type | Label | Description), enums (value table), services (method table
with unary / client / server / bidi streaming label).
Field type cells render scalar names, full type names for
message/enum references, and map<K, V> for maps; synthetic
map-entry messages are skipped. Leading comments are preserved via
SourceCodeInfo (squashed to a single line inside table cells).
Build with just build-doc; generate sample docs into
examples/docs/ with just gen-docs.require('protobuf') to
require('pb').For each message we emit one _encode and one _decode function. Tag
bytes are precomputed string literals; field-presence checks are inlined.
The runtime is reduced to wire-format primitives.
function M.Address_encode(t)
local out, n = {}, 0
local v
v = t.street
if v ~= nil and v ~= '' then
n = n + 1; out[n] = '\x0a' -- tag(1, LEN)
n = n + 1; out[n] = wire.encode_string(v)
end
-- ...
return table.concat(out)
end
Decoder uses an if-elseif chain on field id (LuaJIT compiles this
well for small chain lengths; for >16 fields we may want a numeric jump
table or tag-byte switching).
map<K,V> is wire-format-equivalent to:
message FooEntry { K key = 1; V value = 2; }
repeated FooEntry foos = N;
Codegen synthesizes the entry message internally but exposes the field as a
Lua table (hash, not array). Encode iterates with pairs; decode merges
on duplicate keys (last wins, per spec).
Descriptor gains oneofs = {[oneof_name] = {field_names...}}. Encode walks
fields in declaration order and emits the first non-nil branch. Decode
clears sibling fields on assignment so callers see exactly one set.
In inline mode, the encode walk becomes an explicit if-chain; the decode
clears are emitted alongside each elseif id == N then arm.
Locked: int64/uint64/fixed64/sfixed64/sint64 are LuaJIT
int64_t / uint64_t cdata. Reasons:
msgpackffi, net.box, box.tuple.0 (numeric coercion in LuaJIT).Two coercion helpers — pb.to_uint64(v) and pb.to_int64(v) (re-exported
from pb.wire) — accept Lua number, cdata, or a numeric string and return
the matching 64-bit cdata. Use them at the boundary when values come from
JSON, text format, or net.box arguments where the type isn't already cdata.
t._unknown_fields is a single Lua string holding the verbatim
concatenation of tag+value bytes for fields the decoder didn't recognize.
Captured during decode in encounter order; re-emitted at the tail of
_encode. Mirrors Tarantool's built-in protobuf module convention.
Map entries don't preserve unknowns (per spec — synthetic Entry messages).
WKT types bypass this too, since they have custom desc.encode/decode.
Both codegen modes implement it: runtime mode in pb.codec, full (inline)
mode emits per-message capture/re-emit blocks. Tests live in
test/unknown_test.lua and run against both modes.
M.Greeter_service = {
name = 'hello.Greeter',
methods = {
SayHello = {
full_name = '/hello.Greeter/SayHello',
input = M.HelloRequest_descriptor,
output = M.HelloReply_descriptor,
client_streaming = false,
server_streaming = false,
},
-- ...
},
}
Client: M.Greeter_client(transport) returns
{SayHello = function(req) ... end, ...}.
Server: M.Greeter_server(impl) returns a table compatible with the
server:register(svc) interface.
Tests live as a flat test/*.lua set (15 files, 39 luatest groups, 639
assertions as of 2026-05-16). Each behavior file is parametrized over
both codegen modes via the pattern:
for _, mode in ipairs({'full', 'runtime'}) do
local g = t.group(name .. '.' .. mode)
local hello = require(mode .. '.hello.hello_pb')
g.test_x = function() ... end
end
Categories — see file names under test/:
protobuf_test
(scalars, repeated, nested, optional, oneof) and interop_test
(the byte-for-byte fixture corpus under test/interop/fixtures/,
10 .txtpb/.bin pairs from mainline protoc --encode).protobuf_test for map/oneof/explicit-optional.struct_value_test, any_fieldmask_test.lazy_test covers MessageView/ArrayView/MapView + the
byte-splice re-encode path.json_test, text_test, text_decode_test,
unknown_test.dynamic_test (.proto source via
pb.parse), fileset_test (FileDescriptorSet via pb.from_pb),
with parity assertions against the generated modules.codegen_doc_comments_test,
doc_test (the protoc-gen-tarantool-doc plugin), conformance_test
(self-test of the runner against crafted requests).grpc_streaming_test against the loopback transport.Cross-mode parity is pinned by parity.full_vs_runtime.* groups
inside each behavior file.
The Google protobuf conformance suite is driven by
cmd/conformance-runner.lua (stdin/stdout framing) and run via
just conformance (Docker-bundled conformance_test_runner). Known
failures live in test/conformance/known_failures.txt (binary+JSON)
and test/conformance/known_failures_text.txt (text-format) — both
empty for proto3 as of 2026-05-16.
Interop fixtures (test/interop/fixtures/*.{txtpb,bin}) are produced
by mainline protoc --encode via just goldens and asserted in
interop_test.lua for round-trip equality across both codegen modes.
Map fixtures use a single key to lock byte-for-byte equality (Lua
pairs ordering won't match protoc's text-proto-order output);
multi-key map behavior is covered via decode-then-compare-Lua-table
rather than byte equality.
bench/baseline.json tracks allocation per op (a deterministic
metric; throughput swings 30%+ across machines). just bench-compare
fails on >5% alloc regression. just jit-trace (bench/jit_trace.lua)
asserts the hot encode/decode paths stay on the JIT trace — a
companion safety net for the "no pairs() on hot paths" rule.
go test. Codegen correctness is asserted end-to-end via
Lua tests against committed expected outputs in
examples/expected/{full,runtime}/.build, gen,
test, goldens, bench, bench-baseline, bench-compare,
jit-trace, clean. Justfile adds just conformance (Docker-bundled
conformance runner).go test -cover for plugin; luacov for runtime.pkgsite for Go; manually maintained Markdown for
Lua until a tool emerges.| Question | Notes |
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add pb.encode_deterministic flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
| Should we ship a stub HTTP/2 transport for gRPC? | Answered in docs/specs/grpc_transports.md: no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. |
| msgpack-flavored encoder for proto schemas? | Design sketched in docs/specs/msgpack_encoding.md. Open: map-keyed-by-int (default) vs name; ARRAY layout for box.space feeders; MP_TUPLE ext opt-in. |
How to handle the existing Tarantool-builtin require('protobuf')? |
Document migration; do not override the loader. |
Lua module path mapping when no lua_package and no proto package? |
Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
| Integration with Tarantool 3.x declarative config? | A pb.types.<name> config role would be nice; defer until use case appears. |
docs/specs/grpc_transports.md.
Connect-JSON over HTTP/1.1 is the default external transport.Edit this file directly. After any milestone closes:
## Done heading.README.md status table.