# tarantool-protobuf

A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto3) and
gRPC service stubs from [Tarantool](https://www.tarantool.io/).

Tarantool ships an in-tree `require('protobuf')` module, but it is
**encode-only** and has no support for `map`, `oneof`, services, or any decode
path. This project fills those gaps with:

- **`protoc-gen-tarantool`** — a protoc plugin (Go) that turns `.proto` files
  into Lua modules.
- **`runtime/pb`** — a pure Lua + LuaJIT-FFI runtime the generated code uses
  for the wire format. Named `pb` rather than `protobuf` to avoid colliding
  with Tarantool's built-in module.

## Status

MVP — proto3 messages and enums, end-to-end round-trip verified.

| Feature                          | State        |
|----------------------------------|--------------|
| proto3 scalars (all 15 types)    | ✅           |
| Repeated, packed by default      | ✅           |
| Nested messages, self-reference  | ✅           |
| Cross-file imports               | ✅           |
| Enums (open semantics)           | ✅           |
| 64-bit integers as LuaJIT cdata  | ✅           |
| Two codegen modes (full + runtime) | ✅         |
| `map<K,V>` (scalar/message values) | ✅         |
| `oneof`                          | ✅           |
| proto3 explicit `optional` + has_/clear_ | ✅   |
| gRPC service stubs (unary)       | ✅           |
| gRPC streaming (server / client / bidi) | ✅      |
| Loopback / multiplex transport   | ✅           |
| WKT: Timestamp ↔ `datetime`      | ✅           |
| WKT: Duration, Empty, wrappers   | ✅           |
| WKT: Struct, Value, ListValue    | ✅           |
| WKT: Any (opaque + registry pack/unpack) | ✅   |
| WKT: FieldMask                   | ✅           |
| Byte-for-byte interop with `protoc` | ✅ (18 fixtures) |
| Google conformance runner (`cmd/conformance-runner.lua`) | ✅ |
| Runtime `.proto` parsing (`pb.parse`) | ✅           |
| Runtime `FileDescriptorSet` ingest (`pb.from_pb`) | ✅           |
| Markdown doc generator (`protoc-gen-tarantool-doc`) | ✅           |
| proto3 JSON (`pb.json.encode`/`.decode`) | ✅       |
| Text format printer (`pb.text.encode` / `<Msg>_text`) | ✅ (encode-only) |
| Unknown-field passthrough (`_unknown_fields`) | ✅      |
| Microbenchmark + alloc regression gate (`make bench`) | ✅ |
| proto2 / editions                | ❌ out of scope |

## Quick start

```bash
# 1. Build the plugin and generate the example.
make gen

# 2. Run the round-trip test in Tarantool.
make test
```

The plugin emits one `.lua` file per `.proto`. By default the output path
mirrors the proto package (`package foo.bar; baz.proto` → `foo/bar/baz_pb.lua`),
required as `foo.bar.baz_pb`. Override the Lua module path with a file option:

```proto
import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";
```

## Generated API

For each message `Foo` the plugin emits:

```lua
local M = require('myapp.proto.foo')

M.Foo_descriptor    -- the descriptor table consumed by the runtime
M.Foo_new(t)        -- returns t (or {}); placeholder for future validation
M.Foo_encode(t)     -- table -> wire bytes (string)
M.Foo_decode(b)     -- wire bytes (string) -> table
M.Foo_decode_lazy(b) -- wire bytes -> MessageView (zero-copy view)
M.Foo_text(t, opts) -- table -> protoc-style text format (debug printer)
```

For each enum `Color`:

```lua
M.Color_descriptor          -- { name, by_name, by_value }
M.Color                     -- alias for by_name: M.Color.RED -> 0
```

Repeated fields are Lua arrays (1-based, contiguous). 64-bit integers
(`int64`, `uint64`, `fixed64`, `sfixed64`, `sint64`) are LuaJIT `int64_t` /
`uint64_t` cdata — lossless and the same convention used by Tarantool's
`net.box`, `msgpack`, and built-in `protobuf` modules.

## Layout

```
cmd/protoc-gen-tarantool/   # Go plugin
  main.go                   # reads CodeGeneratorRequest, hands off to gen
  internal/gen/             # codegen package
runtime/pb/                 # Lua runtime (require('pb'))
  init.lua                  # public surface
  wire.lua                  # varint / zigzag / fixed / float / LEN
  codec.lua                 # descriptor-driven encode/decode
options/tarantool/          # custom proto options
  tarantool.proto           # (tarantool.lua_package) file option
examples/proto/             # demo .proto inputs
examples/expected/          # generated output (committed for inspection)
test/                       # roundtrip test
```

## Conformance

`cmd/conformance-runner.lua` speaks the [Google protobuf conformance
protocol][gconf] on stdin/stdout. Drive it with the canonical
`conformance_test_runner` binary like so:

```bash
make gen
conformance_test_runner --enforce_recommended \
    tarantool cmd/conformance-runner.lua
```

Homebrew's `protobuf` package does not ship `conformance_test_runner`, so a
Dockerfile under `docker/conformance.Dockerfile` builds it from upstream
protobuf source and bundles Tarantool. Run the full suite locally with:

```bash
just conformance
```

(Mounts the repo into the container — generated Lua from `make gen` on the
host is what gets tested.) Known failures live in `test/conformance/known_failures.txt` (binary +
JSON suite) and `test/conformance/known_failures_text.txt` (text-format
suite); the runner exits zero only when actual failures match those
lists exactly.

Current baseline (2026-05-15, protobuf v34.1):

| Suite | Successes | Skipped | Expected failures |
|-------|-----------|---------|-------------------|
| Binary + JSON | 803 | 1864 | 139 |
| Text-format   |   0 |  430 |   4 |

JSON output is gated behind `PB_CONFORMANCE_SKIP_JSON=1` (set in the
container `ENV`) until the JSON codec is hardened — the harness's
strict jsoncpp comparator crashes on a subset of our half-finished
output. Host-side `make test` still exercises the full JSON path.

The runner currently supports `protobuf_test_messages.proto3.TestAllTypesProto3`
in both protobuf and JSON formats; proto2 / editions / JSPB / text-format
cases return `skipped`. The self-test in `test/conformance_test.lua`
exercises the runner with crafted requests on every `make test` run.

[gconf]: https://github.com/protocolbuffers/protobuf/tree/main/conformance

## Benchmarks

```bash
make bench           # print throughput + alloc per op (5 sizes × 2 modes)
make bench-baseline  # overwrite bench/baseline.json (run on a quiet machine)
make bench-compare   # exit 1 if any alloc-per-op regressed >5% vs baseline
```

The committed `bench/baseline.json` tracks only allocation per op — that's
reproducible across machines because it counts bytes, not time. Throughput
in stderr is informational; it swings 30%+ on a contended CPU.

Current baseline (LuaJIT 2.1, hello.Person):

| Payload | encode alloc (full / runtime) | decode alloc (full / runtime) |
|---------|-------------------------------|-------------------------------|
| 10 B    | 0.37 / 0.63 KB                | 0.51 / 0.51 KB                |
| 100 B   | 0.37 / 0.63 KB                | 0.51 / 0.51 KB                |
| 1 KB    | 5.98 / 7.04 KB                | 7.75 / 7.88 KB                |
| 10 KB   | 47.5 / 48.6 KB                | 59.5 / 59.6 KB                |
| 100 KB  | 444 / 445 KB                  | 568 / 568 KB                  |

## Why named `pb` instead of `protobuf`?

Tarantool's loader prefers the built-in `require('protobuf')` over any
filesystem module of the same name. Trying to override it would break code
that uses the built-in's encode API. `pb` is short, unambiguous, and lives
alongside the built-in.

## License

TBD.