From cef42bf3d8349e953917d9a0ba1522edadf36bd6 Mon Sep 17 00:00:00 2001 From: Eugene Blikh Date: Sun, 17 May 2026 08:17:14 +0300 Subject: [PATCH] docs: proto2 is supported for the core surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README's status table swaps the single \"proto2 deferred\" row for four entries that name what works (required / optional / custom defaults / packed semantics) and what doesn't (extend, extensions, group, MessageSet) so users can decide before they generate. docs/codegen.md replaces the proto2 deferral section with a \"Proto2 support\" page: the three new descriptor attributes (required, default_value, optional everywhere), the existing IsPacked() trick that already produces the right unpacked default, and a concrete list of the four features that still refuse to generate (with the MessageSet upstream-rejection caveat). PLAN.md gains M9 — Proto2 baseline. Wire format, codecs, parser, dynamic, tests are checked off; extend / group / MessageSet stay open because the conformance schema needs all three and they're each their own slice. --- PLAN.md | 45 ++++++++++++++++--- README.md | 36 +++++++++++----- docs/codegen.md | 112 +++++++++++++++++++++++++----------------------- 3 files changed, 124 insertions(+), 69 deletions(-) diff --git a/PLAN.md b/PLAN.md index 336ad61feefa70590aa798fd47d8a48b441e0a88..7c0ece341fb394cc320597d26bf5c2a1a534c93b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -151,7 +151,7 @@ release on sourcecraft.dev. fiber and bridges client ↔ handler via `fiber.channel`. All four flavors (unary + 3 streaming) are exercised by parameterized luatest groups. -### M5 — Conformance + interop *(proto3 closed; CI wire-up + proto2 deferred)* +### M5 — Conformance + interop *(proto3 closed; CI wire-up + proto2 conformance deferred)* - [x] Wire up Google's [protobuf conformance test runner][conformance]. `cmd/conformance-runner.lua` is the testee: reads length-prefixed @@ -179,8 +179,10 @@ fiber and bridges client ↔ handler via `fiber.channel`. All four flavors The 1313 + 18 skipped all target `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen - is a separate slice — see [docs/codegen.md](../docs/codegen.md) - for what it would take. + itself is now supported for the core feature set (M9 below), but + that schema also needs `extend`/`extensions`/`group`/`MessageSet` + which we don't implement — see + [docs/codegen.md#proto2-support](../docs/codegen.md) for the gap. Strict-validation closures landed across three commits on the `text-conformance-output` branch: @@ -385,6 +387,38 @@ fiber and bridges client ↔ handler via `fiber.channel`. All four flavors - [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via protobuf-encoded events on a queue. +### M9 — Proto2 baseline *(core done; conformance gated on legacy features)* + +- [x] Plugin accepts `syntax = "proto2"`. Field descriptors now carry + `required = true`, `default_value = …`, and `optional = true` for + every singular field. Existing `IsPacked()`-based packing logic + already produced the correct proto2 unpacked-by-default. +- [x] Runtime parser captures `required`, `[default = X]`, and threads + them through `pb.dynamic`. The dynamic builder now calls + `codec.compile_writers/compile_readers` so source-parsed schemas + get the same per-field specializations build-time codegen does. +- [x] Codec: `required` writer errors on missing-on-encode in both + inline and runtime modes. +- [x] JSON + text codecs: presence-tracked elision (`required`, + `optional`, oneof) bypasses the proto3 implicit-zero rules. +- [x] 40 luatest cases (`proto2_basic.*` groups) — full, runtime, and + dynamic-from-source byte parity. +- [ ] **`extend` blocks / `extensions` ranges.** Currently extensions + survive as `_unknown_fields` bytes (so re-encode is lossless), + but the plugin doesn't emit accessors. Needed for the proto2 + conformance schema. +- [ ] **`group` fields** (legacy `optional group Foo = 1 { ... }`). + Wire types 3/4 SGROUP/EGROUP aren't in the encode/decode tables; + `protoreflect.GroupKind` falls off `scalarName` so any schema + containing a group field fails to generate. +- [ ] **`MessageSet`** wire format. `protobuf-go`'s protoreflect + refuses to load a FileDescriptor declaring + `option message_set_wire_format = true;`, calling it a removed + proto1 feature. Can't bypass without forking the library. +- [ ] **Conformance numbers.** All 1313 + 18 currently-skipped tests + target `TestAllTypesProto2`, which exercises all three of the + above. Won't move until at least extend + group are wired up. + ## 4. Per-feature design notes ### 4.1 Inline (full) codegen — wire bytes @@ -645,8 +679,9 @@ companion safety net for the "no `pairs()` on hot paths" rule. ## 8. Non-goals (for now) -- **proto2** syntax — spec is more complex (required, default values, - groups), real demand is rare on Tarantool. +- **proto2 `extend` / `extensions` / `group` / `MessageSet`** — the + conformance-only edges. Core proto2 (required, optional, custom + defaults, packed semantics) is in; see M9. - **Protobuf editions** beyond what proto3 enables — see §4.6 for the per-feature analysis (what we'd gain, what it would cost, the conditions that would force a revisit). diff --git a/README.md b/README.md index e5c1f650c8a78a4d01489f38320433e94117860b..afc9c91d99974148012f774cceffac8a92bbfc05 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # tarantool-protobuf -A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto3) and -gRPC service stubs from [Tarantool](https://www.tarantool.io/). +A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto2 + +proto3) and gRPC service stubs from [Tarantool](https://www.tarantool.io/). Tarantool ships an in-tree `require('protobuf')` module, but it is **encode-only** and has no support for `map`, `oneof`, services, or any decode @@ -50,7 +50,12 @@ both the binary+JSON and text-format suites passes. | Text format (`pb.text.encode` / `pb.text.decode`) | ✅ | | Unknown-field passthrough (`_unknown_fields`) | ✅ | | Microbenchmark + alloc regression gate (`just bench`) | ✅ | -| proto2 / editions | ❌ deferred (separate slice; see [docs/codegen.md](docs/codegen.md)) | +| proto2 `required` / `optional` / custom `[default = X]` | ✅ | +| proto2 repeated unpacked-by-default + `[packed = true]` | ✅ | +| proto2 extensions / `extend` blocks | ❌ (passes through as unknown fields) | +| proto2 groups (legacy) | ❌ (parse rejected; no wire-type 3/4 codec) | +| Google conformance suite — proto2 | ❌ blocked on groups + extensions + MessageSet | +| Editions | ❌ deferred | ## Install @@ -144,10 +149,13 @@ done protoc -I proto --tarantool_out=prefix=myapp.proto:out $(find proto -name '*.proto') ``` -One pitfall: this plugin is proto3-only. If an upstream schema mixes -proto2 and proto3, either skip the proto2 files (provided your proto3 -side doesn't import them) or wait for proto2 support — see PLAN.md's -deferred non-goals. +Proto2 sources work for the core surface — `required`, `optional`, +custom `[default = X]`, and the proto2 unpacked-by-default rule for +repeated scalars all generate correctly. The unsupported edges are +`extend` blocks, `extensions` ranges, and legacy `group` fields; +schemas that use any of those will fail to generate. See +[docs/codegen.md](docs/codegen.md#proto2-support) for the field-by-field +mapping. [strip]: https://git.srht.bigb.es/tarantool-etcd/tree/master/item/proto/_strip_annotations.py @@ -164,8 +172,10 @@ M.Foo_encode(t) -- table -> wire bytes (string) M.Foo_decode(b) -- wire bytes (string) -> table M.Foo_decode_lazy(b) -- wire bytes -> MessageView (zero-copy view) M.Foo_text(t, opts) -- table -> protoc-style text format (debug printer) -M.Foo_has_(t) -- only emitted for proto3 explicit-`optional` fields +M.Foo_has_(t) -- emitted for every presence-tracked field M.Foo_clear_(t) -- same + -- (proto3 explicit `optional`, proto2 `optional`, + -- oneof branches) M.Foo_fields -- strict {field = "field", ...} for lazy-view callers M.Foo_oneofs -- strict {oneof_group = "oneof_group", ...}, when any ``` @@ -354,9 +364,13 @@ Current baseline (2026-05-16, protobuf v34.1): | Text-format | **416** | 18 | 0 | 0 | The 1313 + 18 skipped tests all target -`protobuf_test_messages.proto2.TestAllTypesProto2`, which we don't generate -Lua for. Proto2 support is a separate slice — see -[docs/codegen.md#proto2-deferral](docs/codegen.md) for what it would take. +`protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen now +works for the core feature set (see Status table), but the conformance +schema relies on extensions (`extend`), `extensions` ranges, legacy +`group` fields, and a `MessageSetCorrect` that protoreflect rejects +outright as a removed proto1 feature. Until those are wired up the +proto2 suite stays at zero — see +[docs/codegen.md#proto2-support](docs/codegen.md) for the scope. The runner supports `protobuf_test_messages.proto3.TestAllTypesProto3` in binary, JSON, and text-format input/output, including the diff --git a/docs/codegen.md b/docs/codegen.md index 9d447e23925e1fc307da0ff452c5a963150f1ebf..2fcc09b1494dad4da80066a2adc774b6cf5e6a27 100644 --- a/docs/codegen.md +++ b/docs/codegen.md @@ -26,9 +26,12 @@ CodeGeneratorRequest (proto descriptor bytes on stdin) one `.lua` per input `.proto` ``` -The plugin only handles `syntax = "proto3"`. Proto2 input is rejected -at the top of `GenerateFile`. See [proto2 deferral](#proto2-deferral) -for the rationale. +The plugin accepts both `syntax = "proto3"` and `syntax = "proto2"`. +The descriptor surface and codegen branch on syntax-aware protoreflect +APIs (`Cardinality`, `HasDefault`, `IsPacked`) rather than maintaining +two parallel pipelines. See [proto2 support](#proto2-support) for the +field-by-field mapping and the (small) list of features that still +fail to generate. ## CLI options @@ -123,7 +126,8 @@ The shape consumed by the runtime is: {name='y', id=2, kind='message', message=}, {name='z', id=3, kind='enum', enum=}, {name='m', id=4, kind='map', key=, value=}, - -- modifiers: repeated, packed, oneof, optional + -- modifiers: repeated, packed, oneof, optional, + -- required (proto2), default_value (proto2) }, -- Filled in by pb.finalize_message: field_by_id = {[1]=, ...}, @@ -342,52 +346,54 @@ the same `CodeGeneratorRequest` and emits one Markdown file per input `examples/docs/` via `just gen-docs`. It's deliberately separate from the Lua codegen to keep the codegen plugin small. -## Proto2 deferral - -The plugin rejects `syntax = "proto2"` outright. The conformance -suites' 1331 skipped tests are all `TestAllTypesProto2`. - -Proto2 support is genuinely a separate slice. The wire format is -identical to proto3 (with the one exception of `group`s — wire-types 3 -and 4, SGROUP/EGROUP, which we already tolerate on the skip path but -don't encode/decode). The schema features that *don't* exist in proto3 -are what would need work: - -- **`required` fields** — codec must error on missing required at both - encode and decode (we currently treat absence as default). -- **`optional` everywhere** — proto2 fields are presence-tracked by - default. proto3's explicit-optional path already handles this; just - needs the plugin to mark every proto2 field as `optional=true`. -- **Extensions** (`extensions 100 to 199;` + `extend Foo {...}`) - — proto2-only mechanism for adding fields to messages defined - elsewhere. Round-trip overlaps with our existing - `_unknown_fields` machinery but explicit support needs a separate - descriptor table per extension and a way to expose it on user - messages. -- **Groups** (`optional group Foo = 1 { ... }`) — actual encode/decode - rather than the skip-only tolerance we have today, plus a different - text-format rendering (`Foo { ... }` with the group's capitalized - name instead of the submessage's field name). -- **Custom defaults** (`[default = X]`) — only matter when callers - query "what's the default value of field X"; for round-trips they - can be ignored. -- **`MessageSet`** wire format — a few specific tests. - -Realistic phasing (if you take this on): - -1. Drop the proto3-only gate in the plugin; mark proto2 fields as - `optional=true`; skip extensions / groups / required harmlessly - (warn but emit the rest of the message). Re-run conformance — - should unlock most of the 1007 binary tests and ~200 of the JSON - tests immediately because the wire format is the same. -2. Implement group encode/decode + the text-format group rendering. - Picks up the remaining text tests and a handful of binary tests. -3. Required-field validation. Small, targeted at the - `TestAllRequiredTypesProto2` cases. -4. Extensions, MessageSet, custom defaults — long tail, optional - depending on demand. - -This is a meaningfully larger slice than the typical addition here — -plan on a few sessions, not an evening. The trade-off is "proto2 is -mostly legacy" (see the README's status table for who actually uses -it) vs "1331 extra ✓ in the conformance numbers looks good". +## Proto2 support + +The plugin and runtime now accept `syntax = "proto2"`. The descriptor +table is the same shape as proto3 — three new attributes carry the +proto2-only semantics: + +| Attribute | Meaning | Set on | +|---|---|---| +| `required = true` | Field declared with the proto2 `required` keyword. Encoder errors if the value is missing on the input table; no default elision. | Singular non-oneof fields with `required` cardinality. | +| `default_value = …` | Lua literal for `[default = X]`. 64-bit ints arrive as cdata; enums use the symbolic name. | Any singular field with an explicit default. | +| `optional = true` | Already used for proto3 explicit-optional; in proto2 it ends up on every singular field with the `optional` keyword (which is most of them). | All proto2 fields declared `optional`. | + +The wire format itself is unchanged between syntaxes — `IsPacked()` +from protoreflect already produces the right packed-default per syntax, +so repeated proto2 scalars stay unpacked unless they carry +`[packed = true]`. The codec's required-writer specialization fires +through `codec.compile_writers`, which dynamic.lua now calls explicitly +(so source-parsed schemas behave the same as build-time codegen). + +`test/proto/proto2_basic.proto` exercises the surface; the +`proto2_basic.*` luatest groups (40 cases) cover defaults, required +validation, the unpacked-by-default rule, nested-required messages, +and full-vs-runtime-vs-dynamic byte parity. + +### What still doesn't work + +The conformance schema (`test_messages_proto2.proto` from upstream +protobuf) leans on three features we don't implement: + +- **`extend` blocks** + **`extensions 100 to 199;`** ranges — proto2's + open-message-extension mechanism. Extension values currently + round-trip through `_unknown_fields` like any other unknown tag, + so a message carrying an extension can be decoded → re-encoded + byte-identically. But the plugin won't emit accessors and there's + no way to read or set extension values from user code. +- **`group`** fields (legacy `optional group Foo = 1 { … }`) — uses + wire types 3 (SGROUP) and 4 (EGROUP) which our wire layer doesn't + encode or decode. We skip-tolerate them on the read side but + generating a message that *contains* a group field fails because + `protoreflect.GroupKind` falls off our `scalarName` switch. +- **`MessageSet`** wire format — `protobuf-go` rejects messages + declaring `option message_set_wire_format = true;` at the + protoreflect layer ("a legacy proto1 feature that is no longer + supported"). We can't bypass this without forking protoreflect. + +Implementing extensions and groups is the realistic path to closing +the conformance gap; MessageSet would need an upstream workaround. +Both are big enough lifts to be separate slices — proto2 is mostly +legacy and the existing extension-via-unknown-fields path covers the +common interop case (you receive bytes, you re-encode them, the +extension data passes through intact).