3 files changed, 124 insertions(+), 69 deletions(-)

M PLAN.md
M README.md
M docs/codegen.md

@@ 151,7 151,7 @@ release on sourcecraft.dev.
 fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
 (unary + 3 streaming) are exercised by parameterized luatest groups.
 
-### M5 — Conformance + interop  *(proto3 closed; CI wire-up + proto2 deferred)*
+### M5 — Conformance + interop  *(proto3 closed; CI wire-up + proto2 conformance deferred)*
 
 - [x] Wire up Google's [protobuf conformance test runner][conformance].
       `cmd/conformance-runner.lua` is the testee: reads length-prefixed


@@ 179,8 179,10 @@ fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
 
       The 1313 + 18 skipped all target
       `protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen
-      is a separate slice — see [docs/codegen.md](../docs/codegen.md)
-      for what it would take.
+      itself is now supported for the core feature set (M9 below), but
+      that schema also needs `extend`/`extensions`/`group`/`MessageSet`
+      which we don't implement — see
+      [docs/codegen.md#proto2-support](../docs/codegen.md) for the gap.
 
       Strict-validation closures landed across three commits on the
       `text-conformance-output` branch:


@@ 385,6 387,38 @@ fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
 - [ ] Example apps: pet-clinic CRUD over gRPC; replication of state via
       protobuf-encoded events on a queue.
 
+### M9 — Proto2 baseline  *(core done; conformance gated on legacy features)*
+
+- [x] Plugin accepts `syntax = "proto2"`. Field descriptors now carry
+      `required = true`, `default_value = …`, and `optional = true` for
+      every singular field. Existing `IsPacked()`-based packing logic
+      already produced the correct proto2 unpacked-by-default.
+- [x] Runtime parser captures `required`, `[default = X]`, and threads
+      them through `pb.dynamic`. The dynamic builder now calls
+      `codec.compile_writers/compile_readers` so source-parsed schemas
+      get the same per-field specializations build-time codegen does.
+- [x] Codec: `required` writer errors on missing-on-encode in both
+      inline and runtime modes.
+- [x] JSON + text codecs: presence-tracked elision (`required`,
+      `optional`, oneof) bypasses the proto3 implicit-zero rules.
+- [x] 40 luatest cases (`proto2_basic.*` groups) — full, runtime, and
+      dynamic-from-source byte parity.
+- [ ] **`extend` blocks / `extensions` ranges.** Currently extensions
+      survive as `_unknown_fields` bytes (so re-encode is lossless),
+      but the plugin doesn't emit accessors. Needed for the proto2
+      conformance schema.
+- [ ] **`group` fields** (legacy `optional group Foo = 1 { ... }`).
+      Wire types 3/4 SGROUP/EGROUP aren't in the encode/decode tables;
+      `protoreflect.GroupKind` falls off `scalarName` so any schema
+      containing a group field fails to generate.
+- [ ] **`MessageSet`** wire format. `protobuf-go`'s protoreflect
+      refuses to load a FileDescriptor declaring
+      `option message_set_wire_format = true;`, calling it a removed
+      proto1 feature. Can't bypass without forking the library.
+- [ ] **Conformance numbers.** All 1313 + 18 currently-skipped tests
+      target `TestAllTypesProto2`, which exercises all three of the
+      above. Won't move until at least extend + group are wired up.
+
 ## 4. Per-feature design notes
 
 ### 4.1 Inline (full) codegen — wire bytes


@@ 645,8 679,9 @@ companion safety net for the "no `pairs()` on hot paths" rule.
 
 ## 8. Non-goals (for now)
 
-- **proto2** syntax — spec is more complex (required, default values,
-  groups), real demand is rare on Tarantool.
+- **proto2 `extend` / `extensions` / `group` / `MessageSet`** — the
+  conformance-only edges. Core proto2 (required, optional, custom
+  defaults, packed semantics) is in; see M9.
 - **Protobuf editions** beyond what proto3 enables — see §4.6 for
   the per-feature analysis (what we'd gain, what it would cost, the
   conditions that would force a revisit).

@@ 1,7 1,7 @@
 # tarantool-protobuf
 
-A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto3) and
-gRPC service stubs from [Tarantool](https://www.tarantool.io/).
+A `protoc` plugin and pure-Lua runtime for using Protocol Buffers (proto2 +
+proto3) and gRPC service stubs from [Tarantool](https://www.tarantool.io/).
 
 Tarantool ships an in-tree `require('protobuf')` module, but it is
 **encode-only** and has no support for `map`, `oneof`, services, or any decode


@@ 50,7 50,12 @@ both the binary+JSON and text-format suites passes.
 | Text format (`pb.text.encode` / `pb.text.decode`) | ✅ |
 | Unknown-field passthrough (`_unknown_fields`)    | ✅ |
 | Microbenchmark + alloc regression gate (`just bench`) | ✅ |
-| proto2 / editions                | ❌ deferred (separate slice; see [docs/codegen.md](docs/codegen.md)) |
+| proto2 `required` / `optional` / custom `[default = X]` | ✅ |
+| proto2 repeated unpacked-by-default + `[packed = true]` | ✅ |
+| proto2 extensions / `extend` blocks  | ❌ (passes through as unknown fields) |
+| proto2 groups (legacy)               | ❌ (parse rejected; no wire-type 3/4 codec) |
+| Google conformance suite — proto2    | ❌ blocked on groups + extensions + MessageSet |
+| Editions                             | ❌ deferred |
 
 ## Install
 


@@ 144,10 149,13 @@ done
 protoc -I proto --tarantool_out=prefix=myapp.proto:out $(find proto -name '*.proto')
 ```
 
-One pitfall: this plugin is proto3-only. If an upstream schema mixes
-proto2 and proto3, either skip the proto2 files (provided your proto3
-side doesn't import them) or wait for proto2 support — see PLAN.md's
-deferred non-goals.
+Proto2 sources work for the core surface — `required`, `optional`,
+custom `[default = X]`, and the proto2 unpacked-by-default rule for
+repeated scalars all generate correctly. The unsupported edges are
+`extend` blocks, `extensions` ranges, and legacy `group` fields;
+schemas that use any of those will fail to generate. See
+[docs/codegen.md](docs/codegen.md#proto2-support) for the field-by-field
+mapping.
 
 [strip]: https://git.srht.bigb.es/tarantool-etcd/tree/master/item/proto/_strip_annotations.py
 


@@ 164,8 172,10 @@ M.Foo_encode(t)        -- table -> wire bytes (string)
 M.Foo_decode(b)        -- wire bytes (string) -> table
 M.Foo_decode_lazy(b)   -- wire bytes -> MessageView (zero-copy view)
 M.Foo_text(t, opts)    -- table -> protoc-style text format (debug printer)
-M.Foo_has_<field>(t)   -- only emitted for proto3 explicit-`optional` fields
+M.Foo_has_<field>(t)   -- emitted for every presence-tracked field
 M.Foo_clear_<field>(t) -- same
+                       -- (proto3 explicit `optional`, proto2 `optional`,
+                       --  oneof branches)
 M.Foo_fields           -- strict {field = "field", ...} for lazy-view callers
 M.Foo_oneofs           -- strict {oneof_group = "oneof_group", ...}, when any
 ```


@@ 354,9 364,13 @@ Current baseline (2026-05-16, protobuf v34.1):
 | Text-format   |  **416** |   18 | 0 | 0 |
 
 The 1313 + 18 skipped tests all target
-`protobuf_test_messages.proto2.TestAllTypesProto2`, which we don't generate
-Lua for. Proto2 support is a separate slice — see
-[docs/codegen.md#proto2-deferral](docs/codegen.md) for what it would take.
+`protobuf_test_messages.proto2.TestAllTypesProto2`. Proto2 codegen now
+works for the core feature set (see Status table), but the conformance
+schema relies on extensions (`extend`), `extensions` ranges, legacy
+`group` fields, and a `MessageSetCorrect` that protoreflect rejects
+outright as a removed proto1 feature. Until those are wired up the
+proto2 suite stays at zero — see
+[docs/codegen.md#proto2-support](docs/codegen.md) for the scope.
 
 The runner supports `protobuf_test_messages.proto3.TestAllTypesProto3` in
 binary, JSON, and text-format input/output, including the

@@ 26,9 26,12 @@ CodeGeneratorRequest (proto descriptor bytes on stdin)
 one `<lua_pkg>.lua` per input `.proto`
 ```
 
-The plugin only handles `syntax = "proto3"`. Proto2 input is rejected
-at the top of `GenerateFile`. See [proto2 deferral](#proto2-deferral)
-for the rationale.
+The plugin accepts both `syntax = "proto3"` and `syntax = "proto2"`.
+The descriptor surface and codegen branch on syntax-aware protoreflect
+APIs (`Cardinality`, `HasDefault`, `IsPacked`) rather than maintaining
+two parallel pipelines. See [proto2 support](#proto2-support) for the
+field-by-field mapping and the (small) list of features that still
+fail to generate.
 
 ## CLI options
 


@@ 123,7 126,8 @@ The shape consumed by the runtime is:
         {name='y', id=2, kind='message', message=<other_descriptor>},
         {name='z', id=3, kind='enum',    enum=<enum_descriptor>},
         {name='m', id=4, kind='map',     key=<sub_field>, value=<sub_field>},
-        -- modifiers: repeated, packed, oneof, optional
+        -- modifiers: repeated, packed, oneof, optional,
+        --            required (proto2),  default_value (proto2)
     },
     -- Filled in by pb.finalize_message:
     field_by_id   = {[1]=<field>, ...},


@@ 342,52 346,54 @@ the same `CodeGeneratorRequest` and emits one Markdown file per input
 `examples/docs/` via `just gen-docs`. It's deliberately separate from
 the Lua codegen to keep the codegen plugin small.
 
-## Proto2 deferral
-
-The plugin rejects `syntax = "proto2"` outright. The conformance
-suites' 1331 skipped tests are all `TestAllTypesProto2`.
-
-Proto2 support is genuinely a separate slice. The wire format is
-identical to proto3 (with the one exception of `group`s — wire-types 3
-and 4, SGROUP/EGROUP, which we already tolerate on the skip path but
-don't encode/decode). The schema features that *don't* exist in proto3
-are what would need work:
-
-- **`required` fields** — codec must error on missing required at both
-  encode and decode (we currently treat absence as default).
-- **`optional` everywhere** — proto2 fields are presence-tracked by
-  default. proto3's explicit-optional path already handles this; just
-  needs the plugin to mark every proto2 field as `optional=true`.
-- **Extensions** (`extensions 100 to 199;` + `extend Foo {...}`)
-  — proto2-only mechanism for adding fields to messages defined
-  elsewhere. Round-trip overlaps with our existing
-  `_unknown_fields` machinery but explicit support needs a separate
-  descriptor table per extension and a way to expose it on user
-  messages.
-- **Groups** (`optional group Foo = 1 { ... }`) — actual encode/decode
-  rather than the skip-only tolerance we have today, plus a different
-  text-format rendering (`Foo { ... }` with the group's capitalized
-  name instead of the submessage's field name).
-- **Custom defaults** (`[default = X]`) — only matter when callers
-  query "what's the default value of field X"; for round-trips they
-  can be ignored.
-- **`MessageSet`** wire format — a few specific tests.
-
-Realistic phasing (if you take this on):
-
-1. Drop the proto3-only gate in the plugin; mark proto2 fields as
-   `optional=true`; skip extensions / groups / required harmlessly
-   (warn but emit the rest of the message). Re-run conformance —
-   should unlock most of the 1007 binary tests and ~200 of the JSON
-   tests immediately because the wire format is the same.
-2. Implement group encode/decode + the text-format group rendering.
-   Picks up the remaining text tests and a handful of binary tests.
-3. Required-field validation. Small, targeted at the
-   `TestAllRequiredTypesProto2` cases.
-4. Extensions, MessageSet, custom defaults — long tail, optional
-   depending on demand.
-
-This is a meaningfully larger slice than the typical addition here —
-plan on a few sessions, not an evening. The trade-off is "proto2 is
-mostly legacy" (see the README's status table for who actually uses
-it) vs "1331 extra ✓ in the conformance numbers looks good".
+## Proto2 support
+
+The plugin and runtime now accept `syntax = "proto2"`. The descriptor
+table is the same shape as proto3 — three new attributes carry the
+proto2-only semantics:
+
+| Attribute | Meaning | Set on |
+|---|---|---|
+| `required = true` | Field declared with the proto2 `required` keyword. Encoder errors if the value is missing on the input table; no default elision. | Singular non-oneof fields with `required` cardinality. |
+| `default_value = …` | Lua literal for `[default = X]`. 64-bit ints arrive as cdata; enums use the symbolic name. | Any singular field with an explicit default. |
+| `optional = true` | Already used for proto3 explicit-optional; in proto2 it ends up on every singular field with the `optional` keyword (which is most of them). | All proto2 fields declared `optional`. |
+
+The wire format itself is unchanged between syntaxes — `IsPacked()`
+from protoreflect already produces the right packed-default per syntax,
+so repeated proto2 scalars stay unpacked unless they carry
+`[packed = true]`. The codec's required-writer specialization fires
+through `codec.compile_writers`, which dynamic.lua now calls explicitly
+(so source-parsed schemas behave the same as build-time codegen).
+
+`test/proto/proto2_basic.proto` exercises the surface; the
+`proto2_basic.*` luatest groups (40 cases) cover defaults, required
+validation, the unpacked-by-default rule, nested-required messages,
+and full-vs-runtime-vs-dynamic byte parity.
+
+### What still doesn't work
+
+The conformance schema (`test_messages_proto2.proto` from upstream
+protobuf) leans on three features we don't implement:
+
+- **`extend` blocks** + **`extensions 100 to 199;`** ranges — proto2's
+  open-message-extension mechanism. Extension values currently
+  round-trip through `_unknown_fields` like any other unknown tag,
+  so a message carrying an extension can be decoded → re-encoded
+  byte-identically. But the plugin won't emit accessors and there's
+  no way to read or set extension values from user code.
+- **`group`** fields (legacy `optional group Foo = 1 { … }`) — uses
+  wire types 3 (SGROUP) and 4 (EGROUP) which our wire layer doesn't
+  encode or decode. We skip-tolerate them on the read side but
+  generating a message that *contains* a group field fails because
+  `protoreflect.GroupKind` falls off our `scalarName` switch.
+- **`MessageSet`** wire format — `protobuf-go` rejects messages
+  declaring `option message_set_wire_format = true;` at the
+  protoreflect layer ("a legacy proto1 feature that is no longer
+  supported"). We can't bypass this without forking protoreflect.
+
+Implementing extensions and groups is the realistic path to closing
+the conformance gap; MessageSet would need an upstream workaround.
+Both are big enough lifts to be separate slices — proto2 is mostly
+legacy and the existing extension-via-unknown-fields path covers the
+common interop case (you receive bytes, you re-encode them, the
+extension data passes through intact).