# Codegen notes

How `protoc-gen-tarantool` works, what it emits, and where to extend it.

This is a companion to [docs/api-modes.md](api-modes.md). API-modes
covers the *user-facing* difference between full / runtime / lazy; this
doc covers what the plugin does to produce the code those modes rely on.

## The pipeline

```
.proto sources
    │
    ▼  protoc (mainline binary, host-installed)
CodeGeneratorRequest (proto descriptor bytes on stdin)
    │
    ▼  cmd/protoc-gen-tarantool/main.go
    │     advertises FEATURE_PROTO3_OPTIONAL so protoc surfaces
    │     explicit-optional fields (without this flag, it omits them)
    │
    ▼  cmd/protoc-gen-tarantool/internal/gen.GenerateFile
    │
    ├─ ModeFull    → inline `_encode` / `_decode` (the default)
    └─ ModeRuntime → wrappers that delegate to pb.encode / pb.decode

one `<lua_pkg>.lua` per input `.proto`
```

The plugin accepts both `syntax = "proto3"` and `syntax = "proto2"`.
The descriptor surface and codegen branch on syntax-aware protoreflect
APIs (`Cardinality`, `HasDefault`, `IsPacked`) rather than maintaining
two parallel pipelines. See [proto2 support](#proto2-support) for the
field-by-field mapping and the (small) list of features that still
fail to generate.

## CLI options

Passed via `protoc --tarantool_opt=<key>=<value>,...`:

| Option | Values | Meaning |
|---|---|---|
| `mode` | `full` (default) or `runtime` | Inline `_encode`/`_decode` vs descriptor-delegating wrappers. See [api-modes.md](api-modes.md). |
| `prefix` | any Lua-require path | Prepended to every generated module's require path **and** its on-disk subpath. The Justfile uses this to generate both modes side-by-side into `examples/expected/{full,runtime}/`. |

The `(tarantool.lua_package)` file option (defined in
`options/tarantool/tarantool.proto`) overrides the per-file Lua module
path:

```proto
import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";
```

Without that option the path mirrors `package` (e.g.
`package my.app; foo.proto` → `my/app/foo_pb.lua`,
`require('my.app.foo_pb')`).

## What gets emitted per .proto

```lua
-- Code generated by protoc-gen-tarantool. DO NOT EDIT.
-- source: hello.proto
-- syntax: proto3

local pb = require('pb')
local wire = pb.wire

local M = {}

-- 1. Enums first (no forward-reference problems).
M.Status_descriptor = pb.enum('hello.Status', { ACTIVE=0, BANNED=1, ... })
M.Status            = M.Status_descriptor.by_name

-- 2. Predeclare all message descriptor tables. Self-references and
--    mutual recursion resolve here because every descriptor is named
--    before any field references it.
M.Person_descriptor = {name = 'hello.Person'}
M.Address_descriptor = {name = 'hello.Address'}
M.Result_descriptor = {name = 'hello.Result'}

-- 3. Fill in fields[] per message, then call pb.finalize_message.
M.Person_descriptor.fields = {
    {name='name',    id=1, kind='scalar', proto_type='string'},
    {name='user_id', id=2, kind='scalar', proto_type='uint64'},
    -- ...
}
M.Person_descriptor.oneofs        = { outcome = {'text', 'code', 'details'} }
M.Person_descriptor.reserved_names = { ['old_field'] = true }
pb.finalize_message(M.Person_descriptor)

-- 4. EmmyLua / lua-language-server type annotations.
---@class hello.Person
---@field name string
---@field user_id integer
-- ...

-- 5. Wrappers: _new / _encode / _decode / _decode_lazy / _text
--    + _has_<field> / _clear_<field> for explicit-optional fields.
function M.Person_new(t) return t or {} end
function M.Person_encode(t) -- inline body in full mode, delegation in runtime mode
function M.Person_decode(b) -- same
function M.Person_decode_lazy(b) return pb.decode_lazy(M.Person_descriptor, b) end
function M.Person_text(t, opts) return pb.text.encode(M.Person_descriptor, t, opts) end

-- 6. Services (mode-independent — they reuse the per-message wrappers).
M.Greeter_service = { ... }
function M.Greeter_client(transport) ... end
function M.Greeter_server(impl) ... end

return M
```

Generation order matters: enums and predeclared descriptors come before
field tables so cross-references inside `fields = {...}` resolve in one
pass.

## The descriptor table — the contract

The shape consumed by the runtime is:

```lua
{
    name = 'pkg.Foo',
    fields = {
        {name='x', id=1, kind='scalar', proto_type='int32'},
        {name='y', id=2, kind='message', message=<other_descriptor>},
        {name='z', id=3, kind='enum',    enum=<enum_descriptor>},
        {name='m', id=4, kind='map',     key=<sub_field>, value=<sub_field>},
        {name='g', id=5, kind='group',   message=<other_descriptor>},
        -- modifiers: repeated, packed, oneof, optional,
        --            required (proto2),  default_value (proto2)
    },
    -- Proto2-only, optional:
    extensions_by_id        = {[N] = <ext_field>, ...},
    extensions_by_full_name = {['pkg.ext_name'] = <ext_field>, ...},
    -- Filled in by pb.finalize_message:
    field_by_id   = {[1]=<field>, ...},
    field_by_name = {x=<field>, ...},
    oneofs_list   = {{name=<n>, members={<names>}}, ...},
    -- Optional, set by the codegen:
    oneofs         = {<name> = {<member-names>}},   -- input form, flattened above
    reserved_names = {[<name>] = true},              -- for text-format parser
    encode = <fn>,  decode = <fn>,                   -- override hook (WKT use this)
    text   = <fn>,                                   -- text-format encode override
}
```

**Four producers emit this exact shape.** They live in different
codepaths but consume the same runtime:

1. `cmd/protoc-gen-tarantool/internal/gen` — build-time codegen.
2. `runtime/pb/dynamic.lua` — runtime synthesis from an AST that
   `runtime/pb/parser.lua` produces from `.proto` source
   (`pb.parse(text)`).
3. `runtime/pb/fileset.lua` — runtime synthesis from a
   `FileDescriptorSet` binary (`pb.from_pb(bytes)`;
   produced by `protoc --descriptor_set_out=...`).
4. `runtime/pb/wkt.lua` — hand-rolled WKT descriptors. Same shape but
   additionally set `desc.encode` / `desc.decode` (and sometimes
   `desc.text`, `desc.json_encode` / `desc.json_decode`) to take over
   the wire path entirely. Covers Timestamp, Duration, Wrappers, Struct,
   Value, ListValue, Any, FieldMask, Empty.

**Pinning this shape is what lets us run every behavior test against
both codegen modes.** The test harness in `test/protobuf_test.lua` and
friends parametrizes by mode (`{'full', 'runtime'}`) and runs the same
assertions against `require('full.hello.hello_pb')` and
`require('runtime.hello.hello_pb')`.

## Inline (full) mode — what it looks like

`inline.go` walks each field and emits Lua statements that call into
`runtime/pb/wire.lua` primitives directly. Tag bytes are precomputed at
codegen time as Lua string literals so the runtime never re-encodes
them.

```lua
function M.Person_encode(t)
    if type(t) ~= 'table' then
        error("expected table for hello.Person, got " .. type(t), 0)
    end
    local out, n = {}, 0
    local v

    -- field 1: name (string, singular)
    v = t.name
    if v ~= nil and v ~= '' then               -- proto3 default elision
        n = n + 1; out[n] = "\x0a"             -- tag(1, LEN), precomputed
        n = n + 1; out[n] = wire.encode_varint(#v)
        n = n + 1; out[n] = v
    end

    -- field 11: weight_kg (double, singular)
    v = t.weight_kg
    if v ~= nil and (v ~= 0 or 1/v == -math.huge) then  -- -0 preserved
        n = n + 1; out[n] = "\x59"
        n = n + 1; out[n] = wire.encode_double(v)
    end

    -- ... oneof pre-pass picks active branch; map fields walk pairs(); etc.

    return table.concat(out)
end
```

The decoder is similarly inlined:

```lua
function M.Person_decode(b)
    local result = {}
    local pos, len = 1, #b
    while pos <= len do
        local id, wt
        id, wt, pos = wire.decode_tag(b, pos)
        if id == 1 then                         -- name
            result.name, pos = wire.decode_string(b, pos)
        elseif id == 2 then                     -- user_id
            result.user_id, pos = wire.decode_uint64(b, pos)
        -- ... if-elseif chain on field id ...
        else
            pos = wire.skip_field(b, pos, wt, id)   -- unknown
        end
    end
    return result
end
```

**Why an `if-elseif` chain rather than a dispatch table?** LuaJIT
compiles short chains into a tight branch sequence on a single trace.
The crossover where a jump table would win is at field counts higher
than anything we hit in practice; pinned by `bench/jit_trace.lua`.

## Runtime mode — the thin wrappers

`mode=runtime` skips the inline emission and produces:

```lua
function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end
```

`pb.encode` / `pb.decode` (in `runtime/pb/codec.lua`) walk the
descriptor table to pick wire primitives at runtime. Slower per
field — a hash lookup + closure call instead of a hard-coded branch —
but smaller generated output (no per-message `_encode` / `_decode`
bodies) and useful when the same code path needs to handle dynamic
descriptors.

`pb.finalize_message` (in `runtime/pb/init.lua`) builds two
specialization layers on top of the descriptor at finalize-time:

1. **Per-field writer closures** (`f._writer`): pre-bound to the
   field's tag, encoder, and default predicate. The encode loop calls
   `f._writer(data, out)` per field instead of dispatching on
   `f.kind` / `f.proto_type` inside the loop.
2. **Per-field reader closures** (`f._reader`): same idea for decode,
   handling typed value extraction + repeated bookkeeping + nested
   merge + oneof sibling clearing in one closure per field.

These specializations bridge most of the gap between runtime mode and
full mode for the *decoder*; encode is still meaningfully slower in
runtime mode because the field walk goes through one extra Lua-function
boundary per field.

## The hot-path rules the generated code observes

These come from CLAUDE.md and are enforced across both the codegen and
the runtime. New code (codegen output or hand-written) needs to follow
them or the LuaJIT trace compiler bails out.

- **No `pairs()` on hot paths.** `pairs()` over a hash compiles to
  bytecode `ISNEXT`, which is NYI in Tarantool's LuaJIT 2.1 fork —
  the trace aborts and the code path falls off the JIT. Map fields are
  the one accepted exception; the JIT-trace gate in `bench/jit_trace.lua`
  pins the limitation. When a descriptor needs hash-shaped lookup state
  (e.g. oneof grouping), build a parallel array view at
  `pb.finalize_message` time (`desc.oneofs_list`) and iterate the array
  on the hot path.
- **64-bit integers stay as cdata.** `int64`/`uint64`/`sint64`/
  `fixed64`/`sfixed64` are LuaJIT `int64_t` / `uint64_t` cdata
  everywhere. Same convention as Tarantool's `msgpackffi`, `net.box`,
  `box.tuple`, and built-in `protobuf`. Don't narrow through
  `tonumber` — precision loss past 2^53 silently corrupts IDs and
  timestamp nanos.
- **`ipairs` / `for i=1,#t`, never `pairs()`, on generated arrays.**
  Repeated fields are 1-based contiguous Lua arrays. Generated encode
  walks them with `for i = 1, #v do`.
- **Keep hot helpers small.** Adding "more fast paths" to a helper that
  the JIT already inlines (e.g. extending `encode_varint`'s 1-byte
  fast path with 2/3/4-byte branches) can push the function past the
  LuaJIT inline budget — parent traces stop inlining it and the
  dominant 1-byte case regresses by ~30%. Multi-byte fast paths live
  in the *slow* helper, not the inlined one.
- **SoA over AoS for many-entry index structures.** For per-element
  records where N is in the thousands, use parallel int arrays, not
  one small Lua table per entry. Each per-entry table is a separate
  allocation with header overhead and hash dispatch; for large N the
  allocations dominate. See `runtime/pb/lazy.lua`'s `index_bytes`
  for the reference shape.
- **WKT routing.** When codegen sees a field referencing a
  `google.protobuf.*` type, it emits a reference to
  `pb.wkt.<Name>_descriptor` instead of treating it as an ordinary
  cross-file import. The plugin does NOT generate a Lua module for
  imported WKT `.proto` files — those are fulfilled by
  `runtime/pb/wkt.lua`.

## Plugin source layout

```
cmd/protoc-gen-tarantool/
  main.go              # reads CodeGeneratorRequest from stdin, sets up
                       # the plugin options + FEATURE_PROTO3_OPTIONAL,
                       # dispatches to gen.GenerateFile per .proto
  internal/gen/
    gen.go             # per-file orchestration + runtime-mode wrappers,
                       # imports collection, enum + message emission,
                       # descriptor field tables, oneofs, reserved_names
    inline.go          # full-mode inline _encode / _decode body emission,
                       # field-kind dispatch, repeated / map / oneof shapes
    service.go         # gRPC client / server factories
    name.go            # Lua name + path mangling rules
    types.go           # protoreflect.Kind → scalar name mapping
    options.go         # (tarantool.lua_package) file option lookup
    emmylua.go         # EmmyLua / lua-language-server type annotation
                       # emission (---@class, ---@field, ---@param)
```

**Adding a new wire type or scalar** means touching three places:

1. `runtime/pb/wire.lua` — add the `encode_xxx` / `decode_xxx`
   primitives + a `TYPE_INFO[xxx]` entry. That's the single source of
   truth: the codec, both codegen modes, and the dynamic parser all
   pick it up from there.
2. `cmd/protoc-gen-tarantool/internal/gen/types.go` —
   `scalarName(k)` mapping for the new `protoreflect.Kind`.
3. `cmd/protoc-gen-tarantool/internal/gen/inline.go` — emission for
   the new type (singular + repeated + packed branches).

If the new type is `LEN`-typed (string/bytes-shaped), check the
existing string/bytes paths for the split-emission pattern
(`tag + varint(#v) + body` as three separate `out` slots instead of
one concatenated `wire.encode_len(v)` call) — that avoids a per-field
string allocation.

## Sibling plugin: protoc-gen-tarantool-doc

`cmd/protoc-gen-tarantool-doc/` is a second Go plugin that consumes
the same `CodeGeneratorRequest` and emits one Markdown file per input
`.proto`. Build with `just build-doc`; sample docs land in
`examples/docs/` via `just gen-docs`. It's deliberately separate from
the Lua codegen to keep the codegen plugin small.

## Proto2 support

The plugin and runtime now accept `syntax = "proto2"`. The descriptor
table is the same shape as proto3 — three new attributes carry the
proto2-only semantics:

| Attribute | Meaning | Set on |
|---|---|---|
| `required = true` | Field declared with the proto2 `required` keyword. Encoder errors if the value is missing on the input table; no default elision. | Singular non-oneof fields with `required` cardinality. |
| `default_value = …` | Lua literal for `[default = X]`. 64-bit ints arrive as cdata; enums use the symbolic name. | Any singular field with an explicit default. |
| `optional = true` | Already used for proto3 explicit-optional; in proto2 it ends up on every singular field with the `optional` keyword (which is most of them). | All proto2 fields declared `optional`. |

The wire format itself is unchanged between syntaxes — `IsPacked()`
from protoreflect already produces the right packed-default per syntax,
so repeated proto2 scalars stay unpacked unless they carry
`[packed = true]`. The codec's required-writer specialization fires
through `codec.compile_writers`, which dynamic.lua now calls explicitly
(so source-parsed schemas behave the same as build-time codegen).

`test/proto/proto2_basic.proto` exercises the surface; the
`proto2_basic.*` luatest groups (40 cases) cover defaults, required
validation, the unpacked-by-default rule, nested-required messages,
and full-vs-runtime-vs-dynamic byte parity.

### Groups

Proto2 `group` fields (legacy `optional group Foo = 1 { … }`) carry
their own wire format: SGROUP (wire type 3) opens, the body follows
as regular field encodings, and EGROUP (4) with the matching field
id closes. No length prefix. The codec exposes
`pb.codec.decode_group(desc, buf, pos, stop_id)` which reuses the
per-tag dispatch from `decode_message` but bails on EGROUP instead
of end-of-buffer. Inline codegen emits a single `kind='group'` field
that brackets its body with start/end tags; repeated groups wrap
each element in its own pair.

Text format uses the group's submessage simple name as the label
(`Data { … }`, `MultiWordGroupField { … }`) and accepts the
lowercase ASCII fold (`data { … }`); the `:` between label and `{`
is optional, matching mainline protoc.

### Extensions

`extend Foo { optional Bar baz = N; }` declarations register an
extra tag on `Foo`. The plugin emits
`pb.register_extension(M.Foo_descriptor, {name='baz', full_name='pkg.baz', id=N, …})`
calls at module load time. The runtime attaches two indices to the
extendee — `extensions_by_id` (decode lookup) and
`extensions_by_full_name` (encode + textual emission order).

Set extension values live on a message under `_extensions`:

```lua
local m = {_extensions = {['pkg.baz'] = {nested = 7}}}
local bytes = M.Foo_encode(m)
local back = M.Foo_decode(bytes)
back._extensions['pkg.baz']  -- {nested = 7}
```

Wire bytes for set extensions interleave with regular fields (the
codec walks `extensions_by_full_name` after the field loop on encode
and routes unknown tags to `decode_extension` on the way in). Text
format speaks the bracketed `[pkg.baz]: value` (or `[pkg.baz] { … }`
for message/group extensions). JSON uses the same bracketed key
form per the proto2 JSON spec. Extensions extending
`google.protobuf.*` descriptors (file/message/field options) are
skipped at codegen — those are meta-only and the WKT module doesn't
surface their descriptors at runtime.

### Closed enums

Proto2 enums are closed: a numeric literal that doesn't match any
declared value is a parse error in text format (and on JSON via the
same path). The plugin emits `closed = true` on every proto2 enum
descriptor (driven by protoreflect's `IsClosed()`). Proto3 enums
stay open to preserve forward-compatibility on the wire.

### What still doesn't work

- **`MessageSet`** wire format. `protobuf-go`'s protoreflect refuses
  to load a FileDescriptor declaring `option message_set_wire_format
  = true;`, calling it "a legacy proto1 feature that is no longer
  supported". Our vendored copy of
  `test/conformance/proto/test_messages_proto2.proto` has the four
  MessageSet-flavored nested messages stripped so the rest of the
  schema compiles — the patch is documented in the file's header.
  Real MessageSet support would need to fork protoreflect; deferred
  until a Tarantool consumer asks.