# Codegen notes

How `protoc-gen-tarantool` works, what it emits, and where to extend it.

This is a companion to [docs/api-modes.md](api-modes.md). API-modes
covers the *user-facing* difference between full / runtime / lazy; this
doc covers what the plugin does to produce the code those modes rely on.

## The pipeline

```
.proto sources
    │
    ▼  protoc (mainline binary, host-installed)
CodeGeneratorRequest (proto descriptor bytes on stdin)
    │
    ▼  cmd/protoc-gen-tarantool/main.go
    │     advertises FEATURE_PROTO3_OPTIONAL so protoc surfaces
    │     explicit-optional fields (without this flag, it omits them)
    │
    ▼  cmd/protoc-gen-tarantool/internal/gen.GenerateFile
    │
    ├─ ModeFull    → inline `_encode` / `_decode` (the default)
    └─ ModeRuntime → wrappers that delegate to pb.encode / pb.decode

one `<lua_pkg>.lua` per input `.proto`
```

The plugin accepts both `syntax = "proto3"` and `syntax = "proto2"`.
The descriptor surface and codegen branch on syntax-aware protoreflect
APIs (`Cardinality`, `HasDefault`, `IsPacked`) rather than maintaining
two parallel pipelines. See [proto2 support](#proto2-support) for the
field-by-field mapping and the (small) list of features that still
fail to generate.

## CLI options

Passed via `protoc --tarantool_opt=<key>=<value>,...`:

| Option | Values | Meaning |
|---|---|---|
| `mode` | `full` (default) or `runtime` | Inline `_encode`/`_decode` vs descriptor-delegating wrappers. See [api-modes.md](api-modes.md). |
| `prefix` | any Lua-require path | Prepended to every generated module's require path **and** its on-disk subpath. The Justfile uses this to generate both modes side-by-side into `examples/expected/{full,runtime}/`. |

The `(tarantool.lua_package)` file option (defined in
`options/tarantool/tarantool.proto`) overrides the per-file Lua module
path:

```proto
import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";
```

Without that option the path mirrors `package` (e.g.
`package my.app; foo.proto` → `my/app/foo_pb.lua`,
`require('my.app.foo_pb')`).

## What gets emitted per .proto

```lua
-- Code generated by protoc-gen-tarantool. DO NOT EDIT.
-- source: hello.proto
-- syntax: proto3

local pb = require('pb')
local wire = pb.wire

local M = {}

-- 1. Enums first (no forward-reference problems).
M.Status_descriptor = pb.enum('hello.Status', { ACTIVE=0, BANNED=1, ... })
M.Status            = M.Status_descriptor.by_name

-- 2. Predeclare all message descriptor tables. Self-references and
--    mutual recursion resolve here because every descriptor is named
--    before any field references it.
M.Person_descriptor = {name = 'hello.Person'}
M.Address_descriptor = {name = 'hello.Address'}
M.Result_descriptor = {name = 'hello.Result'}

-- 3. Fill in fields[] per message, then call pb.finalize_message.
M.Person_descriptor.fields = {
    {name='name',    id=1, kind='scalar', proto_type='string'},
    {name='user_id', id=2, kind='scalar', proto_type='uint64'},
    -- ...
}
M.Person_descriptor.oneofs        = { outcome = {'text', 'code', 'details'} }
M.Person_descriptor.reserved_names = { ['old_field'] = true }
pb.finalize_message(M.Person_descriptor)

-- 4. EmmyLua / lua-language-server type annotations.
---@class hello.Person
---@field name string
---@field user_id integer
-- ...

-- 5. Wrappers: _new / _encode / _decode / _decode_lazy / _text
--    + _has_<field> / _clear_<field> for explicit-optional fields.
function M.Person_new(t) return t or {} end
function M.Person_encode(t) -- inline body in full mode, delegation in runtime mode
function M.Person_decode(b) -- same
function M.Person_decode_lazy(b) return pb.decode_lazy(M.Person_descriptor, b) end
function M.Person_text(t, opts) return pb.text.encode(M.Person_descriptor, t, opts) end

-- 6. Services (mode-independent — they reuse the per-message wrappers).
M.Greeter_service = { ... }
function M.Greeter_client(transport) ... end
function M.Greeter_server(impl) ... end

return M
```

Generation order matters: enums and predeclared descriptors come before
field tables so cross-references inside `fields = {...}` resolve in one
pass.

## The descriptor table — the contract

The shape consumed by the runtime is:

```lua
{
    name = 'pkg.Foo',
    fields = {
        {name='x', id=1, kind='scalar', proto_type='int32'},
        {name='y', id=2, kind='message', message=<other_descriptor>},
        {name='z', id=3, kind='enum',    enum=<enum_descriptor>},
        {name='m', id=4, kind='map',     key=<sub_field>, value=<sub_field>},
        -- modifiers: repeated, packed, oneof, optional,
        --            required (proto2),  default_value (proto2)
    },
    -- Filled in by pb.finalize_message:
    field_by_id   = {[1]=<field>, ...},
    field_by_name = {x=<field>, ...},
    oneofs_list   = {{name=<n>, members={<names>}}, ...},
    -- Optional, set by the codegen:
    oneofs         = {<name> = {<member-names>}},   -- input form, flattened above
    reserved_names = {[<name>] = true},              -- for text-format parser
    encode = <fn>,  decode = <fn>,                   -- override hook (WKT use this)
    text   = <fn>,                                   -- text-format encode override
}
```

**Four producers emit this exact shape.** They live in different
codepaths but consume the same runtime:

1. `cmd/protoc-gen-tarantool/internal/gen` — build-time codegen.
2. `runtime/pb/dynamic.lua` — runtime synthesis from an AST that
   `runtime/pb/parser.lua` produces from `.proto` source
   (`pb.parse(text)`).
3. `runtime/pb/fileset.lua` — runtime synthesis from a
   `FileDescriptorSet` binary (`pb.from_pb(bytes)`;
   produced by `protoc --descriptor_set_out=...`).
4. `runtime/pb/wkt.lua` — hand-rolled WKT descriptors. Same shape but
   additionally set `desc.encode` / `desc.decode` (and sometimes
   `desc.text`, `desc.json_encode` / `desc.json_decode`) to take over
   the wire path entirely. Covers Timestamp, Duration, Wrappers, Struct,
   Value, ListValue, Any, FieldMask, Empty.

**Pinning this shape is what lets us run every behavior test against
both codegen modes.** The test harness in `test/protobuf_test.lua` and
friends parametrizes by mode (`{'full', 'runtime'}`) and runs the same
assertions against `require('full.hello.hello_pb')` and
`require('runtime.hello.hello_pb')`.

## Inline (full) mode — what it looks like

`inline.go` walks each field and emits Lua statements that call into
`runtime/pb/wire.lua` primitives directly. Tag bytes are precomputed at
codegen time as Lua string literals so the runtime never re-encodes
them.

```lua
function M.Person_encode(t)
    if type(t) ~= 'table' then
        error("expected table for hello.Person, got " .. type(t), 0)
    end
    local out, n = {}, 0
    local v

    -- field 1: name (string, singular)
    v = t.name
    if v ~= nil and v ~= '' then               -- proto3 default elision
        n = n + 1; out[n] = "\x0a"             -- tag(1, LEN), precomputed
        n = n + 1; out[n] = wire.encode_varint(#v)
        n = n + 1; out[n] = v
    end

    -- field 11: weight_kg (double, singular)
    v = t.weight_kg
    if v ~= nil and (v ~= 0 or 1/v == -math.huge) then  -- -0 preserved
        n = n + 1; out[n] = "\x59"
        n = n + 1; out[n] = wire.encode_double(v)
    end

    -- ... oneof pre-pass picks active branch; map fields walk pairs(); etc.

    return table.concat(out)
end
```

The decoder is similarly inlined:

```lua
function M.Person_decode(b)
    local result = {}
    local pos, len = 1, #b
    while pos <= len do
        local id, wt
        id, wt, pos = wire.decode_tag(b, pos)
        if id == 1 then                         -- name
            result.name, pos = wire.decode_string(b, pos)
        elseif id == 2 then                     -- user_id
            result.user_id, pos = wire.decode_uint64(b, pos)
        -- ... if-elseif chain on field id ...
        else
            pos = wire.skip_field(b, pos, wt, id)   -- unknown
        end
    end
    return result
end
```

**Why an `if-elseif` chain rather than a dispatch table?** LuaJIT
compiles short chains into a tight branch sequence on a single trace.
The crossover where a jump table would win is at field counts higher
than anything we hit in practice; pinned by `bench/jit_trace.lua`.

## Runtime mode — the thin wrappers

`mode=runtime` skips the inline emission and produces:

```lua
function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end
```

`pb.encode` / `pb.decode` (in `runtime/pb/codec.lua`) walk the
descriptor table to pick wire primitives at runtime. Slower per
field — a hash lookup + closure call instead of a hard-coded branch —
but smaller generated output (no per-message `_encode` / `_decode`
bodies) and useful when the same code path needs to handle dynamic
descriptors.

`pb.finalize_message` (in `runtime/pb/init.lua`) builds two
specialization layers on top of the descriptor at finalize-time:

1. **Per-field writer closures** (`f._writer`): pre-bound to the
   field's tag, encoder, and default predicate. The encode loop calls
   `f._writer(data, out)` per field instead of dispatching on
   `f.kind` / `f.proto_type` inside the loop.
2. **Per-field reader closures** (`f._reader`): same idea for decode,
   handling typed value extraction + repeated bookkeeping + nested
   merge + oneof sibling clearing in one closure per field.

These specializations bridge most of the gap between runtime mode and
full mode for the *decoder*; encode is still meaningfully slower in
runtime mode because the field walk goes through one extra Lua-function
boundary per field.

## The hot-path rules the generated code observes

These come from CLAUDE.md and are enforced across both the codegen and
the runtime. New code (codegen output or hand-written) needs to follow
them or the LuaJIT trace compiler bails out.

- **No `pairs()` on hot paths.** `pairs()` over a hash compiles to
  bytecode `ISNEXT`, which is NYI in Tarantool's LuaJIT 2.1 fork —
  the trace aborts and the code path falls off the JIT. Map fields are
  the one accepted exception; the JIT-trace gate in `bench/jit_trace.lua`
  pins the limitation. When a descriptor needs hash-shaped lookup state
  (e.g. oneof grouping), build a parallel array view at
  `pb.finalize_message` time (`desc.oneofs_list`) and iterate the array
  on the hot path.
- **64-bit integers stay as cdata.** `int64`/`uint64`/`sint64`/
  `fixed64`/`sfixed64` are LuaJIT `int64_t` / `uint64_t` cdata
  everywhere. Same convention as Tarantool's `msgpackffi`, `net.box`,
  `box.tuple`, and built-in `protobuf`. Don't narrow through
  `tonumber` — precision loss past 2^53 silently corrupts IDs and
  timestamp nanos.
- **`ipairs` / `for i=1,#t`, never `pairs()`, on generated arrays.**
  Repeated fields are 1-based contiguous Lua arrays. Generated encode
  walks them with `for i = 1, #v do`.
- **Keep hot helpers small.** Adding "more fast paths" to a helper that
  the JIT already inlines (e.g. extending `encode_varint`'s 1-byte
  fast path with 2/3/4-byte branches) can push the function past the
  LuaJIT inline budget — parent traces stop inlining it and the
  dominant 1-byte case regresses by ~30%. Multi-byte fast paths live
  in the *slow* helper, not the inlined one.
- **SoA over AoS for many-entry index structures.** For per-element
  records where N is in the thousands, use parallel int arrays, not
  one small Lua table per entry. Each per-entry table is a separate
  allocation with header overhead and hash dispatch; for large N the
  allocations dominate. See `runtime/pb/lazy.lua`'s `index_bytes`
  for the reference shape.
- **WKT routing.** When codegen sees a field referencing a
  `google.protobuf.*` type, it emits a reference to
  `pb.wkt.<Name>_descriptor` instead of treating it as an ordinary
  cross-file import. The plugin does NOT generate a Lua module for
  imported WKT `.proto` files — those are fulfilled by
  `runtime/pb/wkt.lua`.

## Plugin source layout

```
cmd/protoc-gen-tarantool/
  main.go              # reads CodeGeneratorRequest from stdin, sets up
                       # the plugin options + FEATURE_PROTO3_OPTIONAL,
                       # dispatches to gen.GenerateFile per .proto
  internal/gen/
    gen.go             # per-file orchestration + runtime-mode wrappers,
                       # imports collection, enum + message emission,
                       # descriptor field tables, oneofs, reserved_names
    inline.go          # full-mode inline _encode / _decode body emission,
                       # field-kind dispatch, repeated / map / oneof shapes
    service.go         # gRPC client / server factories
    name.go            # Lua name + path mangling rules
    types.go           # protoreflect.Kind → scalar name mapping
    options.go         # (tarantool.lua_package) file option lookup
    emmylua.go         # EmmyLua / lua-language-server type annotation
                       # emission (---@class, ---@field, ---@param)
```

**Adding a new wire type or scalar** means touching three places:

1. `runtime/pb/wire.lua` — add the `encode_xxx` / `decode_xxx`
   primitives + a `TYPE_INFO[xxx]` entry. That's the single source of
   truth: the codec, both codegen modes, and the dynamic parser all
   pick it up from there.
2. `cmd/protoc-gen-tarantool/internal/gen/types.go` —
   `scalarName(k)` mapping for the new `protoreflect.Kind`.
3. `cmd/protoc-gen-tarantool/internal/gen/inline.go` — emission for
   the new type (singular + repeated + packed branches).

If the new type is `LEN`-typed (string/bytes-shaped), check the
existing string/bytes paths for the split-emission pattern
(`tag + varint(#v) + body` as three separate `out` slots instead of
one concatenated `wire.encode_len(v)` call) — that avoids a per-field
string allocation.

## Sibling plugin: protoc-gen-tarantool-doc

`cmd/protoc-gen-tarantool-doc/` is a second Go plugin that consumes
the same `CodeGeneratorRequest` and emits one Markdown file per input
`.proto`. Build with `just build-doc`; sample docs land in
`examples/docs/` via `just gen-docs`. It's deliberately separate from
the Lua codegen to keep the codegen plugin small.

## Proto2 support

The plugin and runtime now accept `syntax = "proto2"`. The descriptor
table is the same shape as proto3 — three new attributes carry the
proto2-only semantics:

| Attribute | Meaning | Set on |
|---|---|---|
| `required = true` | Field declared with the proto2 `required` keyword. Encoder errors if the value is missing on the input table; no default elision. | Singular non-oneof fields with `required` cardinality. |
| `default_value = …` | Lua literal for `[default = X]`. 64-bit ints arrive as cdata; enums use the symbolic name. | Any singular field with an explicit default. |
| `optional = true` | Already used for proto3 explicit-optional; in proto2 it ends up on every singular field with the `optional` keyword (which is most of them). | All proto2 fields declared `optional`. |

The wire format itself is unchanged between syntaxes — `IsPacked()`
from protoreflect already produces the right packed-default per syntax,
so repeated proto2 scalars stay unpacked unless they carry
`[packed = true]`. The codec's required-writer specialization fires
through `codec.compile_writers`, which dynamic.lua now calls explicitly
(so source-parsed schemas behave the same as build-time codegen).

`test/proto/proto2_basic.proto` exercises the surface; the
`proto2_basic.*` luatest groups (40 cases) cover defaults, required
validation, the unpacked-by-default rule, nested-required messages,
and full-vs-runtime-vs-dynamic byte parity.

### What still doesn't work

The conformance schema (`test_messages_proto2.proto` from upstream
protobuf) leans on three features we don't implement:

- **`extend` blocks** + **`extensions 100 to 199;`** ranges — proto2's
  open-message-extension mechanism. Extension values currently
  round-trip through `_unknown_fields` like any other unknown tag,
  so a message carrying an extension can be decoded → re-encoded
  byte-identically. But the plugin won't emit accessors and there's
  no way to read or set extension values from user code.
- **`group`** fields (legacy `optional group Foo = 1 { … }`) — uses
  wire types 3 (SGROUP) and 4 (EGROUP) which our wire layer doesn't
  encode or decode. We skip-tolerate them on the read side but
  generating a message that *contains* a group field fails because
  `protoreflect.GroupKind` falls off our `scalarName` switch.
- **`MessageSet`** wire format — `protobuf-go` rejects messages
  declaring `option message_set_wire_format = true;` at the
  protoreflect layer ("a legacy proto1 feature that is no longer
  supported"). We can't bypass this without forking protoreflect.

Implementing extensions and groups is the realistic path to closing
the conformance gap; MessageSet would need an upstream workaround.
Both are big enough lifts to be separate slices — proto2 is mostly
legacy and the existing extension-via-unknown-fields path covers the
common interop case (you receive bytes, you re-encode them, the
extension data passes through intact).