# Codegen notes

How `protoc-gen-tarantool` works, what it emits, and where to extend it.

This is a companion to [docs/api-modes.md](api-modes.md). API-modes
covers the *user-facing* difference between full / runtime / lazy; this
doc covers what the plugin does to produce the code those modes rely on.

## The pipeline

```
.proto sources
    │
    ▼  protoc (mainline binary, host-installed)
CodeGeneratorRequest (proto descriptor bytes on stdin)
    │
    ▼  cmd/protoc-gen-tarantool/main.go
    │     advertises FEATURE_PROTO3_OPTIONAL so protoc surfaces
    │     explicit-optional fields (without this flag, it omits them)
    │
    ▼  cmd/protoc-gen-tarantool/internal/gen.GenerateFile
    │
    ├─ ModeFull    → inline `_encode` / `_decode` (the default)
    └─ ModeRuntime → wrappers that delegate to pb.encode / pb.decode

one `<lua_pkg>.lua` per input `.proto`
```

The plugin only handles `syntax = "proto3"`. Proto2 input is rejected
at the top of `GenerateFile`. See [proto2 deferral](#proto2-deferral)
for the rationale.

## CLI options

Passed via `protoc --tarantool_opt=<key>=<value>,...`:

| Option | Values | Meaning |
|---|---|---|
| `mode` | `full` (default) or `runtime` | Inline `_encode`/`_decode` vs descriptor-delegating wrappers. See [api-modes.md](api-modes.md). |
| `prefix` | any Lua-require path | Prepended to every generated module's require path **and** its on-disk subpath. The Justfile uses this to generate both modes side-by-side into `examples/expected/{full,runtime}/`. |

The `(tarantool.lua_package)` file option (defined in
`options/tarantool/tarantool.proto`) overrides the per-file Lua module
path:

```proto
import "tarantool/tarantool.proto";
option (tarantool.lua_package) = "myapp.proto.foo";
```

Without that option the path mirrors `package` (e.g.
`package my.app; foo.proto` → `my/app/foo_pb.lua`,
`require('my.app.foo_pb')`).

## What gets emitted per .proto

```lua
-- Code generated by protoc-gen-tarantool. DO NOT EDIT.
-- source: hello.proto
-- syntax: proto3

local pb = require('pb')
local wire = pb.wire

local M = {}

-- 1. Enums first (no forward-reference problems).
M.Status_descriptor = pb.enum('hello.Status', { ACTIVE=0, BANNED=1, ... })
M.Status            = M.Status_descriptor.by_name

-- 2. Predeclare all message descriptor tables. Self-references and
--    mutual recursion resolve here because every descriptor is named
--    before any field references it.
M.Person_descriptor = {name = 'hello.Person'}
M.Address_descriptor = {name = 'hello.Address'}
M.Result_descriptor = {name = 'hello.Result'}

-- 3. Fill in fields[] per message, then call pb.finalize_message.
M.Person_descriptor.fields = {
    {name='name',    id=1, kind='scalar', proto_type='string'},
    {name='user_id', id=2, kind='scalar', proto_type='uint64'},
    -- ...
}
M.Person_descriptor.oneofs        = { outcome = {'text', 'code', 'details'} }
M.Person_descriptor.reserved_names = { ['old_field'] = true }
pb.finalize_message(M.Person_descriptor)

-- 4. EmmyLua / lua-language-server type annotations.
---@class hello.Person
---@field name string
---@field user_id integer
-- ...

-- 5. Wrappers: _new / _encode / _decode / _decode_lazy / _text
--    + _has_<field> / _clear_<field> for explicit-optional fields.
function M.Person_new(t) return t or {} end
function M.Person_encode(t) -- inline body in full mode, delegation in runtime mode
function M.Person_decode(b) -- same
function M.Person_decode_lazy(b) return pb.decode_lazy(M.Person_descriptor, b) end
function M.Person_text(t, opts) return pb.text.encode(M.Person_descriptor, t, opts) end

-- 6. Services (mode-independent — they reuse the per-message wrappers).
M.Greeter_service = { ... }
function M.Greeter_client(transport) ... end
function M.Greeter_server(impl) ... end

return M
```

Generation order matters: enums and predeclared descriptors come before
field tables so cross-references inside `fields = {...}` resolve in one
pass.

## The descriptor table — the contract

The shape consumed by the runtime is:

```lua
{
    name = 'pkg.Foo',
    fields = {
        {name='x', id=1, kind='scalar', proto_type='int32'},
        {name='y', id=2, kind='message', message=<other_descriptor>},
        {name='z', id=3, kind='enum',    enum=<enum_descriptor>},
        {name='m', id=4, kind='map',     key=<sub_field>, value=<sub_field>},
        -- modifiers: repeated, packed, oneof, optional
    },
    -- Filled in by pb.finalize_message:
    field_by_id   = {[1]=<field>, ...},
    field_by_name = {x=<field>, ...},
    oneofs_list   = {{name=<n>, members={<names>}}, ...},
    -- Optional, set by the codegen:
    oneofs         = {<name> = {<member-names>}},   -- input form, flattened above
    reserved_names = {[<name>] = true},              -- for text-format parser
    encode = <fn>,  decode = <fn>,                   -- override hook (WKT use this)
    text   = <fn>,                                   -- text-format encode override
}
```

**Four producers emit this exact shape.** They live in different
codepaths but consume the same runtime:

1. `cmd/protoc-gen-tarantool/internal/gen` — build-time codegen.
2. `runtime/pb/dynamic.lua` — runtime synthesis from an AST that
   `runtime/pb/parser.lua` produces from `.proto` source
   (`pb.parse(text)`).
3. `runtime/pb/fileset.lua` — runtime synthesis from a
   `FileDescriptorSet` binary (`pb.from_pb(bytes)`;
   produced by `protoc --descriptor_set_out=...`).
4. `runtime/pb/wkt.lua` — hand-rolled WKT descriptors. Same shape but
   additionally set `desc.encode` / `desc.decode` (and sometimes
   `desc.text`, `desc.json_encode` / `desc.json_decode`) to take over
   the wire path entirely. Covers Timestamp, Duration, Wrappers, Struct,
   Value, ListValue, Any, FieldMask, Empty.

**Pinning this shape is what lets us run every behavior test against
both codegen modes.** The test harness in `test/protobuf_test.lua` and
friends parametrizes by mode (`{'full', 'runtime'}`) and runs the same
assertions against `require('full.hello.hello_pb')` and
`require('runtime.hello.hello_pb')`.

## Inline (full) mode — what it looks like

`inline.go` walks each field and emits Lua statements that call into
`runtime/pb/wire.lua` primitives directly. Tag bytes are precomputed at
codegen time as Lua string literals so the runtime never re-encodes
them.

```lua
function M.Person_encode(t)
    if type(t) ~= 'table' then
        error("expected table for hello.Person, got " .. type(t), 0)
    end
    local out, n = {}, 0
    local v

    -- field 1: name (string, singular)
    v = t.name
    if v ~= nil and v ~= '' then               -- proto3 default elision
        n = n + 1; out[n] = "\x0a"             -- tag(1, LEN), precomputed
        n = n + 1; out[n] = wire.encode_varint(#v)
        n = n + 1; out[n] = v
    end

    -- field 11: weight_kg (double, singular)
    v = t.weight_kg
    if v ~= nil and (v ~= 0 or 1/v == -math.huge) then  -- -0 preserved
        n = n + 1; out[n] = "\x59"
        n = n + 1; out[n] = wire.encode_double(v)
    end

    -- ... oneof pre-pass picks active branch; map fields walk pairs(); etc.

    return table.concat(out)
end
```

The decoder is similarly inlined:

```lua
function M.Person_decode(b)
    local result = {}
    local pos, len = 1, #b
    while pos <= len do
        local id, wt
        id, wt, pos = wire.decode_tag(b, pos)
        if id == 1 then                         -- name
            result.name, pos = wire.decode_string(b, pos)
        elseif id == 2 then                     -- user_id
            result.user_id, pos = wire.decode_uint64(b, pos)
        -- ... if-elseif chain on field id ...
        else
            pos = wire.skip_field(b, pos, wt, id)   -- unknown
        end
    end
    return result
end
```

**Why an `if-elseif` chain rather than a dispatch table?** LuaJIT
compiles short chains into a tight branch sequence on a single trace.
The crossover where a jump table would win is at field counts higher
than anything we hit in practice; pinned by `bench/jit_trace.lua`.

## Runtime mode — the thin wrappers

`mode=runtime` skips the inline emission and produces:

```lua
function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end
```

`pb.encode` / `pb.decode` (in `runtime/pb/codec.lua`) walk the
descriptor table to pick wire primitives at runtime. Slower per
field — a hash lookup + closure call instead of a hard-coded branch —
but smaller generated output (no per-message `_encode` / `_decode`
bodies) and useful when the same code path needs to handle dynamic
descriptors.

`pb.finalize_message` (in `runtime/pb/init.lua`) builds two
specialization layers on top of the descriptor at finalize-time:

1. **Per-field writer closures** (`f._writer`): pre-bound to the
   field's tag, encoder, and default predicate. The encode loop calls
   `f._writer(data, out)` per field instead of dispatching on
   `f.kind` / `f.proto_type` inside the loop.
2. **Per-field reader closures** (`f._reader`): same idea for decode,
   handling typed value extraction + repeated bookkeeping + nested
   merge + oneof sibling clearing in one closure per field.

These specializations bridge most of the gap between runtime mode and
full mode for the *decoder*; encode is still meaningfully slower in
runtime mode because the field walk goes through one extra Lua-function
boundary per field.

## The hot-path rules the generated code observes

These come from CLAUDE.md and are enforced across both the codegen and
the runtime. New code (codegen output or hand-written) needs to follow
them or the LuaJIT trace compiler bails out.

- **No `pairs()` on hot paths.** `pairs()` over a hash compiles to
  bytecode `ISNEXT`, which is NYI in Tarantool's LuaJIT 2.1 fork —
  the trace aborts and the code path falls off the JIT. Map fields are
  the one accepted exception; the JIT-trace gate in `bench/jit_trace.lua`
  pins the limitation. When a descriptor needs hash-shaped lookup state
  (e.g. oneof grouping), build a parallel array view at
  `pb.finalize_message` time (`desc.oneofs_list`) and iterate the array
  on the hot path.
- **64-bit integers stay as cdata.** `int64`/`uint64`/`sint64`/
  `fixed64`/`sfixed64` are LuaJIT `int64_t` / `uint64_t` cdata
  everywhere. Same convention as Tarantool's `msgpackffi`, `net.box`,
  `box.tuple`, and built-in `protobuf`. Don't narrow through
  `tonumber` — precision loss past 2^53 silently corrupts IDs and
  timestamp nanos.
- **`ipairs` / `for i=1,#t`, never `pairs()`, on generated arrays.**
  Repeated fields are 1-based contiguous Lua arrays. Generated encode
  walks them with `for i = 1, #v do`.
- **Keep hot helpers small.** Adding "more fast paths" to a helper that
  the JIT already inlines (e.g. extending `encode_varint`'s 1-byte
  fast path with 2/3/4-byte branches) can push the function past the
  LuaJIT inline budget — parent traces stop inlining it and the
  dominant 1-byte case regresses by ~30%. Multi-byte fast paths live
  in the *slow* helper, not the inlined one.
- **SoA over AoS for many-entry index structures.** For per-element
  records where N is in the thousands, use parallel int arrays, not
  one small Lua table per entry. Each per-entry table is a separate
  allocation with header overhead and hash dispatch; for large N the
  allocations dominate. See `runtime/pb/lazy.lua`'s `index_bytes`
  for the reference shape.
- **WKT routing.** When codegen sees a field referencing a
  `google.protobuf.*` type, it emits a reference to
  `pb.wkt.<Name>_descriptor` instead of treating it as an ordinary
  cross-file import. The plugin does NOT generate a Lua module for
  imported WKT `.proto` files — those are fulfilled by
  `runtime/pb/wkt.lua`.

## Plugin source layout

```
cmd/protoc-gen-tarantool/
  main.go              # reads CodeGeneratorRequest from stdin, sets up
                       # the plugin options + FEATURE_PROTO3_OPTIONAL,
                       # dispatches to gen.GenerateFile per .proto
  internal/gen/
    gen.go             # per-file orchestration + runtime-mode wrappers,
                       # imports collection, enum + message emission,
                       # descriptor field tables, oneofs, reserved_names
    inline.go          # full-mode inline _encode / _decode body emission,
                       # field-kind dispatch, repeated / map / oneof shapes
    service.go         # gRPC client / server factories
    name.go            # Lua name + path mangling rules
    types.go           # protoreflect.Kind → scalar name mapping
    options.go         # (tarantool.lua_package) file option lookup
    emmylua.go         # EmmyLua / lua-language-server type annotation
                       # emission (---@class, ---@field, ---@param)
```

**Adding a new wire type or scalar** means touching three places:

1. `runtime/pb/wire.lua` — add the `encode_xxx` / `decode_xxx`
   primitives + a `TYPE_INFO[xxx]` entry. That's the single source of
   truth: the codec, both codegen modes, and the dynamic parser all
   pick it up from there.
2. `cmd/protoc-gen-tarantool/internal/gen/types.go` —
   `scalarName(k)` mapping for the new `protoreflect.Kind`.
3. `cmd/protoc-gen-tarantool/internal/gen/inline.go` — emission for
   the new type (singular + repeated + packed branches).

If the new type is `LEN`-typed (string/bytes-shaped), check the
existing string/bytes paths for the split-emission pattern
(`tag + varint(#v) + body` as three separate `out` slots instead of
one concatenated `wire.encode_len(v)` call) — that avoids a per-field
string allocation.

## Sibling plugin: protoc-gen-tarantool-doc

`cmd/protoc-gen-tarantool-doc/` is a second Go plugin that consumes
the same `CodeGeneratorRequest` and emits one Markdown file per input
`.proto`. Build with `just build-doc`; sample docs land in
`examples/docs/` via `just gen-docs`. It's deliberately separate from
the Lua codegen to keep the codegen plugin small.

## Proto2 deferral

The plugin rejects `syntax = "proto2"` outright. The conformance
suites' 1331 skipped tests are all `TestAllTypesProto2`.

Proto2 support is genuinely a separate slice. The wire format is
identical to proto3 (with the one exception of `group`s — wire-types 3
and 4, SGROUP/EGROUP, which we already tolerate on the skip path but
don't encode/decode). The schema features that *don't* exist in proto3
are what would need work:

- **`required` fields** — codec must error on missing required at both
  encode and decode (we currently treat absence as default).
- **`optional` everywhere** — proto2 fields are presence-tracked by
  default. proto3's explicit-optional path already handles this; just
  needs the plugin to mark every proto2 field as `optional=true`.
- **Extensions** (`extensions 100 to 199;` + `extend Foo {...}`)
  — proto2-only mechanism for adding fields to messages defined
  elsewhere. Round-trip overlaps with our existing
  `_unknown_fields` machinery but explicit support needs a separate
  descriptor table per extension and a way to expose it on user
  messages.
- **Groups** (`optional group Foo = 1 { ... }`) — actual encode/decode
  rather than the skip-only tolerance we have today, plus a different
  text-format rendering (`Foo { ... }` with the group's capitalized
  name instead of the submessage's field name).
- **Custom defaults** (`[default = X]`) — only matter when callers
  query "what's the default value of field X"; for round-trips they
  can be ignored.
- **`MessageSet`** wire format — a few specific tests.

Realistic phasing (if you take this on):

1. Drop the proto3-only gate in the plugin; mark proto2 fields as
   `optional=true`; skip extensions / groups / required harmlessly
   (warn but emit the rest of the message). Re-run conformance —
   should unlock most of the 1007 binary tests and ~200 of the JSON
   tests immediately because the wire format is the same.
2. Implement group encode/decode + the text-format group rendering.
   Picks up the remaining text tests and a handful of binary tests.
3. Required-field validation. Small, targeted at the
   `TestAllRequiredTypesProto2` cases.
4. Extensions, MessageSet, custom defaults — long tail, optional
   depending on demand.

This is a meaningfully larger slice than the typical addition here —
plan on a few sessions, not an evening. The trade-off is "proto2 is
mostly legacy" (see the README's status table for who actually uses
it) vs "1331 extra ✓ in the conformance numbers looks good".