#API modes: full, runtime, lazy

The same descriptor table feeds three encode/decode call shapes. They are not exclusive — every generated _pb.lua exposes all three (_encode, _decode, _decode_lazy), and pb.encode / pb.decode / pb.decode_lazy work directly against any descriptor. Picking is per-call-site, not per-message and not per-build.

#TL;DR

If you don't know which to use: start with full. Switch to lazy when profiling shows you're decoding more than you read; switch to runtime when the schema isn't known at build time.

#Full (inline codegen)

The default, generated by mode=full (the codegen default). For each message the plugin emits dedicated _encode and _decode functions with the wire calls and tag bytes inlined.

local hello = require('myapp.proto.hello')

local bytes = hello.Person_encode({
    name    = 'Alice',
    user_id = require('ffi').cast('uint64_t', 42),
    emails  = {'a@x', 'b@x'},
})
local p = hello.Person_decode(bytes)

Under the hood the generated Person_encode looks like:

function M.Person_encode(t)
    if type(t) ~= 'table' then
        error("expected table for hello.Person, got " .. type(t), 0)
    end
    local out, n = {}, 0
    local v
    v = t.name
    if v ~= nil and v ~= '' then
        n = n + 1; out[n] = "\x0a"           -- precomputed tag bytes
        n = n + 1; out[n] = wire.encode_varint(#v)
        n = n + 1; out[n] = v
    end
    v = t.user_id
    if v ~= nil and v ~= 0 then
        n = n + 1; out[n] = "\x10"
        n = n + 1; out[n] = wire.encode_uint64(v)
    end
    -- ... one branch per field ...
    return table.concat(out)
end

No descriptor table lookups, no kind dispatch, no pairs() over a hash on the hot path. Tag bytes are precomputed Lua string literals. The LuaJIT trace compiler sees a flat sequence of monomorphic wire calls and specializes the whole encode into a single trace.

This is what make bench measures by default. The bench/baseline.json numbers under the "full" column are this path.

#Runtime (descriptor-driven / reflect)

The plugin's mode=runtime emits thin wrappers that delegate to pb.encode(desc, t) and pb.decode(desc, b):

function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end

You can also call those directly with any descriptor — that's the "reflect" use case:

local pb = require('pb')

-- Parse a .proto file at runtime; no codegen, no Go plugin.
local mod  = pb.parse(io.open('hello.proto'):read('*a'))
local desc = mod.Person_descriptor

-- Same encode / decode surface, just bound to a runtime-built descriptor.
local bytes = pb.encode(desc, {name = 'Alice'})
local p     = pb.decode(desc, bytes)

-- Or ingest a binary FileDescriptorSet from protoc --descriptor_set_out:
local set  = pb.from_pb(io.open('build/all.pb', 'rb'):read('*a'))
local desc = set.lookup('hello.Person')
local p    = pb.decode(desc, bytes)

The descriptor table is the contract:

{
    name = 'hello.Person',
    fields = {
        {name='name', id=1, kind='scalar', proto_type='string'},
        {name='user_id', id=2, kind='scalar', proto_type='uint64'},
        {name='emails', id=3, kind='scalar', proto_type='string', repeated=true},
        {name='status', id=4, kind='enum', enum=<enum_descriptor>},
        {name='address', id=5, kind='message', message=<other_descriptor>},
        {name='ages_by_nickname', id=13, kind='map',
         key={kind='scalar', proto_type='string'},
         value={kind='scalar', proto_type='int32'}},
        -- modifiers: repeated, packed, oneof, optional
    },
    field_by_id   = {[1]=<field>, ...},  -- filled by pb.finalize_message
    field_by_name = {name=<field>, ...},
    oneofs        = {<oneof_name> = {<field_names>}},
    reserved_names = {[<name>]=true},     -- for text-format parser
}

Three producers emit this shape — generated codegen (runtime mode), pb.parse (from .proto source), pb.from_pb (from FileDescriptorSet bytes). All three flow through the same pb.codec.

The pb.finalize_message(desc) helper builds field_by_id / field_by_name / oneofs_list and attaches per-field _writer / _reader specializations. Always call it on hand-rolled descriptors.

Cost vs full: per-field table lookup + kind dispatch. Allocation overhead is one extra pairs() walk in encode and a small per-field function-call cost; on make bench numbers the runtime mode is ~5-15% slower for encode and within noise for decode (the decoder fast paths are shared via f._reader closures).

WKTs (google.protobuf.Timestamp etc) carry their own desc.encode / desc.decode overrides; the runtime codec dispatches to them in place of the generic field walk. That's the extension point for any descriptor that wants custom handling without a special case in the codec.

#Lazy (zero-copy view)

pb.decode_lazy(desc, bytes) returns a MessageView instead of a Lua table. Nothing past the field index gets decoded eagerly: scalar materialization happens on :get, repeated/map fields become ArrayView / MapView sub-views, sub-messages become nested MessageViews on access.

local view = hello.Person_decode_lazy(bytes)

-- Direct field reads (decoded on demand, then cached on the view).
local name = view:get('name')
local uid  = view:get('user_id')

-- Repeated field: ArrayView.
local emails = view:get('emails')   -- ArrayView, no per-element table
for i, e in emails:iter() do
    print(i, e)
end
print('count:', emails:len())

-- Map field: MapView.
local ages = view:get('ages_by_nickname')
print(ages:get('alice'))

-- Sub-message: a nested MessageView.
local addr = view:get('address')
print(addr:get('city'))

-- Presence + oneof which-branch checks without forcing a decode.
view:has('user_id')        -- bool
view:which('outcome')      -- string?

-- Iterate fields actually present on the wire (skip-aware).
for name, val in view:iter() do print(name, val) end

-- Mutate, then re-encode. Untouched fields are spliced byte-for-byte
-- from the original payload; only dirty fields run through encode.
view:set('user_id', require('ffi').cast('uint64_t', 99))
local new_bytes = view:encode()

-- Force a fully-materialized table when you actually need one.
local t = view:totable()

#When lazy wins

• Sparse reads — payload is large, you only need a few fields. Index build is ~one Lua table + four SoA int arrays sized to <wire-segment-count>. Per-field reads are O(1) after the index.
• Mostly-passthrough re-encode — proxy or router shapes that decode, touch a couple of fields, and re-encode. The byte-splice path skips encode entirely for untouched fields; only dirty fields pay the encode cost. On the 100 KB hello.Person bench this puts the mutate-then-reencode workload at 1.09-1.26× of eager despite the extra index build.
• Avoiding intermediate table churn when iterating large repeated fields: ArrayView:iter() doesn't allocate a flat array; values flow through one at a time.

#When lazy loses

• Dense reads — if you read every field, the index build is wasted work and each :get adds a Lua-call boundary the eager decoder avoided. Eager wins by ~10-30% on full-field walks.
• Tiny messages — for a 10 B payload the index build dominates. The cross-over (on hello.Person) is around 1 KB.
• Anything that wants the result as a plain Lua table — :totable() reverses the win.

#Implementation note (why SoA)

The lazy index is structure-of-arrays, not array-of-structs: for N wire segments the indexer keeps four int arrays of size N (id, tag_start, val_start, next_start) instead of N tiny tables of 4 keys each. Tiny per-entry tables in LuaJIT carry header overhead + hash dispatch costs that dominate for large N. Switching from AoS to SoA on the protobuf-lazy slice took sparse-read from 0.66× of eager back to 1.0×-1.16×.

If you write your own index-style structure with many small entries, follow the same pattern. See runtime/pb/lazy.lua's index_bytes for the reference shape.

#Mixing modes

All three modes interoperate freely:

• A MessageView from decode_lazy can be :set(...) and :encode()'d back to wire bytes that pb.decode (eager) parses identically.
• A descriptor from pb.parse (runtime mode) plugs into pb.decode_lazy exactly like a generated descriptor.
• The codec's desc.encode / desc.decode override mechanism lets WKT and user-registered descriptors short-circuit any of the three paths without special cases in the codec.

The contract is the descriptor table — the codec, both codegen modes, the dynamic parser, the JSON codec, the text codec, and the lazy view all consume the same shape. That's the design property that lets these three APIs coexist without forks.

#Choosing per call site, not per build

Both codegen modes always emit Foo_decode_lazy and Foo_text wrappers. The plugin's mode=full/mode=runtime switch only changes how Foo_encode / Foo_decode are emitted — everything else lives in the shared runtime. So you can:

• Generate everything with mode=full (the default) and still call pb.encode(desc, t) against the descriptor when you need a descriptor-driven path.
• Generate everything with mode=runtime if you want minimum generated-code size, and pay the modest descriptor-dispatch overhead uniformly.
• Reach for lazy on a per-call basis when the workload shape calls for it — typically in handlers that operate on large messages but only inspect a few fields.

The test suite runs every behavior test against both codegen modes (full and runtime) via the same descriptor; cross-mode parity is pinned by parity.full_vs_runtime.* test groups.