#API modes: full, runtime, lazy

The same descriptor table feeds three encode/decode call shapes. They are not exclusive — every generated _pb.lua exposes all three (_encode, _decode, _decode_lazy), and pb.encode / pb.decode / pb.decode_lazy work directly against any descriptor. Picking is per-call-site, not per-message and not per-build.

#TL;DR

If you don't know which to use: start with full. Switch to lazy when profiling shows you're decoding more than you read; switch to runtime when the schema isn't known at build time.

#Full (inline codegen)

The default, generated by mode=full (the codegen default). For each message the plugin emits dedicated _encode and _decode functions with the wire calls and tag bytes inlined.

local hello = require('myapp.proto.hello')

local bytes = hello.Person_encode({
    name    = 'Alice',
    user_id = require('ffi').cast('uint64_t', 42),
    emails  = {'a@x', 'b@x'},
})
local p = hello.Person_decode(bytes)

Under the hood the generated Person_encode looks like:

function M.Person_encode(t)
    if type(t) ~= 'table' then
        error("expected table for hello.Person, got " .. type(t), 0)
    end
    local out, n = {}, 0
    local v
    v = t.name
    if v ~= nil and v ~= '' then
        n = n + 1; out[n] = "\x0a"           -- precomputed tag bytes
        n = n + 1; out[n] = wire.encode_varint(#v)
        n = n + 1; out[n] = v
    end
    v = t.user_id
    if v ~= nil and v ~= 0 then
        n = n + 1; out[n] = "\x10"
        n = n + 1; out[n] = wire.encode_uint64(v)
    end
    -- ... one branch per field ...
    return table.concat(out)
end

No descriptor table lookups, no kind dispatch, no pairs() over a hash on the hot path. Tag bytes are precomputed Lua string literals. The LuaJIT trace compiler sees a flat sequence of monomorphic wire calls and specializes the whole encode into a single trace.

This is what just bench measures by default. The bench/baseline.json numbers under the "full" column are this path.

#_decode_unsafe — opt-out of UTF-8 validation

For string fields, _decode runs utf8.len on every payload — the proto3 spec demands it, and the conformance suite enforces it. When the bytes you are decoding came from a trusted producer (typically your own encoder across a typed RPC, JSON or text round-trip, or in-process pipeline), that re-validation is wasted work. Full-mode codegen emits a sister M.<Name>_decode_unsafe(buf) alongside _decode that drops the check at every string site and recurses into sub-messages' _decode_unsafe.

-- Trusted source (we encoded these bytes ourselves):
local p = hello.Person_decode_unsafe(bytes)

-- Untrusted source (arrived over the wire from a peer): keep the safe path.
local p = hello.Person_decode(bytes)

On a string-heavy ~1KB Person payload (26 emails) the unsafe variant runs ~20-30% faster end-to-end. The variant is opt-in by an explicit function name rather than a per-call flag so the JIT trace shape and inlining budget for the safe path are unchanged. The cost is generated code size: each message with string content emits a second decoder body.

Runtime mode (mode=runtime codegen and the reflective pb.decode_unsafe entry) also supports the unsafe path: pb.finalize_message eagerly compiles a parallel f._reader_unsafe set against a swapped scalar table (string -> bytes handler), and pb.decode_unsafe(desc, buf) dispatches through those readers with sub-message / extension / group recursion all on the unsafe path. Runtime-mode wins are smaller — ~5-10% on the same payload — because the descriptor-dispatch and closure indirection swamp the utf8_len savings, but it's still net positive and the API is symmetric with full mode.

Caveats:

• C-acceleration is bypassed in the unsafe variant — the C runtime validates unconditionally today, so _decode_unsafe skips the pb.c_runtime dispatch and runs the inline Lua decoder. On payloads where the C runtime wins, the validating _decode may still beat the Lua _decode_unsafe; measure before adopting.

#Runtime (descriptor-driven / reflect)

The plugin's mode=runtime emits thin wrappers that delegate to pb.encode(desc, t) and pb.decode(desc, b):

function M.Person_encode(t) return pb.encode(M.Person_descriptor, t) end
function M.Person_decode(b) return pb.decode(M.Person_descriptor, b) end

You can also call those directly with any descriptor — that's the "reflect" use case:

local pb = require('pb')

-- Parse a .proto file at runtime; no codegen, no Go plugin.
local mod  = pb.parse(io.open('hello.proto'):read('*a'))
local desc = mod.Person_descriptor

-- Same encode / decode surface, just bound to a runtime-built descriptor.
local bytes = pb.encode(desc, {name = 'Alice'})
local p     = pb.decode(desc, bytes)

-- Or ingest a binary FileDescriptorSet from protoc --descriptor_set_out:
local set  = pb.from_pb(io.open('build/all.pb', 'rb'):read('*a'))
local desc = set.lookup('hello.Person')
local p    = pb.decode(desc, bytes)

The descriptor table is the contract — every field-name, presence rule, and kind tag the runtime cares about lives there. Canonical shape: codegen.md → descriptor table. A minimal hello.Person looks like:

{
    name = 'hello.Person',
    fields = {
        {name='name', id=1, kind='scalar', proto_type='string'},
        {name='user_id', id=2, kind='scalar', proto_type='uint64'},
        {name='emails', id=3, kind='scalar', proto_type='string', repeated=true},
        {name='address', id=5, kind='message', message=<other_descriptor>},
        -- modifiers: repeated, packed, oneof, optional
    },
}
-- field_by_id / field_by_name / oneofs_list filled by pb.finalize_message

Four producers emit this shape — generated codegen (full or runtime), pb.parse (from .proto source), pb.from_pb (from FileDescriptorSet bytes), and runtime/pb/wkt.lua (hand-rolled WKTs). All flow through the same pb.codec.

The pb.finalize_message(desc) helper builds field_by_id / field_by_name / oneofs_list and attaches per-field _writer / _reader specializations. Always call it on hand-rolled descriptors.

Cost vs full: per-field table lookup + kind dispatch. Allocation overhead is one extra pairs() walk in encode and a small per-field function-call cost; on just bench numbers the runtime mode is ~5-15% slower for encode and within noise for decode (the decoder fast paths are shared via f._reader closures).

WKTs (google.protobuf.Timestamp etc) carry their own desc.encode / desc.decode overrides; the runtime codec dispatches to them in place of the generic field walk. That's the extension point for any descriptor that wants custom handling without a special case in the codec.

#Lazy (zero-copy view)

pb.decode_lazy(desc, bytes) returns a MessageView instead of a Lua table. Nothing past the field index gets decoded eagerly: scalar materialization happens on :get, repeated/map fields become ArrayView / MapView sub-views, sub-messages become nested MessageViews on access.

local view = hello.Person_decode_lazy(bytes)
local F = hello.Person_fields            -- field-name constants
local AF = hello.Address_fields
local R = hello.Result_oneofs            -- oneof-group constants

-- Direct field reads (decoded on demand, then cached on the view).
local name = view:get(F.name)
local uid  = view:get(F.user_id)

-- Repeated field: ArrayView.
local emails = view:get(F.emails)        -- ArrayView, no per-element table
for i, e in emails:iter() do
    print(i, e)
end
print('count:', emails:len())

-- Map field: MapView. Field name uses the constants table; the map
-- key passed to MapView:get is a raw value, not a field name.
local ages = view:get(F.ages_by_nickname)
print(ages:get('alice'))

-- Sub-message: a nested MessageView. Switch to the sub-message's
-- own constants table for its fields.
local addr = view:get(F.address)
print(addr:get(AF.city))

-- Presence + oneof which-branch checks without forcing a decode.
view:has(F.user_id)        -- bool
view:which(R.outcome)      -- string?

-- Iterate fields actually present on the wire (skip-aware).
for name, val in view:iter() do print(name, val) end

-- Mutate, then re-encode. Untouched fields are spliced byte-for-byte
-- from the original payload; only dirty fields run through encode.
view:set(F.user_id, require('ffi').cast('uint64_t', 99))
local new_bytes = view:encode()

-- Force a fully-materialized table when you actually need one.
local t = view:totable()

#Field-name constants — required

Every generated message exports a M.<Type>_fields table mapping each field name to itself, and (when applicable) M.<Type>_oneofs for oneof group names. Both tables are frozen behind a strict __index that errors on unknown keys and a read-only __newindex.

The lazy view API (:get / :has / :set / :clear / :which) takes a field name string — route every field-name argument through the constants table rather than passing a string literal:

-- right
view:get(hello.Person_fields.user_id)
-- typo here errors at the read site:
-- "unknown field name: 'user_di'"

-- wrong (silently returns nil; indistinguishable from absent optional)
view:get('user_di')

Why required: view:get('user_di') returns nil whether the field name is wrong or the field was legitimately absent on the wire. The ambiguity tends to surface as missing data far downstream. The constants table catches the typo where it was written.

This is a lazy-view contract. The eager _decode / _encode path round-trips plain Lua tables whose keys are written by the caller's own code, so the same safety net doesn't apply there — eager users already see misspelled keys as direct test failures.

Map keys (MapView:get(k), MapView:has(k)) and array indices (ArrayView:at(i)) are values, not field names — pass them raw.

#When lazy wins

• Sparse reads — payload is large, you only need a few fields. Index build is ~one Lua table + four SoA int arrays sized to <wire-segment-count>. Per-field reads are O(1) after the index.
• Mostly-passthrough re-encode — proxy or router shapes that decode, touch a couple of fields, and re-encode. The byte-splice path skips encode entirely for untouched fields; only dirty fields pay the encode cost. On the 100 KB hello.Person bench this puts the mutate-then-reencode workload at 1.09-1.26× of eager despite the extra index build.
• Avoiding intermediate table churn when iterating large repeated fields: ArrayView:iter() doesn't allocate a flat array; values flow through one at a time.

#When lazy loses

• Dense reads — if you read every field, the index build is wasted work and each :get adds a Lua-call boundary the eager decoder avoided. Eager wins by ~10-30% on full-field walks.
• Tiny messages — for a 10 B payload the index build dominates. The cross-over (on hello.Person) is around 1 KB.
• Anything that wants the result as a plain Lua table — :totable() reverses the win.

#Implementation note (why SoA)

The lazy index is structure-of-arrays, not array-of-structs: for N wire segments the indexer keeps four int arrays of size N (id, tag_start, val_start, next_start) instead of N tiny tables of 4 keys each. Tiny per-entry tables in LuaJIT carry header overhead + hash dispatch costs that dominate for large N. Switching from AoS to SoA on the protobuf-lazy slice took sparse-read from 0.66× of eager back to 1.0×-1.16×.

If you write your own index-style structure with many small entries, follow the same pattern. See runtime/pb/lazy.lua's index_bytes for the reference shape.

#Mixing modes

All three modes interoperate freely:

• A MessageView from decode_lazy can be :set(...) and :encode()'d back to wire bytes that pb.decode (eager) parses identically.
• A descriptor from pb.parse (runtime mode) plugs into pb.decode_lazy exactly like a generated descriptor.
• The codec's desc.encode / desc.decode override mechanism lets WKT and user-registered descriptors short-circuit any of the three paths without special cases in the codec.

The contract is the descriptor table — the codec, both codegen modes, the dynamic parser, the JSON codec, the text codec, and the lazy view all consume the same shape. That's the design property that lets these three APIs coexist without forks.

#Choosing per call site, not per build

Both codegen modes always emit Foo_decode_lazy and Foo_text wrappers. The plugin's mode=full/mode=runtime switch only changes how Foo_encode / Foo_decode are emitted — everything else lives in the shared runtime. So you can:

• Generate everything with mode=full (the default) and still call pb.encode(desc, t) against the descriptor when you need a descriptor-driven path.
• Generate everything with mode=runtime if you want minimum generated-code size, and pay the modest descriptor-dispatch overhead uniformly.
• Reach for lazy on a per-call basis when the workload shape calls for it — typically in handlers that operate on large messages but only inspect a few fields.

The test suite runs every behavior test against both codegen modes (full and runtime) via the same descriptor; cross-mode parity is pinned by parity.full_vs_runtime.* test groups.