#How-to: dynamic schemas from a Tarantool space

When you can't (or don't want to) run protoc at build time — schema registries, multi-tenant apps where each tenant has their own messages, plugin systems where users upload protos — pb.parse and pb.from_pb let you build descriptor modules at runtime from .proto source or binary FileDescriptorSet bytes.

#Two entry points

Both return modules shaped exactly like generated mode=runtime output — same <Msg>_descriptor, <Msg>_encode, <Msg>_decode, <Msg>_decode_lazy, <Msg>_text surface. Anything that consumes a generated module consumes these.

#pb.parse — source string in, module out

local pb = require('pb')

local source = [[
syntax = "proto3";
package demo;

message User {
    int32 id = 1;
    string name = 2;
    repeated string tags = 3;
}
]]

local demo = pb.parse(source)

local bytes = demo.User_encode({
    id = 42, name = 'Alice', tags = {'admin', 'active'},
})
local user = demo.User_decode(bytes)

The parser handles every proto3 grammar bucket: services, options, imports, nested messages, oneofs, maps, explicit-optional. WKT imports (google/protobuf/*.proto) resolve to the runtime's pb.wkt descriptors automatically — no extra setup needed.

#Storing schemas in a space

The runnable example lives at examples/dynamic/load_from_space.lua. Pattern:

local pb = require('pb')

-- One row per schema, keyed by name + carrying a version for invalidation.
box.schema.space.create('proto_schemas')
box.space.proto_schemas:format({
    {name = 'name',    type = 'string'},
    {name = 'version', type = 'unsigned'},
    {name = 'source',  type = 'string'},
})
box.space.proto_schemas:create_index('pk', {parts = {'name'}})

-- Cache parsed modules by (name, version). Bumping the version in the
-- row evicts the cache entry naturally on next get_module.
local cache = {}

local function get_module(name)
    local row = box.space.proto_schemas:get(name)
    if row == nil then error('schema not found: ' .. name, 0) end
    local key = name .. '@' .. tostring(row.version)
    local mod = cache[key]
    if mod == nil then
        mod = pb.parse(row.source)
        cache[key] = mod
    end
    return mod
end

Why version-keyed cache: an UPDATE on the row bumps the version, so the next get_module parses the new source and the old module naturally becomes unreachable.

#Schema evolution

Adding a field is wire-compatible. Bytes encoded with the old schema decode fine with the new one (the new field reads as its proto3 default; explicit-optional reads as nil):

local bytes_v1 = old.User_encode({id = 42, name = 'Alice'})

box.space.proto_schemas:replace{'demo.user', 2, [[
syntax = "proto3";
package demo;
message User {
    int32 id = 1;
    string name = 2;
    repeated string tags = 3;
    string email = 4;  -- new
}
]]}

local user_via_v2 = get_module('demo.user').User_decode(bytes_v1)
-- user_via_v2.email == ""  (proto3 default for implicit-presence string)

Removing a field is not wire-compatible — the bytes for that field become "unknown" to the new schema. The runtime preserves them via _unknown_fields so a round-trip back through the same instance keeps the data, but consumers that only see the new schema can't see the field.

Reusing a field number after removal is a category of data corruption the wire format can't catch. If you remove a field, reserved 4; the number to prevent reuse.

#pb.from_pb — for compiled descriptor sets

When you have a FileDescriptorSet (from protoc --descriptor_set_out or a gRPC reflection response), pb.from_pb ingests it:

local pb = require('pb')

-- Build the FileDescriptorSet at deploy time, ship it as a blob.
--   protoc --descriptor_set_out=schemas.pb --include_imports my/app/*.proto
local bytes = io.open('schemas.pb', 'rb'):read('*a')
local set = pb.from_pb(bytes)

-- Lookup by file name (the original .proto path):
local hello = set.files['hello.proto']
local user_bytes = hello.User_encode({...})

-- Or by fully-qualified message name across all files:
local desc = set.lookup('hello.Person')
local user_bytes = pb.encode(desc, {...})

--include_imports is important — it embeds every transitively-imported file in the set. Without it, references to imports won't resolve.

#When to pick which

For known-at-build-time schemas, generated _pb.lua is still the right choice — same module shape, no startup cost, JIT-friendly inline encode/decode in mode=full.

#Cost notes

pb.parse does real lexing/parsing work — keep it out of hot paths. Parse once at boot (or per-schema-update) and cache the resulting module. Encode/decode against a parsed-descriptor module runs the same codec as generated mode=runtime, ~5-15% slower than full-mode inline; allocation profile matches.

pb.from_pb skips parsing entirely (it consumes binary protobuf itself, via the hand-built descriptor.proto descriptors in runtime/pb/descriptor_pb.lua).

#What's next

• Reference: runtime API — full pb.parse / pb.from_pb signatures.
• How-to: build integration — generating --descriptor_set_out from your build system.