From 6d06f23bde35085120b67e4e793ac992c02aa438 Mon Sep 17 00:00:00 2001 From: Eugene Blikh Date: Sat, 16 May 2026 19:58:48 +0300 Subject: [PATCH] docs: spec out msgpack encoding and gRPC transport story MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two design slices captured under docs/specs/ for later pickup: - msgpack_encoding.md: proto schemas as IDL, msgpack as wire format. Default mp_map keyed by field number; ARRAY/MP_TUPLE opt-ins for box.space feeders; WKT bridging to MP_DATETIME / MP_INTERVAL. - grpc_transports.md: Connect-JSON over HTTP/1.1 as default external transport, net.box tunnel for in-cluster, explicit no on HTTP/2 termination (recommend Envoy). Conformance anchored on connectrpc/conformance — same framed-runner shape as the existing protobuf conformance harness. PLAN.md cross-links both specs from M4, Open Questions, and Non-goals. --- PLAN.md | 27 ++- docs/specs/grpc_transports.md | 292 +++++++++++++++++++++++++++++++++ docs/specs/msgpack_encoding.md | 273 ++++++++++++++++++++++++++++++ 3 files changed, 586 insertions(+), 6 deletions(-) create mode 100644 docs/specs/grpc_transports.md create mode 100644 docs/specs/msgpack_encoding.md diff --git a/PLAN.md b/PLAN.md index d4859c4145c0af55afbabd4841e26f2dc0471b51..da9d22e9aa4f14a5cbc0f47477f3c66c0445b40b 100644 --- a/PLAN.md +++ b/PLAN.md @@ -131,10 +131,21 @@ release on sourcecraft.dev. client_stream: function(stream, ctx) -> resp bidi: function(stream, ctx) ``` -- [ ] Reference network transports (separate projects, deferred): +- [ ] Reference network transports (separate projects, deferred). Full + design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): + - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via + `tarantool/http`. Default external transport; works with browsers + and `curl` without an HTTP/2 proxy. - `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls. - - HTTP/2 transport — out of scope; the transport interface above is - deliberately HTTP/2-shaped so a plug-in is straightforward. + First-class in-cluster path. + - `pb.grpc.transport.http_client_unary` — outbound, unary only, + via `http_client`. + - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in + front; the transport contract is HTTP/2-shaped so the same + generated code works behind it. + - Conformance anchor: `connectrpc/conformance` (same framed-runner + shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect + from one harness). **Done.** The loopback transport runs each streaming handler on its own fiber and bridges client ↔ handler via `fiber.channel`. All four flavors @@ -582,7 +593,8 @@ Run on a fixed corpus across 5 message sizes; track results in |---|---| | Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. | | Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. | -| Should we ship a stub HTTP/2 transport for gRPC? | Probably no — large dependency. Recommend `lua-http` or write a focused project. | +| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. | +| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. | | How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. | | Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. | | Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. | @@ -593,9 +605,12 @@ Run on a fixed corpus across 5 message sizes; track results in - **proto2** syntax — spec is more complex (required, default values, groups), real demand is rare on Tarantool. - **Protobuf editions** beyond what proto3 enables. -- **HTTP/2 transport** for gRPC (separate project). +- **HTTP/2 termination** for gRPC. Push it to Envoy; see + [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md). + Connect-JSON over HTTP/1.1 is the default external transport. - **Reflection service** (gRPC server reflection) — implement after M5. -- **gRPC-Web** — separate project. +- **gRPC-Web** — covered by the same Connect conformance harness if we + add it, but not on the v1 path. ## 9. How to update this plan diff --git a/docs/specs/grpc_transports.md b/docs/specs/grpc_transports.md new file mode 100644 index 0000000000000000000000000000000000000000..a38cdc501e5d1a656eab77c348572f69d6bf4e08 --- /dev/null +++ b/docs/specs/grpc_transports.md @@ -0,0 +1,292 @@ +# Spec: gRPC transports for Tarantool + +Status: **draft / decision deferred**. The transport *contract* is +shipped and stable (see [`runtime/pb/grpc.lua`](../../runtime/pb/grpc.lua)). +What's open is **which concrete transports we recommend and/or ship, +and how a user picks between them**. + +This spec maps the protocol landscape, says where Tarantool fits, and +flags what we'd build vs. recommend an external library for. + +## What's already shipped (and won't change) + +Generated `M._client(transport)` and `M._server(impl)` +talk to the transport in a single contract, regardless of wire protocol: + +``` +transport:unary(path, req_bytes, ctx) -> resp_bytes +transport:server_stream(path, req_bytes, ctx) -> stream +transport:client_stream(path, ctx) -> stream +transport:bidi(path, ctx) -> stream +``` + +`path` is `/pkg.Service/Method`. `ctx` is an opaque Lua table (headers, +deadline, metadata, …). Generated code encodes the request, hands raw +bytes to the transport, and decodes the response. + +This is deliberately HTTP/2-shaped — `path`, byte-oriented messages, +streams — but **the contract makes no commitment to a wire protocol**. +Every transport in this spec is a different plug-in behind the same +four methods. + +Reference transports already in `runtime/pb/grpc.lua`: + +- `pb.grpc.loopback(server)` — in-process; uses `fiber.channel`. Bridges + client → server fiber for tests and same-process apps. +- `pb.grpc.multiplex({srv1, srv2})` — fans several servers onto one + transport. Errors on duplicate paths. + +## The protocol matrix + +For a request that crosses a process boundary, you pick a pair: a +**wire protocol** (how bytes flow between processes) and a **codec** +(how a message turns into bytes). The Lua-side transport bridges between +the generated code and that wire/codec combo. + +| Wire protocol | Body codec | Streaming | Status signaling | Browser-friendly | Off-the-shelf clients/servers | Tarantool fit | +| ------------------------- | ------------ | ------------------- | ------------------------- | :--------------: | ----------------------------- | ------------------------------ | +| **gRPC over HTTP/2** | proto wire | unary + all 3 | HTTP/2 trailers | no | every gRPC lib | needs an external HTTP/2 lib | +| **gRPC-Web over HTTP/2** | proto wire | unary + server | trailers in body | yes (with proxy) | grpc-web JS, Envoy | same problem as gRPC + framing | +| **gRPC-Web over HTTP/1.1**| proto wire | unary + server | trailers in body | yes | grpc-web JS | works with `tarantool/http` | +| **Connect over HTTP/1.1** | proto wire | unary only | HTTP status + body | yes | connectrpc clients | works with `tarantool/http` | +| **Connect over HTTP/2** | proto wire | unary + all 3 | HTTP status / trailers | yes | connectrpc clients | same HTTP/2 problem | +| **Connect-JSON** | proto3 JSON | unary only | HTTP status + body | yes | curl + connectrpc clients | drop-in for `tarantool/http` | +| **gRPC-Gateway / transcoded REST + JSON** | proto3 JSON | unary | HTTP status + body | yes | any HTTP client | drop-in for `tarantool/http` | +| **gRPC over IProto tunnel** (Tarantool-native) | proto wire | unary + all 3 | IProto error code | n/a | this project, custom clients | first-class | +| **gRPC over net.box tunnel** | proto wire | unary + all 3 | net.box error | n/a | this project, custom clients | first-class | + +What's **not** in the matrix and why: + +- **gRPC over HTTP/3 / QUIC** — too early; the Go and C++ gRPC stacks + themselves treat it as experimental. Not worth specifying yet. +- **JSON-RPC, Thrift, etc.** — different IDL; off-topic. + +## What "Tarantool fit" actually means + +Tarantool gives us: + +- **`tarantool/http` server (HTTP/1.1)** — solid, idiomatic, lives in a + Lua rock. No HTTP/2, no server-pushed trailers. Good substrate for + Connect-JSON and gRPC-Gateway-style REST. +- **`http_client` (libcurl-based)** — HTTP/1.1 and HTTP/2 client. Has + streaming via callbacks, but trailers and gRPC framing are not + first-class. Usable for HTTP/2 unary; awkward for streaming. +- **`net.box`** — Tarantool's binary RPC protocol. Already gives us + request/response, streaming via long-poll, error propagation. Sane + default for in-cluster Tarantool→Tarantool calls. +- **IProto** — the wire protocol under net.box. Lower level; lets us + define our own request type that carries gRPC framing if we want + zero overhead. +- **No HTTP/2 server.** Real HTTP/2 termination needs a sidecar (Envoy, + nginx) or a new Lua library. Neither is something we'd ship. + +So the practical bands are: + +1. **In-cluster Tarantool↔Tarantool** → IProto or net.box tunnel. +2. **External clients calling Tarantool over HTTP** → Connect-JSON or + gRPC-Gateway-style REST behind `tarantool/http`. Both are HTTP/1.1 + only, both speak JSON, both work with `curl`/browsers without a + proxy. +3. **External clients that insist on real gRPC** → put Envoy or + grpcurl in front, terminate HTTP/2 there, and send unary calls into + Tarantool over HTTP/1.1 or IProto. We don't terminate HTTP/2 + ourselves. +4. **Outbound calls from Tarantool to external gRPC services** → + `http_client` for HTTP/2 unary. For streaming, accept "we don't + support that yet" rather than shipping a half-baked HTTP/2 client. + +## Recommended transports to build + +Names below refer to packages we'd publish; nothing here lives in this +repo yet beyond the contract. + +### `pb.grpc.transport.http_server` (HTTP/1.1 server-side) + +Adapts an `M._server(impl)` result into a `tarantool/http` +route handler. Wire protocol: **Connect-JSON over HTTP/1.1** by default, +with content-negotiation for Connect-protobuf. + +- POST `/{package.Service}/{Method}` with `Content-Type: application/json` + → decode body via `pb.json.decode(input_desc, body)`, call the + generated handler, encode reply via `pb.json.encode`. +- `application/proto` content type → use `pb.encode/pb.decode` instead. +- Streaming methods: respond 501 for now. Connect's `application/connect+json` + framed streaming over HTTP/1.1 chunked transfer is feasible later; + out of scope for v1. +- Errors: surface as Connect's JSON error envelope. Map common + gRPC status codes to HTTP status per the Connect spec. + +Why this first: it's the lowest-effort transport that gives us a real +external interface, and it works with browsers and `curl`. It also +covers the gRPC-Gateway use case without needing the gateway: +`POST /myapp.v1.Greeter/SayHello` with a JSON body is a fine REST +shape on its own. + +### `pb.grpc.transport.netbox` (in-cluster) + +`net.box` connection → speaks `pb.grpc` over a single user-defined +function (e.g. `box.schema.func.create('grpc_dispatch')`). Body is +a 2-tuple `{path, req_bytes}`; reply is `{ok, resp_bytes}` or +`{err, status_code, message}`. + +Streaming: lean on net.box's stream/iterator support. Server runs the +handler on a fiber; messages flow through `box.iproto.override` / +`box.session.push`. Concretely tractable; out of scope for v1 but +straightforward to add. + +Why second: in-cluster Tarantool clusters are a real and ready use +case. The transport is small and self-contained. + +### `pb.grpc.transport.http_client_unary` (outbound, optional) + +Adapts `http_client` to speak Connect-JSON or Connect-protobuf to +external services. Unary only. Easy. Useful for calling out from +Tarantool app code to a Connect or HTTP/1.1 gRPC-Web server. + +## Not recommended (don't build) + +- **Tarantool-side HTTP/2 server.** Would require a new HTTP/2 library + in Lua or a C module. Effort vastly exceeds payoff — anyone needing + HTTP/2 termination should run Envoy in front. Document the Envoy + setup instead. +- **Tarantool-side HTTP/2 streaming client.** `http_client`'s streaming + API isn't a clean fit for gRPC trailers and per-message framing. + Anyone needing this should bind to a real gRPC client (C, Go), not + reimplement in Lua. Document this limitation. +- **gRPC-Web framing.** It's a small spec, but every modern stack + (browser SDK, mobile SDK) prefers Connect now. Don't fragment effort + unless a user demands it. + +## How a user picks + +Decision tree, top-down: + +1. Both endpoints in a Tarantool cluster? + → `pb.grpc.transport.netbox`. +2. External clients only (browsers, curl, mobile)? + → `pb.grpc.transport.http_server` (Connect-JSON). +3. Need to call an external gRPC service from Tarantool? + → Unary: `pb.grpc.transport.http_client_unary` (Connect or HTTP/1.1 + gateway). Streaming: not supported; document the Envoy/sidecar + alternative. +4. External clients insist on real HTTP/2 gRPC? + → Envoy in front. Envoy terminates HTTP/2, talks Connect to + Tarantool. Document the Envoy config; we don't ship it. + +## Conformance + +Validate against [`connectrpc/conformance`](https://github.com/connectrpc/conformance). +Operationally identical to the protobuf conformance suite already +running here (`docker/conformance.Dockerfile`, `cmd/conformance-runner.lua`): + +- Our impl runs as a subprocess that reads framed `ClientCompatRequest` + / writes `ClientCompatResponse` on stdin/stdout. +- Two modes: `--mode server` (our impl is the server; Connect's + reference client drives it — fits `pb.grpc.transport.http_server` + validation) and `--mode client` (our impl is the client — fits + `pb.grpc.transport.http_client_unary` validation). +- One harness covers all three protocols in scope: **gRPC over HTTP/2**, + **gRPC-Web**, and **Connect**. So if HTTP/2 termination ever ships + via Envoy or otherwise, the same runner re-validates it without a + second suite. +- Coverage: unary, server-stream, client-stream, bidi, errors with + details, cancellation, deadlines, trailers, gzip/deflate compression, + TLS, HTTP/1.1 vs HTTP/2 negotiation. +- Watchlist discipline: maintain `test/grpc_conformance/known_failures.txt` + (server mode) and `..._client.txt` (client mode) mirroring the + proto suite's pattern at `test/conformance/known_failures.txt`. + +Canonical [gRPC interop tests](https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md) +(`empty_unary`, `large_unary`, `ping_pong`, …) are pre-Connect, +HTTP/2-only, and use a client-and-server-binary model rather than a +framed pipe. Skip them: less useful while we don't terminate HTTP/2, +and operationally distant from what we already run. + +## Status code mapping + +We adopt gRPC's canonical status codes (12 of them) as the cross-wire +status type. Every transport translates to and from its native error +representation: + +| gRPC status | HTTP (Connect) | net.box / IProto error | +| ------------------ | --------------- | --------------------------- | +| `OK` | 200 | success | +| `CANCELLED` | 499 | `ER_CANCELLED` | +| `INVALID_ARGUMENT` | 400 | `ER_PROC_LUA` (categorized) | +| `DEADLINE_EXCEEDED`| 504 | `ER_TIMEOUT` | +| `NOT_FOUND` | 404 | `ER_NO_SUCH_PROC` | +| `ALREADY_EXISTS` | 409 | `ER_TUPLE_FOUND` | +| `PERMISSION_DENIED`| 403 | `ER_ACCESS_DENIED` | +| `RESOURCE_EXHAUSTED`| 429 | `ER_MEMORY_ISSUE` (etc.) | +| `FAILED_PRECONDITION`| 400 | `ER_*` | +| `INTERNAL` | 500 | `ER_PROC_LUA` | +| `UNAVAILABLE` | 503 | `ER_NO_CONNECTION` | +| `UNAUTHENTICATED` | 401 | `ER_LOGIN_REQUIRED` | + +The mapping table belongs in `runtime/pb/grpc.lua`. Each transport +references it. + +## Context propagation + +The `ctx` argument in the transport contract carries metadata between +caller and transport. We standardize three keys: + +- `ctx.deadline` — fiber-clock timestamp (seconds, double). Transport + enforces by cancelling on overrun. +- `ctx.headers` — flat `{string -> string}` map. Wire-side translation + is transport-specific (HTTP headers, IProto headers, …). +- `ctx.trace_id`, `ctx.span_id` — optional tracing hooks. Transports + inject/extract per W3C `traceparent` for HTTP, custom IProto field + for net.box. + +Per-call overrides go in `ctx.options` (e.g. retry policy). User code +shouldn't put anything else in `ctx`; we may add more standard keys. + +## File / module layout + +``` +runtime/pb/grpc.lua already exists; gains status-code + + ctx-key constants +runtime/pb/grpc/http_server.lua new — Connect-style HTTP/1.1 server +runtime/pb/grpc/netbox.lua new — net.box tunnel (in-cluster) +runtime/pb/grpc/http_client.lua new — outbound, unary only +docs/grpc-howto.md new — user-facing recipes +``` + +Tests follow the existing pattern: each transport plugs into the +loopback's test harness by replacing the in-process transport with the +networked one, asserting end-to-end round-trip equality. + +## Open questions (defer) + +1. **Connect protocol version.** Connect v1 is stable; do we target the + spec verbatim, or shave it down to "POST + JSON body + JSON error" + without the framing layer? Probably full spec — clients depend on it. +2. **Streaming over HTTP/1.1.** Connect frames bidi over HTTP/1.1 + chunked transfer. Doable but adds parser surface. v1 ships unary + only and 501s on streaming; revisit when a user asks. +3. **Tarantool admin protocol surface.** Should `box.iproto.override` + carry a dedicated `IPROTO_GRPC` request type so net.box transport + doesn't sit on top of `func_call`? Probably eventually; not now. +4. **Auth.** Out of this spec. Each transport delegates to its host: + HTTP transports honor `Authorization` headers, net.box uses + Tarantool users. +5. **Metadata semantics for in-cluster.** Net.box has no concept of + metadata; we'd thread it as an extra map argument. Cleanly resolved + once `IPROTO_GRPC` is its own request type. + +## What "later" decisions look like + +When this spec gets picked back up, the load-bearing calls are: + +1. **Connect-JSON as the default external transport.** Picking it + because it works with `tarantool/http` as-is, browsers can call it + without a proxy, and gRPC-Gateway folks have a clean migration. + Revisit if a user has a hard dependency on grpc-web or REST shapes + that don't match Connect's URL convention. +2. **Don't ship HTTP/2 termination.** Push the HTTP/2 frontier to + Envoy. Revisit only if a Lua HTTP/2 library appears that's not a + sandcastle. +3. **net.box tunnel before IProto type.** Cheaper to start, retains + the door for an `IPROTO_GRPC` type later. Once net.box is in real + use, we'll know what's missing. diff --git a/docs/specs/msgpack_encoding.md b/docs/specs/msgpack_encoding.md new file mode 100644 index 0000000000000000000000000000000000000000..52371ca0f8c428e57cc1b308a1a2b5c861c8dcf6 --- /dev/null +++ b/docs/specs/msgpack_encoding.md @@ -0,0 +1,273 @@ +# Spec: msgpack encoding for protobuf schemas + +Status: **draft / brainstorm**. This is a design sketch to come back to — +not an approved plan. Open questions are called out explicitly. + +## Goal + +Use proto3 `.proto` files as the IDL, but encode/decode payloads as +**MsgPack** instead of the protobuf wire format. The bytes are valid +MsgPack consumable by `msgpackffi`, `box.tuple.new`, net.box, IProto, +and anything else in Tarantool's ecosystem — the schema simply happens +to come from a `.proto` file. + +This is **not** "encode protobuf wire format and stuff it in `MP_BIN`." +The output is structurally MsgPack throughout — maps, arrays, ints, +strings, ext types — typed by the proto descriptor. + +## Non-goals + +- Wire-compatible with anything else's "protobuf-over-msgpack" — there + is no such standard. We define ours. +- Replacing the existing proto wire codec. This is a sibling encoder, + selected per call. The hot wire codec stays exactly as it is today. +- Schema parsing changes. We reuse the existing descriptor format + (see `CLAUDE.md` → "Descriptor format"). + +## Module surface + +New module `runtime/pb/msgpack.lua`, mirroring `pb.json`'s shape: + +```lua +local pb = require('pb') + +local bytes = pb.msgpack.encode(desc, t) -- table -> msgpack bytes +local t = pb.msgpack.decode(desc, bytes) -- msgpack bytes -> table + +-- Same `desc` table the wire codec uses. Same input/output Lua shape. +-- Only the wire format on the byte side differs. +``` + +Implementation rides on `msgpackffi` (the same module net.box and +`box.tuple` use). That gets us cdata `int64_t`/`uint64_t` round-tripping +for free — consistent with the rest of the project ([CLAUDE.md] → +"64-bit integers"). + +WKT/extension routing reuses `desc.encode` / `desc.decode` overrides; +we add a parallel `desc.msgpack_encode` / `desc.msgpack_decode` pair so +WKTs can plug into both codecs side-by-side. + +## Default message layout: msgpack map keyed by field number + +```text +message Foo { + int32 a = 1; + string b = 2; + Bar c = 3; +} + +{ a = 7, b = "hi", c = {...} } + --> mp_map{ 1: 7, 2: "hi", 3: mp_map{...} } +``` + +- **Int keys, not string keys.** Field numbers are the proto identity; + names are cosmetic. Int keys also pack tighter in msgpack + (1 byte for fields 1–127). +- **Unknown fields survive round-trip.** A decoder that doesn't know + field 999 keeps the `{999: }` entry in a sidecar and + re-emits it on encode. Mirrors how the proto wire codec preserves + unknown fields today, except the values are msgpack-typed instead of + raw wire bytes. +- **Defaults are omitted on encode.** Proto3 zero-values do not appear + in the map. Decoder fills them in. Explicit `optional` fields encode + when set, omit when unset — presence = "is the int key present?" + +Tradeoff: not human-readable in raw form. Mitigated by a debug helper +that joins the map against the descriptor when printing. + +## Scalar mapping + +| proto type | msgpack | notes | +| ------------------------------------------ | ------------------- | ------------------------------------------- | +| `int32`, `sint32`, `sfixed32` | `mp_int` | signed | +| `uint32`, `fixed32` | `mp_uint` | unsigned | +| `int64`, `sint64`, `sfixed64` | `mp_int` | cdata `int64_t`, no narrowing to double | +| `uint64`, `fixed64` | `mp_uint` | cdata `uint64_t` | +| `float` | `mp_float32` | | +| `double` | `mp_float64` | | +| `bool` | `mp_bool` | | +| `string` | `mp_str` | UTF-8 validation per proto3 (reuse `utf8.len`) | +| `bytes` | `mp_bin` | | +| `enum` | `mp_int` (numeric) | keep numeric so unknown enum values survive | + +The "fixed" / "varint" / "zigzag" distinction is wire-encoding-specific +and irrelevant here — every integer goes through the same `mp_int` path. + +## Composite mapping + +- **`repeated T`** → `mp_array` of T-encoded values. The packed-vs- + unpacked distinction disappears; every repeated field is an array. +- **`map`** → `mp_map` with the actual map keys. Not a list of + `{key, value}` entries — a real map. K must be a scalar per proto3. +- **Nested message** → nested `mp_map` (embedded, not a bytes blob). +- **`oneof`** → only the active branch's field-number key appears in + the parent map. Decoder reconstructs which branch is active from + "which key is present." Setting a new branch on encode drops the + others (same as the wire codec). + +## Field presence summary + +| field shape | encoded when value is... | decoder default | +| -------------------------- | ------------------------ | -------------------------- | +| proto3 implicit (no `optional`) | non-default | proto3 zero | +| proto3 explicit `optional` | set | absent (key not in result) | +| `repeated` | non-empty | empty array `{}` | +| `map` | non-empty | empty map `{}` | + +## Unknown-field preservation + +Decoder collects `int_key -> raw mp value (bytes)` for any int key not +in `desc.field_by_id`, stashes them under `t._unknown_fields_msgpack` +(distinct from the wire codec's `t._unknown_fields`, which holds raw +wire bytes). Encoder re-emits the stashed entries verbatim. + +Open: do we want bidirectional unknown-field passthrough between the +two codecs (msgpack ↔ wire)? Probably no — the codecs aren't paired. +A wire-format message that hit a msgpack decoder is malformed by +definition; if you want both you write twice or transcode explicitly. + +## WKT mapping + +| WKT | msgpack | rationale | +| ---------------------------- | ------------------------------------ | ----------------------------------------------- | +| `google.protobuf.Timestamp` | `mp_ext`/MP_DATETIME (ext 4) | Native datetime in Tarantool; box-space friendly | +| `google.protobuf.Duration` | `mp_ext`/MP_INTERVAL (ext 6) | Native interval | +| `google.protobuf.Empty` | empty `mp_map` | Trivial | +| `google.protobuf.*Value` wrappers | the raw scalar | Presence already captured by "key present in parent" — wrapper layer is redundant | +| `google.protobuf.Struct` | `mp_map` (string keys, dynamic values) | `Struct` is JSON-shaped by design | +| `google.protobuf.Value` | native msgpack of matching shape | Same | +| `google.protobuf.ListValue` | `mp_array` | Same | +| `google.protobuf.FieldMask` | `mp_array` of `mp_str` (the paths) | Just a path list | +| `google.protobuf.Any` | `mp_map{ type_url = mp_str, value = mp_bin }` | Keep `value` as raw protobuf wire bytes — type_url's contract is "value is proto wire of that type" | + +### Timestamp ↔ MP_DATETIME loss + +`struct datetime` (Tarantool, `src/lib/core/datetime.h:85`): +```c +double epoch; int32_t nsec; +int16_t tzoffset; int16_t tzindex; +``` +`google.protobuf.Timestamp` is `(seconds, nanos)` in UTC. + +- **proto → mp**: emit datetime ext with `tzoffset = 0, tzindex = 0`. + Lossless. +- **mp → proto**: drop tz fields, keep the UTC instant. Loses any + non-UTC presentation hint, preserves the actual instant. Benign for + the Timestamp contract. + +Same shape applies to Duration ↔ MP_INTERVAL (Tarantool's interval +struct also has component breakdown beyond seconds+nanos). + +## Tarantool-specific options + +New options on `options/tarantool/tarantool.proto`. Sketch — names +subject to change: + +```proto +// File or message option. +enum MsgpackLayout { + MAP = 0; // default — mp_map keyed by field number + ARRAY = 1; // mp_array positional by field number, holes filled with mp_nil +} + +extend google.protobuf.FileOptions { + MsgpackLayout msgpack_layout = 60010; +} + +extend google.protobuf.MessageOptions { + MsgpackLayout msgpack_layout = 60011; // overrides file default + // Reserved tuple-format id; only honored when msgpack_layout = ARRAY + // and msgpack_tuple_ext = true. + uint32 tuple_format_id = 60012; + bool msgpack_tuple_ext = 60013; // emit MP_TUPLE (ext 7) instead of bare mp_array +} +``` + +### Three target shapes for top-level messages + +| option combination | output | use case | +| ----------------------------------------------------- | ------------------------------------------------ | --------------------------------- | +| `msgpack_layout = MAP` (default) | `mp_map{ 1: ..., 2: ... }` | RPC payload, log entry, anything where sparse fields and unknown-field passthrough matter | +| `msgpack_layout = ARRAY` | `mp_array{ ..., ..., ... }` padded with `mp_nil` | Drop-in for `box.space:replace{...}` | +| `msgpack_layout = ARRAY` + `msgpack_tuple_ext = true` | `mp_ext`/MP_TUPLE (ext 7), `{format_id, array}` | Typed tuple, requires a registered `tuple_format_id` | + +ARRAY layout caveats: +- Sparse high field numbers waste bytes (lots of `mp_nil` padding). + Reasonable for messages whose field numbers are dense and stable. + The author opted in; this is on them. +- Unknown-field passthrough is impossible — any field number past the + array length is data loss. Encoder must error if the input has + `_unknown_fields_msgpack` entries. +- MP_TUPLE ext requires you've registered the format somewhere + reachable to the decoder. Without that registration the bytes are + inert. Defer the registration story until someone needs it. + +## Where things live + +``` +runtime/pb/msgpack.lua new — encoder/decoder +options/tarantool/tarantool.proto extended with msgpack_* options +cmd/protoc-gen-tarantool/internal/gen/options.go parse new options +test/msgpack_test.lua new — round-trip + WKT + option matrix +test/interop/fixtures/*.mp optional — golden bytes per fixture +``` + +The plugin does **not** generate msgpack-specific helpers per message — +`pb.msgpack.encode(desc, t)` walks the descriptor at runtime, same as +runtime-mode wire codec does today. Full-mode wire codec inlines for +JIT reasons; msgpack codec doesn't need that complexity yet (the +msgpackffi C-level encoder does the heavy lifting on the byte side). + +If perf becomes a concern, a follow-up could emit +`M._msgpack_encode` inlined per-message, mirroring the wire codec's +full mode. Out of scope here. + +## Tests + +- Round-trip: every existing `test/interop/fixtures/*.txtpb` decoded + via proto wire codec, re-encoded via msgpack codec, decoded via + msgpack codec, asserted equal to the original Lua table. +- Parity: msgpack-encoded then transcoded back to wire format equals + the original `.bin` golden. Same parity discipline as the + full-vs-runtime suite. +- WKT: explicit fixtures for Timestamp/Duration/Struct/Any covering + the ext-type bridges. +- Layout: a few messages with both `MAP` and `ARRAY` layouts; assert + shape on the byte side via `msgpackffi.decode_unchecked` introspection. +- Unknown-field passthrough: MAP-layout only; ARRAY-layout errors. + +## Open questions (defer) + +1. **Does proto-side `repeated` of a WKT survive MP_DATETIME packing?** + `mp_array{ mp_ext, mp_ext, ... }` should just work. Worth a fixture. +2. **Custom ext types beyond WKTs.** Users may want their own Lua + types (UUID, decimal) to ride the same ext channel. Probably + another `(tarantool.msgpack_ext)` field option on `bytes` fields. + Defer until someone asks. +3. **Schema evolution on ARRAY layout.** Renumbering fields breaks + ARRAY consumers silently. Should the plugin refuse to compile ARRAY + messages with gaps, or just warn? Probably refuse. +4. **JSON ↔ msgpack interop.** Existing `pb.json` and proposed + `pb.msgpack` are both alternate codecs over the same descriptor; + should they share a "set this Lua table from any codec" entry + point? Probably yes (`pb.from(desc, bytes, format)`) but not part + of this slice. +5. **Streaming.** Wire codec is one-shot; msgpack is too. Streaming + adds complexity disproportionate to current needs. Skip. + +## What "later" decisions look like + +When picking this back up, the load-bearing calls are: + +1. **MAP keyed by int vs by name.** Picking int now; revisit if + real users find it intolerable to debug. Switching is a one-line + change in the encoder. +2. **ARRAY + MP_TUPLE story.** Whether to ship at all in v1, or punt + to a follow-up. ARRAY-as-`mp_array` is enough for `box.space` + feeders; MP_TUPLE requires a format registry we don't have. +3. **Wrapper unwrapping.** Some users may want `Int32Value` to stay + as `{value = N}` because their proto-side code treats it as a + distinct presence-bearing type. Unwrapping is opinionated. + +Everything else in this doc is mechanical and follows from those +choices.