M PLAN.md => PLAN.md +21 -6
@@ 131,10 131,21 @@ release on sourcecraft.dev.
client_stream: function(stream, ctx) -> resp
bidi: function(stream, ctx)
```
-- [ ] Reference network transports (separate projects, deferred):
+- [ ] Reference network transports (separate projects, deferred). Full
+ design in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md):
+ - `pb.grpc.transport.http_server` — Connect-JSON over HTTP/1.1 via
+ `tarantool/http`. Default external transport; works with browsers
+ and `curl` without an HTTP/2 proxy.
- `pb.grpc.transport.netbox` — gRPC tunneled over `net.box` calls.
- - HTTP/2 transport — out of scope; the transport interface above is
- deliberately HTTP/2-shaped so a plug-in is straightforward.
+ First-class in-cluster path.
+ - `pb.grpc.transport.http_client_unary` — outbound, unary only,
+ via `http_client`.
+ - HTTP/2 termination — explicitly *not* shipped. Recommend Envoy in
+ front; the transport contract is HTTP/2-shaped so the same
+ generated code works behind it.
+ - Conformance anchor: `connectrpc/conformance` (same framed-runner
+ shape as protobuf conformance — covers gRPC, gRPC-Web, and Connect
+ from one harness).
**Done.** The loopback transport runs each streaming handler on its own
fiber and bridges client ↔ handler via `fiber.channel`. All four flavors
@@ 582,7 593,8 @@ Run on a fixed corpus across 5 message sizes; track results in
|---|---|
| Should map encoder be deterministic (sorted by key)? | Spec says no; some users want yes. Add `pb.encode_deterministic` flag later. |
| Public C-FFI accelerator for varint? | Defer until perf benchmarks show pure-Lua bottleneck. |
-| Should we ship a stub HTTP/2 transport for gRPC? | Probably no — large dependency. Recommend `lua-http` or write a focused project. |
+| Should we ship a stub HTTP/2 transport for gRPC? | Answered in [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md): no — recommend Envoy in front, ship Connect-JSON over HTTP/1.1 as the default external transport. |
+| msgpack-flavored encoder for proto schemas? | Design sketched in [`docs/specs/msgpack_encoding.md`](docs/specs/msgpack_encoding.md). Open: map-keyed-by-int (default) vs name; ARRAY layout for `box.space` feeders; MP_TUPLE ext opt-in. |
| How to handle the existing Tarantool-builtin `require('protobuf')`? | Document migration; do not override the loader. |
| Lua module path mapping when no `lua_package` and no proto package? | Currently uses bare filename; consider erroring out instead. |
| Schema upgrade support (v1 → v2 of a message)? | Out of scope; protobuf is forward-compatible by design. |
@@ 593,9 605,12 @@ Run on a fixed corpus across 5 message sizes; track results in
- **proto2** syntax — spec is more complex (required, default values,
groups), real demand is rare on Tarantool.
- **Protobuf editions** beyond what proto3 enables.
-- **HTTP/2 transport** for gRPC (separate project).
+- **HTTP/2 termination** for gRPC. Push it to Envoy; see
+ [`docs/specs/grpc_transports.md`](docs/specs/grpc_transports.md).
+ Connect-JSON over HTTP/1.1 is the default external transport.
- **Reflection service** (gRPC server reflection) — implement after M5.
-- **gRPC-Web** — separate project.
+- **gRPC-Web** — covered by the same Connect conformance harness if we
+ add it, but not on the v1 path.
## 9. How to update this plan
A docs/specs/grpc_transports.md => docs/specs/grpc_transports.md +292 -0
@@ 0,0 1,292 @@
+# Spec: gRPC transports for Tarantool
+
+Status: **draft / decision deferred**. The transport *contract* is
+shipped and stable (see [`runtime/pb/grpc.lua`](../../runtime/pb/grpc.lua)).
+What's open is **which concrete transports we recommend and/or ship,
+and how a user picks between them**.
+
+This spec maps the protocol landscape, says where Tarantool fits, and
+flags what we'd build vs. recommend an external library for.
+
+## What's already shipped (and won't change)
+
+Generated `M.<Service>_client(transport)` and `M.<Service>_server(impl)`
+talk to the transport in a single contract, regardless of wire protocol:
+
+```
+transport:unary(path, req_bytes, ctx) -> resp_bytes
+transport:server_stream(path, req_bytes, ctx) -> stream
+transport:client_stream(path, ctx) -> stream
+transport:bidi(path, ctx) -> stream
+```
+
+`path` is `/pkg.Service/Method`. `ctx` is an opaque Lua table (headers,
+deadline, metadata, …). Generated code encodes the request, hands raw
+bytes to the transport, and decodes the response.
+
+This is deliberately HTTP/2-shaped — `path`, byte-oriented messages,
+streams — but **the contract makes no commitment to a wire protocol**.
+Every transport in this spec is a different plug-in behind the same
+four methods.
+
+Reference transports already in `runtime/pb/grpc.lua`:
+
+- `pb.grpc.loopback(server)` — in-process; uses `fiber.channel`. Bridges
+ client → server fiber for tests and same-process apps.
+- `pb.grpc.multiplex({srv1, srv2})` — fans several servers onto one
+ transport. Errors on duplicate paths.
+
+## The protocol matrix
+
+For a request that crosses a process boundary, you pick a pair: a
+**wire protocol** (how bytes flow between processes) and a **codec**
+(how a message turns into bytes). The Lua-side transport bridges between
+the generated code and that wire/codec combo.
+
+| Wire protocol | Body codec | Streaming | Status signaling | Browser-friendly | Off-the-shelf clients/servers | Tarantool fit |
+| ------------------------- | ------------ | ------------------- | ------------------------- | :--------------: | ----------------------------- | ------------------------------ |
+| **gRPC over HTTP/2** | proto wire | unary + all 3 | HTTP/2 trailers | no | every gRPC lib | needs an external HTTP/2 lib |
+| **gRPC-Web over HTTP/2** | proto wire | unary + server | trailers in body | yes (with proxy) | grpc-web JS, Envoy | same problem as gRPC + framing |
+| **gRPC-Web over HTTP/1.1**| proto wire | unary + server | trailers in body | yes | grpc-web JS | works with `tarantool/http` |
+| **Connect over HTTP/1.1** | proto wire | unary only | HTTP status + body | yes | connectrpc clients | works with `tarantool/http` |
+| **Connect over HTTP/2** | proto wire | unary + all 3 | HTTP status / trailers | yes | connectrpc clients | same HTTP/2 problem |
+| **Connect-JSON** | proto3 JSON | unary only | HTTP status + body | yes | curl + connectrpc clients | drop-in for `tarantool/http` |
+| **gRPC-Gateway / transcoded REST + JSON** | proto3 JSON | unary | HTTP status + body | yes | any HTTP client | drop-in for `tarantool/http` |
+| **gRPC over IProto tunnel** (Tarantool-native) | proto wire | unary + all 3 | IProto error code | n/a | this project, custom clients | first-class |
+| **gRPC over net.box tunnel** | proto wire | unary + all 3 | net.box error | n/a | this project, custom clients | first-class |
+
+What's **not** in the matrix and why:
+
+- **gRPC over HTTP/3 / QUIC** — too early; the Go and C++ gRPC stacks
+ themselves treat it as experimental. Not worth specifying yet.
+- **JSON-RPC, Thrift, etc.** — different IDL; off-topic.
+
+## What "Tarantool fit" actually means
+
+Tarantool gives us:
+
+- **`tarantool/http` server (HTTP/1.1)** — solid, idiomatic, lives in a
+ Lua rock. No HTTP/2, no server-pushed trailers. Good substrate for
+ Connect-JSON and gRPC-Gateway-style REST.
+- **`http_client` (libcurl-based)** — HTTP/1.1 and HTTP/2 client. Has
+ streaming via callbacks, but trailers and gRPC framing are not
+ first-class. Usable for HTTP/2 unary; awkward for streaming.
+- **`net.box`** — Tarantool's binary RPC protocol. Already gives us
+ request/response, streaming via long-poll, error propagation. Sane
+ default for in-cluster Tarantool→Tarantool calls.
+- **IProto** — the wire protocol under net.box. Lower level; lets us
+ define our own request type that carries gRPC framing if we want
+ zero overhead.
+- **No HTTP/2 server.** Real HTTP/2 termination needs a sidecar (Envoy,
+ nginx) or a new Lua library. Neither is something we'd ship.
+
+So the practical bands are:
+
+1. **In-cluster Tarantool↔Tarantool** → IProto or net.box tunnel.
+2. **External clients calling Tarantool over HTTP** → Connect-JSON or
+ gRPC-Gateway-style REST behind `tarantool/http`. Both are HTTP/1.1
+ only, both speak JSON, both work with `curl`/browsers without a
+ proxy.
+3. **External clients that insist on real gRPC** → put Envoy or
+ grpcurl in front, terminate HTTP/2 there, and send unary calls into
+ Tarantool over HTTP/1.1 or IProto. We don't terminate HTTP/2
+ ourselves.
+4. **Outbound calls from Tarantool to external gRPC services** →
+ `http_client` for HTTP/2 unary. For streaming, accept "we don't
+ support that yet" rather than shipping a half-baked HTTP/2 client.
+
+## Recommended transports to build
+
+Names below refer to packages we'd publish; nothing here lives in this
+repo yet beyond the contract.
+
+### `pb.grpc.transport.http_server` (HTTP/1.1 server-side)
+
+Adapts an `M.<Service>_server(impl)` result into a `tarantool/http`
+route handler. Wire protocol: **Connect-JSON over HTTP/1.1** by default,
+with content-negotiation for Connect-protobuf.
+
+- POST `/{package.Service}/{Method}` with `Content-Type: application/json`
+ → decode body via `pb.json.decode(input_desc, body)`, call the
+ generated handler, encode reply via `pb.json.encode`.
+- `application/proto` content type → use `pb.encode/pb.decode` instead.
+- Streaming methods: respond 501 for now. Connect's `application/connect+json`
+ framed streaming over HTTP/1.1 chunked transfer is feasible later;
+ out of scope for v1.
+- Errors: surface as Connect's JSON error envelope. Map common
+ gRPC status codes to HTTP status per the Connect spec.
+
+Why this first: it's the lowest-effort transport that gives us a real
+external interface, and it works with browsers and `curl`. It also
+covers the gRPC-Gateway use case without needing the gateway:
+`POST /myapp.v1.Greeter/SayHello` with a JSON body is a fine REST
+shape on its own.
+
+### `pb.grpc.transport.netbox` (in-cluster)
+
+`net.box` connection → speaks `pb.grpc` over a single user-defined
+function (e.g. `box.schema.func.create('grpc_dispatch')`). Body is
+a 2-tuple `{path, req_bytes}`; reply is `{ok, resp_bytes}` or
+`{err, status_code, message}`.
+
+Streaming: lean on net.box's stream/iterator support. Server runs the
+handler on a fiber; messages flow through `box.iproto.override` /
+`box.session.push`. Concretely tractable; out of scope for v1 but
+straightforward to add.
+
+Why second: in-cluster Tarantool clusters are a real and ready use
+case. The transport is small and self-contained.
+
+### `pb.grpc.transport.http_client_unary` (outbound, optional)
+
+Adapts `http_client` to speak Connect-JSON or Connect-protobuf to
+external services. Unary only. Easy. Useful for calling out from
+Tarantool app code to a Connect or HTTP/1.1 gRPC-Web server.
+
+## Not recommended (don't build)
+
+- **Tarantool-side HTTP/2 server.** Would require a new HTTP/2 library
+ in Lua or a C module. Effort vastly exceeds payoff — anyone needing
+ HTTP/2 termination should run Envoy in front. Document the Envoy
+ setup instead.
+- **Tarantool-side HTTP/2 streaming client.** `http_client`'s streaming
+ API isn't a clean fit for gRPC trailers and per-message framing.
+ Anyone needing this should bind to a real gRPC client (C, Go), not
+ reimplement in Lua. Document this limitation.
+- **gRPC-Web framing.** It's a small spec, but every modern stack
+ (browser SDK, mobile SDK) prefers Connect now. Don't fragment effort
+ unless a user demands it.
+
+## How a user picks
+
+Decision tree, top-down:
+
+1. Both endpoints in a Tarantool cluster?
+ → `pb.grpc.transport.netbox`.
+2. External clients only (browsers, curl, mobile)?
+ → `pb.grpc.transport.http_server` (Connect-JSON).
+3. Need to call an external gRPC service from Tarantool?
+ → Unary: `pb.grpc.transport.http_client_unary` (Connect or HTTP/1.1
+ gateway). Streaming: not supported; document the Envoy/sidecar
+ alternative.
+4. External clients insist on real HTTP/2 gRPC?
+ → Envoy in front. Envoy terminates HTTP/2, talks Connect to
+ Tarantool. Document the Envoy config; we don't ship it.
+
+## Conformance
+
+Validate against [`connectrpc/conformance`](https://github.com/connectrpc/conformance).
+Operationally identical to the protobuf conformance suite already
+running here (`docker/conformance.Dockerfile`, `cmd/conformance-runner.lua`):
+
+- Our impl runs as a subprocess that reads framed `ClientCompatRequest`
+ / writes `ClientCompatResponse` on stdin/stdout.
+- Two modes: `--mode server` (our impl is the server; Connect's
+ reference client drives it — fits `pb.grpc.transport.http_server`
+ validation) and `--mode client` (our impl is the client — fits
+ `pb.grpc.transport.http_client_unary` validation).
+- One harness covers all three protocols in scope: **gRPC over HTTP/2**,
+ **gRPC-Web**, and **Connect**. So if HTTP/2 termination ever ships
+ via Envoy or otherwise, the same runner re-validates it without a
+ second suite.
+- Coverage: unary, server-stream, client-stream, bidi, errors with
+ details, cancellation, deadlines, trailers, gzip/deflate compression,
+ TLS, HTTP/1.1 vs HTTP/2 negotiation.
+- Watchlist discipline: maintain `test/grpc_conformance/known_failures.txt`
+ (server mode) and `..._client.txt` (client mode) mirroring the
+ proto suite's pattern at `test/conformance/known_failures.txt`.
+
+Canonical [gRPC interop tests](https://github.com/grpc/grpc/blob/master/doc/interop-test-descriptions.md)
+(`empty_unary`, `large_unary`, `ping_pong`, …) are pre-Connect,
+HTTP/2-only, and use a client-and-server-binary model rather than a
+framed pipe. Skip them: less useful while we don't terminate HTTP/2,
+and operationally distant from what we already run.
+
+## Status code mapping
+
+We adopt gRPC's canonical status codes (12 of them) as the cross-wire
+status type. Every transport translates to and from its native error
+representation:
+
+| gRPC status | HTTP (Connect) | net.box / IProto error |
+| ------------------ | --------------- | --------------------------- |
+| `OK` | 200 | success |
+| `CANCELLED` | 499 | `ER_CANCELLED` |
+| `INVALID_ARGUMENT` | 400 | `ER_PROC_LUA` (categorized) |
+| `DEADLINE_EXCEEDED`| 504 | `ER_TIMEOUT` |
+| `NOT_FOUND` | 404 | `ER_NO_SUCH_PROC` |
+| `ALREADY_EXISTS` | 409 | `ER_TUPLE_FOUND` |
+| `PERMISSION_DENIED`| 403 | `ER_ACCESS_DENIED` |
+| `RESOURCE_EXHAUSTED`| 429 | `ER_MEMORY_ISSUE` (etc.) |
+| `FAILED_PRECONDITION`| 400 | `ER_*` |
+| `INTERNAL` | 500 | `ER_PROC_LUA` |
+| `UNAVAILABLE` | 503 | `ER_NO_CONNECTION` |
+| `UNAUTHENTICATED` | 401 | `ER_LOGIN_REQUIRED` |
+
+The mapping table belongs in `runtime/pb/grpc.lua`. Each transport
+references it.
+
+## Context propagation
+
+The `ctx` argument in the transport contract carries metadata between
+caller and transport. We standardize three keys:
+
+- `ctx.deadline` — fiber-clock timestamp (seconds, double). Transport
+ enforces by cancelling on overrun.
+- `ctx.headers` — flat `{string -> string}` map. Wire-side translation
+ is transport-specific (HTTP headers, IProto headers, …).
+- `ctx.trace_id`, `ctx.span_id` — optional tracing hooks. Transports
+ inject/extract per W3C `traceparent` for HTTP, custom IProto field
+ for net.box.
+
+Per-call overrides go in `ctx.options` (e.g. retry policy). User code
+shouldn't put anything else in `ctx`; we may add more standard keys.
+
+## File / module layout
+
+```
+runtime/pb/grpc.lua already exists; gains status-code
+ + ctx-key constants
+runtime/pb/grpc/http_server.lua new — Connect-style HTTP/1.1 server
+runtime/pb/grpc/netbox.lua new — net.box tunnel (in-cluster)
+runtime/pb/grpc/http_client.lua new — outbound, unary only
+docs/grpc-howto.md new — user-facing recipes
+```
+
+Tests follow the existing pattern: each transport plugs into the
+loopback's test harness by replacing the in-process transport with the
+networked one, asserting end-to-end round-trip equality.
+
+## Open questions (defer)
+
+1. **Connect protocol version.** Connect v1 is stable; do we target the
+ spec verbatim, or shave it down to "POST + JSON body + JSON error"
+ without the framing layer? Probably full spec — clients depend on it.
+2. **Streaming over HTTP/1.1.** Connect frames bidi over HTTP/1.1
+ chunked transfer. Doable but adds parser surface. v1 ships unary
+ only and 501s on streaming; revisit when a user asks.
+3. **Tarantool admin protocol surface.** Should `box.iproto.override`
+ carry a dedicated `IPROTO_GRPC` request type so net.box transport
+ doesn't sit on top of `func_call`? Probably eventually; not now.
+4. **Auth.** Out of this spec. Each transport delegates to its host:
+ HTTP transports honor `Authorization` headers, net.box uses
+ Tarantool users.
+5. **Metadata semantics for in-cluster.** Net.box has no concept of
+ metadata; we'd thread it as an extra map argument. Cleanly resolved
+ once `IPROTO_GRPC` is its own request type.
+
+## What "later" decisions look like
+
+When this spec gets picked back up, the load-bearing calls are:
+
+1. **Connect-JSON as the default external transport.** Picking it
+ because it works with `tarantool/http` as-is, browsers can call it
+ without a proxy, and gRPC-Gateway folks have a clean migration.
+ Revisit if a user has a hard dependency on grpc-web or REST shapes
+ that don't match Connect's URL convention.
+2. **Don't ship HTTP/2 termination.** Push the HTTP/2 frontier to
+ Envoy. Revisit only if a Lua HTTP/2 library appears that's not a
+ sandcastle.
+3. **net.box tunnel before IProto type.** Cheaper to start, retains
+ the door for an `IPROTO_GRPC` type later. Once net.box is in real
+ use, we'll know what's missing.
A docs/specs/msgpack_encoding.md => docs/specs/msgpack_encoding.md +273 -0
@@ 0,0 1,273 @@
+# Spec: msgpack encoding for protobuf schemas
+
+Status: **draft / brainstorm**. This is a design sketch to come back to —
+not an approved plan. Open questions are called out explicitly.
+
+## Goal
+
+Use proto3 `.proto` files as the IDL, but encode/decode payloads as
+**MsgPack** instead of the protobuf wire format. The bytes are valid
+MsgPack consumable by `msgpackffi`, `box.tuple.new`, net.box, IProto,
+and anything else in Tarantool's ecosystem — the schema simply happens
+to come from a `.proto` file.
+
+This is **not** "encode protobuf wire format and stuff it in `MP_BIN`."
+The output is structurally MsgPack throughout — maps, arrays, ints,
+strings, ext types — typed by the proto descriptor.
+
+## Non-goals
+
+- Wire-compatible with anything else's "protobuf-over-msgpack" — there
+ is no such standard. We define ours.
+- Replacing the existing proto wire codec. This is a sibling encoder,
+ selected per call. The hot wire codec stays exactly as it is today.
+- Schema parsing changes. We reuse the existing descriptor format
+ (see `CLAUDE.md` → "Descriptor format").
+
+## Module surface
+
+New module `runtime/pb/msgpack.lua`, mirroring `pb.json`'s shape:
+
+```lua
+local pb = require('pb')
+
+local bytes = pb.msgpack.encode(desc, t) -- table -> msgpack bytes
+local t = pb.msgpack.decode(desc, bytes) -- msgpack bytes -> table
+
+-- Same `desc` table the wire codec uses. Same input/output Lua shape.
+-- Only the wire format on the byte side differs.
+```
+
+Implementation rides on `msgpackffi` (the same module net.box and
+`box.tuple` use). That gets us cdata `int64_t`/`uint64_t` round-tripping
+for free — consistent with the rest of the project ([CLAUDE.md] →
+"64-bit integers").
+
+WKT/extension routing reuses `desc.encode` / `desc.decode` overrides;
+we add a parallel `desc.msgpack_encode` / `desc.msgpack_decode` pair so
+WKTs can plug into both codecs side-by-side.
+
+## Default message layout: msgpack map keyed by field number
+
+```text
+message Foo {
+ int32 a = 1;
+ string b = 2;
+ Bar c = 3;
+}
+
+{ a = 7, b = "hi", c = {...} }
+ --> mp_map{ 1: 7, 2: "hi", 3: mp_map{...} }
+```
+
+- **Int keys, not string keys.** Field numbers are the proto identity;
+ names are cosmetic. Int keys also pack tighter in msgpack
+ (1 byte for fields 1–127).
+- **Unknown fields survive round-trip.** A decoder that doesn't know
+ field 999 keeps the `{999: <opaque mp value>}` entry in a sidecar and
+ re-emits it on encode. Mirrors how the proto wire codec preserves
+ unknown fields today, except the values are msgpack-typed instead of
+ raw wire bytes.
+- **Defaults are omitted on encode.** Proto3 zero-values do not appear
+ in the map. Decoder fills them in. Explicit `optional` fields encode
+ when set, omit when unset — presence = "is the int key present?"
+
+Tradeoff: not human-readable in raw form. Mitigated by a debug helper
+that joins the map against the descriptor when printing.
+
+## Scalar mapping
+
+| proto type | msgpack | notes |
+| ------------------------------------------ | ------------------- | ------------------------------------------- |
+| `int32`, `sint32`, `sfixed32` | `mp_int` | signed |
+| `uint32`, `fixed32` | `mp_uint` | unsigned |
+| `int64`, `sint64`, `sfixed64` | `mp_int` | cdata `int64_t`, no narrowing to double |
+| `uint64`, `fixed64` | `mp_uint` | cdata `uint64_t` |
+| `float` | `mp_float32` | |
+| `double` | `mp_float64` | |
+| `bool` | `mp_bool` | |
+| `string` | `mp_str` | UTF-8 validation per proto3 (reuse `utf8.len`) |
+| `bytes` | `mp_bin` | |
+| `enum` | `mp_int` (numeric) | keep numeric so unknown enum values survive |
+
+The "fixed" / "varint" / "zigzag" distinction is wire-encoding-specific
+and irrelevant here — every integer goes through the same `mp_int` path.
+
+## Composite mapping
+
+- **`repeated T`** → `mp_array` of T-encoded values. The packed-vs-
+ unpacked distinction disappears; every repeated field is an array.
+- **`map<K, V>`** → `mp_map` with the actual map keys. Not a list of
+ `{key, value}` entries — a real map. K must be a scalar per proto3.
+- **Nested message** → nested `mp_map` (embedded, not a bytes blob).
+- **`oneof`** → only the active branch's field-number key appears in
+ the parent map. Decoder reconstructs which branch is active from
+ "which key is present." Setting a new branch on encode drops the
+ others (same as the wire codec).
+
+## Field presence summary
+
+| field shape | encoded when value is... | decoder default |
+| -------------------------- | ------------------------ | -------------------------- |
+| proto3 implicit (no `optional`) | non-default | proto3 zero |
+| proto3 explicit `optional` | set | absent (key not in result) |
+| `repeated` | non-empty | empty array `{}` |
+| `map` | non-empty | empty map `{}` |
+
+## Unknown-field preservation
+
+Decoder collects `int_key -> raw mp value (bytes)` for any int key not
+in `desc.field_by_id`, stashes them under `t._unknown_fields_msgpack`
+(distinct from the wire codec's `t._unknown_fields`, which holds raw
+wire bytes). Encoder re-emits the stashed entries verbatim.
+
+Open: do we want bidirectional unknown-field passthrough between the
+two codecs (msgpack ↔ wire)? Probably no — the codecs aren't paired.
+A wire-format message that hit a msgpack decoder is malformed by
+definition; if you want both you write twice or transcode explicitly.
+
+## WKT mapping
+
+| WKT | msgpack | rationale |
+| ---------------------------- | ------------------------------------ | ----------------------------------------------- |
+| `google.protobuf.Timestamp` | `mp_ext`/MP_DATETIME (ext 4) | Native datetime in Tarantool; box-space friendly |
+| `google.protobuf.Duration` | `mp_ext`/MP_INTERVAL (ext 6) | Native interval |
+| `google.protobuf.Empty` | empty `mp_map` | Trivial |
+| `google.protobuf.*Value` wrappers | the raw scalar | Presence already captured by "key present in parent" — wrapper layer is redundant |
+| `google.protobuf.Struct` | `mp_map` (string keys, dynamic values) | `Struct` is JSON-shaped by design |
+| `google.protobuf.Value` | native msgpack of matching shape | Same |
+| `google.protobuf.ListValue` | `mp_array` | Same |
+| `google.protobuf.FieldMask` | `mp_array` of `mp_str` (the paths) | Just a path list |
+| `google.protobuf.Any` | `mp_map{ type_url = mp_str, value = mp_bin }` | Keep `value` as raw protobuf wire bytes — type_url's contract is "value is proto wire of that type" |
+
+### Timestamp ↔ MP_DATETIME loss
+
+`struct datetime` (Tarantool, `src/lib/core/datetime.h:85`):
+```c
+double epoch; int32_t nsec;
+int16_t tzoffset; int16_t tzindex;
+```
+`google.protobuf.Timestamp` is `(seconds, nanos)` in UTC.
+
+- **proto → mp**: emit datetime ext with `tzoffset = 0, tzindex = 0`.
+ Lossless.
+- **mp → proto**: drop tz fields, keep the UTC instant. Loses any
+ non-UTC presentation hint, preserves the actual instant. Benign for
+ the Timestamp contract.
+
+Same shape applies to Duration ↔ MP_INTERVAL (Tarantool's interval
+struct also has component breakdown beyond seconds+nanos).
+
+## Tarantool-specific options
+
+New options on `options/tarantool/tarantool.proto`. Sketch — names
+subject to change:
+
+```proto
+// File or message option.
+enum MsgpackLayout {
+ MAP = 0; // default — mp_map keyed by field number
+ ARRAY = 1; // mp_array positional by field number, holes filled with mp_nil
+}
+
+extend google.protobuf.FileOptions {
+ MsgpackLayout msgpack_layout = 60010;
+}
+
+extend google.protobuf.MessageOptions {
+ MsgpackLayout msgpack_layout = 60011; // overrides file default
+ // Reserved tuple-format id; only honored when msgpack_layout = ARRAY
+ // and msgpack_tuple_ext = true.
+ uint32 tuple_format_id = 60012;
+ bool msgpack_tuple_ext = 60013; // emit MP_TUPLE (ext 7) instead of bare mp_array
+}
+```
+
+### Three target shapes for top-level messages
+
+| option combination | output | use case |
+| ----------------------------------------------------- | ------------------------------------------------ | --------------------------------- |
+| `msgpack_layout = MAP` (default) | `mp_map{ 1: ..., 2: ... }` | RPC payload, log entry, anything where sparse fields and unknown-field passthrough matter |
+| `msgpack_layout = ARRAY` | `mp_array{ ..., ..., ... }` padded with `mp_nil` | Drop-in for `box.space:replace{...}` |
+| `msgpack_layout = ARRAY` + `msgpack_tuple_ext = true` | `mp_ext`/MP_TUPLE (ext 7), `{format_id, array}` | Typed tuple, requires a registered `tuple_format_id` |
+
+ARRAY layout caveats:
+- Sparse high field numbers waste bytes (lots of `mp_nil` padding).
+ Reasonable for messages whose field numbers are dense and stable.
+ The author opted in; this is on them.
+- Unknown-field passthrough is impossible — any field number past the
+ array length is data loss. Encoder must error if the input has
+ `_unknown_fields_msgpack` entries.
+- MP_TUPLE ext requires you've registered the format somewhere
+ reachable to the decoder. Without that registration the bytes are
+ inert. Defer the registration story until someone needs it.
+
+## Where things live
+
+```
+runtime/pb/msgpack.lua new — encoder/decoder
+options/tarantool/tarantool.proto extended with msgpack_* options
+cmd/protoc-gen-tarantool/internal/gen/options.go parse new options
+test/msgpack_test.lua new — round-trip + WKT + option matrix
+test/interop/fixtures/*.mp optional — golden bytes per fixture
+```
+
+The plugin does **not** generate msgpack-specific helpers per message —
+`pb.msgpack.encode(desc, t)` walks the descriptor at runtime, same as
+runtime-mode wire codec does today. Full-mode wire codec inlines for
+JIT reasons; msgpack codec doesn't need that complexity yet (the
+msgpackffi C-level encoder does the heavy lifting on the byte side).
+
+If perf becomes a concern, a follow-up could emit
+`M.<Msg>_msgpack_encode` inlined per-message, mirroring the wire codec's
+full mode. Out of scope here.
+
+## Tests
+
+- Round-trip: every existing `test/interop/fixtures/*.txtpb` decoded
+ via proto wire codec, re-encoded via msgpack codec, decoded via
+ msgpack codec, asserted equal to the original Lua table.
+- Parity: msgpack-encoded then transcoded back to wire format equals
+ the original `.bin` golden. Same parity discipline as the
+ full-vs-runtime suite.
+- WKT: explicit fixtures for Timestamp/Duration/Struct/Any covering
+ the ext-type bridges.
+- Layout: a few messages with both `MAP` and `ARRAY` layouts; assert
+ shape on the byte side via `msgpackffi.decode_unchecked` introspection.
+- Unknown-field passthrough: MAP-layout only; ARRAY-layout errors.
+
+## Open questions (defer)
+
+1. **Does proto-side `repeated` of a WKT survive MP_DATETIME packing?**
+ `mp_array{ mp_ext, mp_ext, ... }` should just work. Worth a fixture.
+2. **Custom ext types beyond WKTs.** Users may want their own Lua
+ types (UUID, decimal) to ride the same ext channel. Probably
+ another `(tarantool.msgpack_ext)` field option on `bytes` fields.
+ Defer until someone asks.
+3. **Schema evolution on ARRAY layout.** Renumbering fields breaks
+ ARRAY consumers silently. Should the plugin refuse to compile ARRAY
+ messages with gaps, or just warn? Probably refuse.
+4. **JSON ↔ msgpack interop.** Existing `pb.json` and proposed
+ `pb.msgpack` are both alternate codecs over the same descriptor;
+ should they share a "set this Lua table from any codec" entry
+ point? Probably yes (`pb.from(desc, bytes, format)`) but not part
+ of this slice.
+5. **Streaming.** Wire codec is one-shot; msgpack is too. Streaming
+ adds complexity disproportionate to current needs. Skip.
+
+## What "later" decisions look like
+
+When picking this back up, the load-bearing calls are:
+
+1. **MAP keyed by int vs by name.** Picking int now; revisit if
+ real users find it intolerable to debug. Switching is a one-line
+ change in the encoder.
+2. **ARRAY + MP_TUPLE story.** Whether to ship at all in v1, or punt
+ to a follow-up. ARRAY-as-`mp_array` is enough for `box.space`
+ feeders; MP_TUPLE requires a format registry we don't have.
+3. **Wrapper unwrapping.** Some users may want `Int32Value` to stay
+ as `{value = N}` because their proto-side code treats it as a
+ distinct presence-bearing type. Unwrapping is opinionated.
+
+Everything else in this doc is mechanical and follows from those
+choices.