~bigbes/tarantool

tarantool-protobuf

a5ba47c793d7f1177e823122c06ce8c290fa4603 — Eugene Blikh 2 months ago 8b5bfc0
docs: document int64_as_number flag (5y9)

Adds end-to-end coverage of the int64_as_number plugin option that
landed in 8b5bfc0:

- docs/reference/cli.md: row in the --tarantool_opt table + a dedicated
  subsection with the workload tradeoff table, the runtime-side wire
  helpers it exposes (wire.decode_<type>_n), and the PB_ENABLE_C=1
  no-op note.
- docs/codegen.md: same row in its own CLI-options table + a shorter
  subsection (the deep version lives in reference/cli.md).
- README.md: status row for the feature; paragraph at the end of the
  "Migrating from a Lua proto library that auto-down-casts int64"
  section pitching the flag for callers who want number-on-fits
  semantics with cdata fallback, plus a pointer to the cli.md table.

No code change.
3 files changed, 120 insertions(+), 0 deletions(-)

M README.md
M docs/codegen.md
M docs/reference/cli.md
M README.md => README.md +21 -0
@@ 33,6 33,7 @@ copy of `test_messages_proto2.proto`.
| Cross-file imports               | ✅           |
| Enums (open semantics)           | ✅           |
| 64-bit integers as LuaJIT cdata  | ✅           |
| Opt-in 64-bit-as-Lua-number decode (`int64_as_number=true`) | ✅ |
| Two codegen modes (full + runtime) | ✅         |
| Zero-copy lazy decode views      | ✅           |
| `map<K,V>` (scalar/message values) | ✅         |


@@ 278,6 279,26 @@ boundary is *Lua primitives that expect a number*. Past 2^53 (≈ 9e15),
`tonumber()` silently truncates; if your IDs can be that large, keep
them as cdata or stringify with `tostring(id):gsub('U?LL$', '')`.

If your schema's 64-bit values are dominated by the small-and-positive
case (counter-derived IDs, small sequence numbers, byte counts under
16 KB, integer enums, second-resolution timestamps in a restricted
range), the codegen can decode those as Lua numbers up front:

```bash
protoc --tarantool_opt=mode=full,int64_as_number=true ...
```

Decoded fields come back as Lua numbers when the value fits
`[-2^53, 2^53]` and cdata otherwise — same precision guarantee, none
of the `tonumber()` calls. The type at decode time becomes value-
dependent (arithmetic works transparently across both, but
`type(v) == 'cdata'` no longer holds for small values). Workload-
specific tradeoff: cdata-allocation saving on the 1-byte-varint fast
path measured at ~17 % on a 5-field 64-bit schema, but multi-byte
varints regress ~10 % because both paths still allocate the
intermediate cdata before the size check. Full table and the per-
size measurements live in [docs/reference/cli.md](docs/reference/cli.md#int64_as_number).

**Three API modes live side-by-side.** Same descriptor, three call shapes —
the inline (full) generated API is the default, the descriptor-driven runtime
API is for dynamic schemas, and the lazy API is a zero-copy view for sparse

M docs/codegen.md => docs/codegen.md +51 -0
@@ 41,6 41,7 @@ Passed via `protoc --tarantool_opt=<key>=<value>,...`:
|---|---|---|
| `mode` | `full` (default) or `runtime` | Inline `_encode`/`_decode` vs descriptor-delegating wrappers. See [api-modes.md](api-modes.md). |
| `prefix` | any Lua-require path | Prepended to every generated module's require path **and** its on-disk subpath. The Justfile uses this to generate both modes side-by-side into `examples/expected/{full,runtime}/`. |
| `int64_as_number` | `true` or `false` (default) | `mode=full` only. Opt-in: decode `int64`/`uint64`/`sint64`/`fixed64`/`sfixed64` fields as Lua numbers when the value fits `[-2^53, 2^53]`, cdata otherwise. Workload-dependent tradeoff — see [int64\_as\_number](#int64_as_number) below. |

The `(tarantool.lua_package)` file option (defined in
`options/tarantool/tarantool.proto`) overrides the per-file Lua module


@@ 55,6 56,56 @@ Without that option the path mirrors `package` (e.g.
`package my.app; foo.proto` → `my/app/foo_pb.lua`,
`require('my.app.foo_pb')`).

### int64_as_number

By default every 64-bit scalar field decodes to LuaJIT `int64_t` /
`uint64_t` cdata — the same convention as Tarantool's `msgpackffi`,
`net.box`, `box.tuple`, and the built-in `protobuf` module. That keeps
precision past 2^53 and matches what application code already does
with 64-bit IDs.

`int64_as_number=true` swaps the decoder for one that returns a Lua
number when the decoded value fits `[-2^53, 2^53]` (inclusive — both
endpoints are powers of two and exact as doubles) and cdata otherwise.
The return type becomes value-dependent: arithmetic (`+`/`-`/`*`/`==`)
works transparently across the two, but `type(v)` is no longer stable,
so call sites that branch on `type(v) == 'cdata'` need updating.

```
protoc --plugin=./protoc-gen-tarantool \
       --tarantool_out=out \
       --tarantool_opt=mode=full,int64_as_number=true \
       myschema.proto
```

When to use it. Workload-specific tradeoff measured on a 5-field 64-bit
schema (`c_int64.Wide`, full mode, no PB_ENABLE_C):

| Per-field magnitude | Default (cdata) | `int64_as_number` | Delta |
|---|---|---|---|
| 1-byte varints (<128) | 2050 ns/op | 1700 ns/op | **−17%** |
| 3-byte varints | 3220 ns/op | 3600 ns/op | +11% |
| past 2^53 | 7575 ns/op | 7750 ns/op | +2% (noise) |

Enable when 64-bit fields typically carry small values that hit the
1-byte varint path — request IDs from a counter, small sequence
numbers, byte counts under 16 KB, integer enums, second-resolution
timestamps in restricted ranges. The win comes from skipping the
cdata allocation entirely on the fast path. For workloads dominated
by full-range int64 / fixed64 (epoch-nanosecond timestamps, large
hash IDs, file-size byte counts), leave it off — the comparison and
`tonumber()` call add net overhead on top of the cdata that's allocated
anyway.

Restrictions:

- `mode=full` only. `mode=runtime` errors at plugin startup; supporting
  it would require a descriptor flag wired through `pb.codec`.
- Under `PB_ENABLE_C=1` the flag is a no-op: the C runtime makes its
  own number-vs-cdata decision via `luaL_pushint64` (Tarantool's small-
  fits-in-double convention) before the codegen-emitted decoder runs.
  See [c-accel.md](c-accel.md).

## What gets emitted per .proto

```lua

M docs/reference/cli.md => docs/reference/cli.md +48 -0
@@ 49,6 49,7 @@ Comma-separated `key=value` pairs:
|---|---|---|---|
| `mode` | `full` / `runtime` | `full` | `full` inlines `_encode` / `_decode` bodies; `runtime` emits one-line delegations to `pb.encode` / `pb.decode`. See [api-modes.md](../api-modes.md). |
| `prefix` | any Lua require path | empty | Prepended to every generated module's require path and on-disk subpath. |
| `int64_as_number` | `true` / `false` | `false` | `mode=full` only. Decode 64-bit scalar fields as Lua numbers when the value fits `[-2^53, 2^53]`, cdata otherwise. Workload-dependent — see [`int64_as_number`](#int64_as_number) below. |

### `prefix`



@@ 69,6 70,53 @@ protoc --tarantool_opt=mode=full,prefix=full         ...   # examples/expected/f
protoc --tarantool_opt=mode=runtime,prefix=runtime   ...   # examples/expected/runtime/...
```

### `int64_as_number`

Decode `int64` / `uint64` / `sint64` / `fixed64` / `sfixed64` fields as
a Lua `number` when the decoded value fits `[-2^53, 2^53]` (inclusive
on both endpoints — both are powers of two and exact as doubles), as
LuaJIT `int64_t` / `uint64_t` cdata otherwise. `mode=full` only;
`mode=runtime` errors at plugin startup. Default off.

```bash
protoc --tarantool_out=out \
       --tarantool_opt=mode=full,int64_as_number=true \
       schema.proto
```

The return type becomes value-dependent under this flag. Arithmetic
(`+`/`-`/`*`/`/`/`==`) works transparently across number and cdata,
but call sites that branch on `type(v) == 'cdata'` or rely on
cdata-only operators need updating.

Workload-dependent tradeoff measured on a 5-field 64-bit schema
(`c_int64.Wide`, `mode=full`, no `PB_ENABLE_C`):

| Field-value magnitude | Default (always cdata) | `int64_as_number` | Delta |
|---|---|---|---|
| 1-byte varints (`< 128`) | 2050 ns/op | 1700 ns/op | **−17 %** |
| 3-byte varints | 3220 ns/op | 3600 ns/op | +11 % |
| past 2^53 | 7575 ns/op | 7750 ns/op | +2 % (noise) |

Enable when the schema's 64-bit fields typically carry small values
(counter-derived IDs, small sequence numbers, byte counts < 16 KB,
integer enums, second-resolution timestamps in restricted ranges).
The win is from skipping the cdata header allocation entirely on the
fast path. For workloads dominated by full-range int64 / fixed64
(epoch-nanosecond timestamps, large hash IDs, file-size byte counts),
leave it off — both decoders allocate a cdata for the intermediate
varint, and the comparison + `tonumber()` adds net overhead.

Under `PB_ENABLE_C=1` the flag is a no-op: the C runtime makes its
own number-vs-cdata decision via `luaL_pushint64` (Tarantool's small-
fits-in-double convention) before the codegen-emitted decoder runs.
See [c-accel.md](../c-accel.md).

The `runtime/` shipped Lua module exposes the matching wire helpers
(`wire.decode_int64_n`, `decode_uint64_n`, `decode_sint64_n`,
`decode_fixed64_n`, `decode_sfixed64_n`) for hand-rolled callers that
want the same semantics without going through codegen.

### `(tarantool.lua_package)` — per-file override

Defined in `options/tarantool/tarantool.proto`. Overrides the require