~bigbes/tarantool

tarantool-protobuf

6471b5d1 — Eugene Blikh 2 months ago
c_runtime: repeated string + repeated message acceptance at 1KB/10KB/100KB (ra6 3f)

The repeated string and repeated message (self-reference) dispatch
landed implicitly with ra6 3e — encode_repeated_field already routes
PB_KIND_MESSAGE through encode_submessage_field and unpacked
string/bytes through encode_one_field, and decode_body's cached
list_stack_idx[] handles every repeated element type. What was
missing was the formal 3f acceptance: explicit-size byte-equality at
1KB / 10KB / 100KB for Person.emails and Person.friends, against
mode=full.

Adds 4 acceptance tests (encode + decode × emails + friends), each
running both codegen modes. Sizing pins each case into a ±30% band
around the named target so a future schema/wire shift fails loudly
instead of silently drifting.

bd-exy
e875519e — Eugene Blikh 2 months ago
c_runtime: repeated + packed scalar encode/decode (ra6 3e)

Add repeated dispatch to the C-runtime encode/decode loop. Encode
side: encode_repeated_field walks Lua arrays via lua_objlen + per-
index rawgeti, dispatches on element kind. Packed numerics build
their payload in a stack-backed sub-buffer then emit `tag(LEN) +
varint(len) + body`; unpacked emit `tag + value` per element via
encode_one_field with force_emit=1 to bypass zero-suppression;
strings/bytes flow through the same path (never packable); repeated
messages reuse encode_submessage_field per element.

Decode side: per-field stack-slot cache (list_stack_idx[]) tied to
list_count[] avoids the per-element lua_getfield(result, name) round
trip the c-accel spike measured at 2x slower at 100KB. On first hit
for a repeated field we lua_createtable + write result[name] AND dup-
push the list onto the stack; subsequent hits lua_rawseti through the
cached absolute stack index. Lists stay valid across recursive sub-
message decodes because each child decode_body cleans up its own
scratch back to the caller's frame.

Packed/unpacked symmetry on read: a wt==LEN payload for any packable
scalar is decoded as a packed blob regardless of the schema's packed
flag, and a per-element-tagged stream is decoded element-by-element
even on a schema that defaults to packed — per proto3 reader rules.

New test/proto/c_repeated.proto fixture carries packed + explicit-
unpacked + repeated string/bytes + repeated message branches. The
encode and decode tests round-trip at 10/100/1000 elements per
branch. The two existing "skip repeated and map" marker tests
collapse to "skip map" — only map fields remain out of scope for
3e (bd-asz / 3h lands them next). 854 → 896 passing tests.

bd-jc9
6e7835a2 — Eugene Blikh 2 months ago
c_runtime: encode/decode singular sub-messages (ra6 3d)

Refactor encode_lua/decode_lua into encode_body/decode_body so the
field-walk loop is callable recursively, then dispatch the MESSAGE
kind into a per-side sub-handler. Repeated and map fields still skip
at the field-walk level — 3e (jc9) and 3h (asz) land them next.

encode_submessage_field force-establishes the parent enc_buf's
heap_idx via a no-op ebuf_grow before recursing. Without that the
final ebuf_reserve on the parent could land its new userdata above
sub-encode's leaked stack slots, making the closing lua_settop drop
the parent's heap.

decode_submessage_field bounds the inner read by temporarily
shrinking c->len to the sub-message end offset; the wire-prim
helpers already bounds-check against c->len, so a malformed inner
payload can't over-read into the outer message's bytes.

New fixture test/proto/c_nested.proto carries a 5-level singular
chain (L1->L2->L3->L4->L5) for the depth test. The two existing
"skip message" tests are renamed to "skip repeated and map" — sub-
messages now encode and decode end-to-end.

bd-hwe
82717b15 — Eugene Blikh 2 months ago
beads: sync mz6 closure to interactions.jsonl
e67a90f2 — Eugene Blikh 2 months ago
c_runtime: decode singular scalars (ra6 3c)

New entry pb.c_runtime.decode(plan, bytes) -> table, symmetric in
scope with 3b: int32/64, uint32/64, sint32/64, bool, fixed32/64,
sfixed32/64, float, double, enum, string, bytes. Repeated, map,
message-typed, and unknown tags skip by wire type — 3d/3e/3i/wyp
will extend later.

Result shape mirrors mode=full pure-Lua decode byte-for-byte: int64
family (int64/uint64/sint64/fixed64/sfixed64) push Tarantool cdata
via luaL_pushint64 / luaL_pushuint64; everything else lands as Lua
number/string/boolean. Pre-sized via lua_createtable(0, n_fields).
Field lookup is a linear scan over plan->fields by field_number; a
tag-keyed dispatch table is a future optimization.

Tests: 26 cases per codegen mode (full + runtime), covering the
bd-mz6 acceptance (round-trip of bd-y1n's Person payload matches
mode=full Person_decode shape-for-shape) plus per-kind coverage,
proto3-optional presence, fixed64 cdata, -0.0 sign preservation
(built via cdata to dodge LuaJIT literal-folding to +0.0),
skip-by-wire-type for out-of-scope shapes, unknown-tag skip, WKT
override rejection, and truncated input. Suite: 832/832 with
PB_ENABLE_C=1, 748/748 + 84 skipped without.

Closes bd-mz6
5e8f5f60 — Eugene Blikh 2 months ago
c_runtime: encode singular scalars (ra6 3b)

New entry pb.c_runtime.encode(plan, msg) -> string. Covers int32/64,
uint32/64, sint32/64, bool, fixed32/64, sfixed32/64, float, double,
enum (number or by_name lookup), string, bytes. Repeated, map, and
message-typed fields are silently skipped — those land in 3d/3e.

Buffer is a 4KB stack scratch promoted to a lua_newuserdata on
overflow, so a mid-encode luaL_error doesn't leak: the userdata is
GC'd at the unwind point.

Proto3 zero-suppression mirrors mode=full byte-for-byte: empty strings,
zero ints/fixed/enum skipped, +0.0 double skipped while -0.0 is emitted
via type-pun u64 equality (matches the `1/v == -math.huge` guard on
the Lua side). Proto3-optional fields bypass suppression. Plans with
desc.encode overrides (WKT) are rejected here — bd-rmf scope.

Tests: 16 cases per codegen mode (full + runtime), covering the bd-y1n
acceptance (Person {name='x', age=42, balance=-7,
user_id=0xDEADBEEFCAFEBABEULL, weight_kg=3.14} byte-equal to mode=full)
plus per-kind sweep and heap-grow path at 8KB. Suite: 806/806 with
PB_ENABLE_C=1, 748/748 + 58 skipped without.

Closes bd-y1n
8721548d — Eugene Blikh 2 months ago
lsp: foundational types + annotate pb runtime entry + lazy views

Add lua-language-server scaffolding so editors and LLM assistants get
parameter / return types for the public pb runtime. Pure metadata --
no runtime behavior change. 748 tests still pass.

- .luarc.json at repo root: declare LuaJIT runtime, project require
  paths (so require('full.hello.hello_pb') resolves), ignored
  directories (.rocks, generated dylibs, bench/starwing third-party).

- runtime/pb/_types.lua (@meta): foundational @class declarations
  for the descriptor contract (pb.Descriptor, pb.Field, pb.Enum-
  Descriptor, pb.OneofDescriptor), the public pb module surface
  (pb.Module), lazy view classes (pb.MessageView / pb.ArrayView /
  pb.MapView), gRPC transport contract (pb.GrpcTransport plus
  Service / Method descriptors), and the json / text / wire
  sub-modules. Mirrors docs/codegen.md's descriptor shape.

- runtime/pb/init.lua: @type pb.Module annotation on the returned
  table so hover on pb.encode / pb.decode_lazy / pb.parse picks up
  the signatures from _types.lua.

- runtime/pb/lazy.lua: @class annotations on the three view locals
  so lua-language-server merges them with the pb.MessageView /
  pb.ArrayView / pb.MapView declarations in _types.lua.

Next phases: annotate codec/grpc/json (phase 2), emit per-message
@class blocks from the plugin codegen (phase 3).
5c541416 — Eugene Blikh 2 months ago
bench(c_accel): drop defensive lua_type checks; refresh numbers

person_codec.c had a per-element lua_type(L,-1)==LUA_TSTRING check
inside the emails loop, plus field-level lua_type checks that
generic_codec.c skipped. At 100KB that's ~2700 extra C calls per
message in the hot path, which made S4 (hand-written) look slower
than S3 (generic) at 1KB+ and led to a wrong "branch prediction on
divergent paths" hypothesis in the original README.

Replaced the per-element check with no check and the field-level
lua_type checks with lua_isnil to match generic_codec.c's semantics.
S3 and S4 are now within +/-5% at every size, which strengthens the
ra6 architecture call (ship the generic one-call codec; codegen-
emitted per-message C buys nothing).

Also added bench/c_accel/compile_flags.txt so clangd resolves
<module.h> and friends -- mirrors runtime/pb/c/compile_flags.txt.
569bc3c9 — Eugene Blikh 2 months ago
fix(wire): cap the Lua-number varint fast path at 2^51 (x86_64 corruption)

encode_varint had a fast path for Lua numbers in [2^28, 2^53) that emitted
each byte via bit.band(n, 0x7f) / math.floor(n / 128). bit.band routes
through LuaJIT's number->int32 conversion, which on x86_64 uses the
magic-number trick (add 2^52 + 2^51, read the low bits). That is exact only
while n + 2^52 + 2^51 < 2^53, i.e. n < 2^51; above it the addition rounds to
an even double and silently drops low bits, corrupting the varint.

arm64 LuaJIT uses an exact FP->int instruction, so the bug was invisible on
Apple-Silicon dev machines and only surfaced on x86_64 (a 64-bit lease ID in
tarantool-etcd round-tripped 3041234677171912 -> 3041234677171940 over gRPC,
breaking lease lookups). Cap the fast path at 2^51; values in [2^51, 2^53)
now fall through to the exact uint64 cdata loop.

Adds test/wire_varint_test.lua pinning the round-trip at the boundaries.
89e781b0 — Eugene Blikh 3 months ago
test: verify build dispatch after hook fix
24245bef — Eugene Blikh 3 months ago
test: trigger hook to debug build dispatch
c420f477 — Eugene Blikh 3 months ago
test: explicit submit option
7a77bcfc — Eugene Blikh 3 months ago
test: debug push hook output
5d29502e — Eugene Blikh 3 months ago
test: verify auto-submit on push
fb7a1e03 — Eugene Blikh 3 months ago
c-accel: add compile_flags.txt for clangd

Without the Tarantool include path, clangd can't find <module.h> and
the entire file cascades into undefined-symbol diagnostics. The actual
make build is unaffected — only the editor experience.

List the four common Tarantool include paths (macOS Homebrew, the
Cellar-style symlink, /usr/local, /usr/include). Missing dirs are
silently ignored by the compiler, so a single file works for both
macOS and Linux.
956bf2a2 — Eugene Blikh 3 months ago
c-accel: fix strdup on glibc with -std=c99

strdup is POSIX, not ISO C99, so glibc's <string.h> only exposes it when
_POSIX_C_SOURCE >= 200809. With -std=c99 (strict mode) gcc otherwise
treats strdup as an implicit-int function, which on 64-bit Linux
truncates the returned pointer to int and yields warnings + likely
crashes. macOS happens to declare strdup unconditionally so the issue
only surfaces on the srht.bigb.es Ubuntu builder.

Define _POSIX_C_SOURCE at the top of c_runtime.c before any include.
52e47523 — Eugene Blikh 3 months ago
beads: sync mq7 closure to issues.jsonl

Passive export catch-up after closing tarantool-protobuf-mq7
(descriptor -> C plan compiler) — committed in 5c70555 but the .jsonl
export hadn't been refreshed.
4fd731af — Eugene Blikh 3 months ago
ci: install Go 1.26 toolchain + cmake on builds.srht

Ubuntu noble's `golang` apt package is too old to satisfy go.mod's
toolchain directive — `go build` aborted with "toolchain not available".
Drop the apt package, install the official Go 1.26.3 tarball into
~/.local/go, and update go.mod / bench/go/go.mod accordingly. Also add
cmake + build-essential so `tt rocks install luatest` can build its
`checks` dependency.
5c705557 — Eugene Blikh 3 months ago
c-accel: descriptor -> C plan compiler (bd-mq7)

First C source for the pb.c_runtime module. Compiles a finalized Lua
descriptor into an opaque pb_plan userdata, the foundation that
bd-ra6's encode/decode entry points will walk.

What the plan carries (per docs/specs/c_accel_strategy.md):

  * Per-field records: field_number, wire_type, kind, repeated/packed/
    optional flags, pre-encoded tag bytes (varint, up to 5 bytes),
    sub_plan_idx (1-based into the sub-plans table), enum_ref
    (luaL_ref for enum descriptor), oneof_idx back-pointer.
  * Map fields capture map_key_kind, map_value_kind, and the value's
    sub_plan_idx when the value is a message.
  * Oneofs as a parallel array of {name, member_indices[]}, with
    fields' oneof_idx pointing back to their group.
  * WKT override detection: when desc.encode/desc.decode are set, the
    plan flips has_override=1 and skips field-walk entirely.
  * Extension range hooks captured (proto2 scaffolding for bd-3i).
  * Field-name cache as a Lua table referenced via luaL_ref, so
    encode/decode can do lua_rawgeti instead of re-interning C strings.

Cycle handling: compile_plan stashes the new plan userdata on
desc.c_plan BEFORE recursing into sub-message fields. Person.friends
→ Person resolves to the same userdata; the test asserts identity.

Build entry: 'just build-c' compiles runtime/pb/c/c_runtime.c into
runtime/pb/c_runtime.dylib (or .so on Linux) via the local Makefile.
Module-h location auto-detected the same way bench/c_accel/Makefile
does it. Tarantool's LuaJIT-on-5.1 means we use lua_objlen (not
lua_rawlen) and provide a local abs_idx helper since lua_absindex
isn't available in the 5.1 compat layer.

Smoke test at test/c_runtime_plan_test.lua exercises both codegen
modes (full + runtime). 26 assertions cover:
  - module surface + ABI version + KIND/WIRE constants
  - Address: 4 fields, names, kinds, wire types, tag bytes, optional
  - Person: 14 fields, scalars/enum/message/map/repeated/packed
  - Sub-plan resolution + Person.friends self-reference cycle break
  - Idempotent compile (second call returns cached plan)
  - Result.outcome oneof: 3 members, oneof_idx back-pointers
  - WKT Timestamp: has_override=true, field-walk skipped

Full suite: 771/771 with PB_ENABLE_C=1 (745 existing + 26 new),
745+26 skipped without (silent fallback verified).

Justfile fix: 'just build-c' / 'clean-c' used $(MAKE) which Just
doesn't expand — switched to plain 'make'.

Unblocks bd-y1n (encode scalars), bd-mz6 (decode scalars),
bd-awv (64-bit cdata), bd-rmf (WKT passthrough). bd-mq7 closed.
3042384b — Eugene Blikh 3 months ago
c-accel: arch prereqs — compat contract, C-side strategy, build scaffolding

Three companion specs under docs/specs/ formalize the boundaries
established in docs/c-accel.md, unblocking bd-mq7 (descriptor → C
plan compiler):

* c_accel_compat.md (bd-47e) — pinpoints what must stay byte-equal
  between PB_ENABLE_C unset and =1: public surface, generated
  module wrappers, 64-bit cdata, WKT shapes, unknown fields,
  extensions, errors. Calls out the lazy-view exclusion.

* c_accel_strategy.md (bd-z7x) — pb_plan struct layout, field-name
  luaL_ref caching, 4 KB stack-backed pb_buf, cached per-field
  stack indices (the 2× win from spike Phase B), sub-buffer over
  backpatching, map/oneof/unknown handling.

* c_accel_build_packaging.md (bd-wky) — where the C module lives
  (runtime/pb/c/), how it builds, what the rockspec gains, the CI
  matrix shape.

Scaffolding that lands now:

* runtime/pb/init.lua — PB_ENABLE_C=1 opt-in pcall hook; the
  loaded module (or nil) is exposed as pb.c_runtime for
  introspection. Silent fallback when the module is absent.

* Justfile — `build-c` / `clean-c` recipes (stub erroring cleanly
  until bd-ra6 lands runtime/pb/c/), new lua_cpath constant,
  LUA_CPATH wired through `test` and `test-one`.

* .builds/{pure-lua,c-enabled}.yml — sourcehut CI manifests, one
  per activation mode (sourcehut has no matrix; parallel jobs go
  in separate files). ubuntu/noble images.

* .sourcehut/conformance.yml — outside .builds/ so it doesn't
  auto-submit; trigger manually with `hut builds submit` before
  releases.

* .gitignore — runtime/pb/c_runtime.{so,dylib} and runtime/pb/c/*.o.

745/745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1
(silent fallback verified).

Closes bd-47e, bd-z7x, bd-wky. Unblocks bd-mq7.
Next