~bigbes/tarantool

tarantool-protobuf

ref: ab214a39023a13ef4c1e534589bdc0d20ff59ce4 tarantool-protobuf/README.md -rw-r--r-- 11.2 KiB
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
4d7fda1f — Eugene Blikh 3 months ago
docs: drop text-format parser handoff brief

The pb.text.decode slice landed (7ec8f2b); the handoff brief is no
longer load-bearing. Removes the file and unlinks references from
README.md and PLAN.md.
314f642e — Eugene Blikh 3 months ago
docs: refresh README + PLAN; add api-modes and codegen notes

README.md
  * Status table reflects current state: text-format encode + decode,
    -0.0 preservation, strict FieldMask, all-green proto3 conformance.
  * Conformance baseline jumped to 1493 / 416 / 0 failures (was
    1389 / 0 / 79 in the old table); explains the 1313+18 skipped
    tests are all TestAllTypesProto2, deferred separately.
  * Layout walks the full runtime/pb/ tree (lazy, text, json, wkt,
    grpc, parser, dynamic, fileset, descriptor_pb).
  * Generated API section calls out the three-mode design and links
    to docs/api-modes.md and docs/codegen.md.

PLAN.md
  * Section 2 ("Current state") rewritten — no longer claims M0;
    lists what's actually in the codebase.
  * M1, M2, M3 marked done with [x] checkboxes (had stale [ ]
    markers across items that have been shipping for months).
  * M5 conformance numbers updated to current baseline; calls out
    the three commits that closed the proto3 suite (text decode,
    -0 codec, JSON strict pass).
  * M7 text-format parser entry updated: 416 ✓ / 0 failures (was
    406 / 10 expected before the codec -0 fix).
  * Added an M7 entry for the JSON strict-validation pass (six
    classes of relaxation now enforced).

docs/api-modes.md (new)
  * When to use full vs runtime (descriptor / reflect) vs lazy.
  * Concrete code shapes for each, plus what the generated
    Person_encode actually looks like in full mode.
  * Descriptor-shape contract that ties all three together.
  * Lazy: SoA index rationale, sparse-read vs dense-read trade-offs,
    cross-over points from bench numbers.

docs/codegen.md (new)
  * Pipeline diagram, CLI options, what gets emitted per .proto.
  * Walk through the inline (full) mode emission with annotated
    generated code.
  * Runtime mode: pb.finalize_message's per-field writer/reader
    closures and why they exist.
  * Hot-path rules the generated code observes (no pairs(), 64-bit
    as cdata, SoA over AoS for large index structures, keep hot
    helpers small).
  * Plugin source map; where to extend for a new wire type.
  * Proto2 deferral section: what it would take and why we punt.
bd0709dd — Eugene Blikh 3 months ago
license: BSD 2-Clause

Matches Tarantool's own license. Closes the "TBD" placeholder in the
README.
37c56322 — Eugene Blikh 3 months ago
json: enable conformance JSON output + close 459 tests

Three coupled changes that turn the JSON output path on for the
conformance harness:

1. string_to_int64 accepts cdata: Tarantool's json.decode parses raw
   JSON integer literals outside double range as int64_t/uint64_t cdata,
   not Lua numbers. The decoder errored "expected JSON string or number
   for int64" on any unquoted 64-bit value (Int64FieldMaxValueNotQuoted
   et al). cdata is now cast through directly, preserving precision.

2. encode_message marks output as a map: an empty proto3 message
   serialized as `[]` because Tarantool's json defaults empty tables to
   array shape. jsoncpp's strict comparator threw Json::LogicError and
   aborted the whole suite. Setting __serialize='map' on the output
   gives `{}` and unblocks all JsonOutput tests.

3. PB_CONFORMANCE_SKIP_JSON gate is opt-in by default. core.lua now
   matches exactly "1" (so docker -e VAR= disables it), and the
   Dockerfile no longer hard-codes "=1" — JSON output runs end-to-end
   for everyone unless they re-enable the gate.

Conformance moves from 930 / 1869 / 11 to 1389 / 1313 / 79
(successes / skipped / expected fails). The 75 new expected fails
are canonical-form edge cases (Duration formatting sign handling,
Timestamp out-of-range rejection, double precision digits, NaN
canonicalization, JSON-input strict rejection) — left for a follow-up.

Drops 7 entries from test/conformance/known_failures.txt that this
change closes; adds 75 newly-visible ones.
d855bcfd — Eugene Blikh 3 months ago
conformance: local Docker pipeline + cdata int64 map dedup

Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.

Six bugs surfaced and got fixed on the way to green:

  1. conformance_test_runner uses execv (not execvp): bare `tarantool`
     hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
  2. The harness strips LUA_PATH from the child: the runner now
     self-bootstraps package.path from debug.getinfo(1, 'S').source.
  3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
     full BUFSIZ before returning, deadlocking against the parent.
     setvbuf('no') on stdin/stdout.
  4. v34.1 fetches libjsoncpp via CMake FetchContent under
     _deps/jsoncpp-build/...; the runtime image now COPYs the matching
     .so* and runs ldconfig.
  5. The harness's strict jsoncpp comparator crashes on our currently-
     imperfect JSON output (enum numerics, map<K,V> shape, oneof
     object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
     set in the container ENV; host-side `make test` still exercises
     the full JSON path.
  6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
     key map entries (per proto3's "last value wins" semantics) split
     across hash buckets even though __eq matches. Codec walks the
     map once on insert to find a canonical key, gated by a
     precomputed `f.key_dedup` flag so the dedup only fires for
     int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
     same `for _k in pairs(map) do` walk only when the static key
     kind is 64-bit, so string/int32-keyed map decode stays
     JIT-traceable.

Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:

  - Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
  - Text-format suite:     0 ✓ /  430 skipped /   4 expected fails

403/403 luatest green, 19/19 jit-trace gate green.
784dea4d — Eugene Blikh 3 months ago
Initial commit: protoc-gen-tarantool plugin + pb runtime

A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.

Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.