~bigbes/tarantool

tarantool-protobuf

ref: e258695ea1fe3b322d0ab4277bbdeed6d4eabe5e tarantool-protobuf/docs/api-modes.md -rw-r--r-- 14.0 KiB
e3f06741 — Eugene Blikh 2 months ago
runtime: pb.decode_unsafe + M.<Name>_decode_unsafe in runtime mode (58u)

Completes the unsafe-decode story 6bb started in full mode. Runtime mode
now exposes the same API via parallel `f._reader_unsafe` closures
compiled in pb.finalize_message against a swapped scalar table where
`string` maps to the bytes handler (no utf8_len). build_reader /
build_repeated_reader / decode_one are now parameterized on
(scalar_tbl, decode_msg_fn, decode_group_fn) so the same builders emit
both reader shapes. decode_message_unsafe, decode_group_unsafe, and
decode_extension_unsafe are literal clones of their safe twins with
three substitutions (documented in codec.lua): the _reader field, the
scalar table in slow paths, and the sub-message / group / extension
dispatchers. Tests in test/decode_unsafe_test.lua are now parameterized
over both modes (14 cases, including a map<string, int32> invalid-key
case that exercises the decode_one map-fallback path).

Runtime-mode microbench shows ~8% throughput vs validating decode on
the string-heavy 1KB Person; smaller than full mode's ~20% because the
descriptor dispatch + closure indirection swamp utf8_len, but still a
net win and the perf-cost-of-validating story is now consistent across
modes. Conformance 3240/3240 + JIT trace 37/37 still pass.

Closes 58u, also closes b12 (already fixed in 2656c97; never closed).
kyt still tracks unifying _decode_unsafe with C accel.
53840866 — Eugene Blikh 2 months ago
codegen: emit <Msg>_decode_unsafe for trusted-source decoding (6bb)

Full-mode codegen now emits a sister <Msg>_decode_unsafe(buf) alongside
<Msg>_decode that drops the per-string utf8_len check (singular,
repeated, map keys/values, extensions, and the >=128-byte fallback all
route through wire.decode_bytes). Sub-messages recurse into their own
_decode_unsafe so nested strings also bypass; WKTs continue to call the
normal pb.wkt.<Name>_decode (no _unsafe twin, no string-validation hot
path). C runtime dispatch is skipped because it validates today (kyt).

Use this when re-decoding bytes from a trusted producer — your own
encoder over typed RPC, JSON/text round-trips, in-process pipelines —
where the spec-required utf8.len check on every string is duplicate
work. Microbench on a string-heavy 1KB Person (26 emails) shows
~20% throughput vs _decode; conformance suite still passes (3240/3240)
because _decode itself is unchanged.

Runtime mode does not yet expose _decode_unsafe (compiled f._reader
closures capture handler.decode by value, so a runtime swap wouldn't
reach them); tracked in 58u.
a0005a05 — Eugene Blikh 3 months ago
docs: full reference + how-to set, migrate Makefile to Justfile

Documentation overhaul that adds the missing user-facing surface:
four reference pages (runtime-api, generated-api, cli, grpc-contract),
twelve how-tos walking from first-message through custom transports,
a troubleshooting page, and a docs/index map. Every how-to references
a runnable artifact under examples/, all of them verified end-to-end.

Build system migration: the Makefile is gone; the Justfile is now
the canonical entry point and absorbs every target. examples/Justfile
ships one recipe per runnable example, forwarded via top-level
'just examples <name>'. The 'examples are part of the documented
surface' convention is pinned in CLAUDE.md, alongside a dedicated
section on updating the conformance harness (PROTOBUF_TAG bumps,
libjsoncpp path drift, new test-category wiring).

Stale-number sweep across README/PLAN/CLAUDE: fixture count 18→10,
test count 613/130→639, wire.lua LOC dropped, M7 marked done.
Descriptor-shape block deduplicated against codegen.md as the
canonical source. gRPC transports spec status reframed from
'draft / decision deferred' to 'shipped contract; external
transports deferred'.

.gitignore picks up *.snap / *.xlog / *.vylog / *.run / *.pid /
512.lock so example state can't leak into the working tree.
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
314f642e — Eugene Blikh 3 months ago
docs: refresh README + PLAN; add api-modes and codegen notes

README.md
  * Status table reflects current state: text-format encode + decode,
    -0.0 preservation, strict FieldMask, all-green proto3 conformance.
  * Conformance baseline jumped to 1493 / 416 / 0 failures (was
    1389 / 0 / 79 in the old table); explains the 1313+18 skipped
    tests are all TestAllTypesProto2, deferred separately.
  * Layout walks the full runtime/pb/ tree (lazy, text, json, wkt,
    grpc, parser, dynamic, fileset, descriptor_pb).
  * Generated API section calls out the three-mode design and links
    to docs/api-modes.md and docs/codegen.md.

PLAN.md
  * Section 2 ("Current state") rewritten — no longer claims M0;
    lists what's actually in the codebase.
  * M1, M2, M3 marked done with [x] checkboxes (had stale [ ]
    markers across items that have been shipping for months).
  * M5 conformance numbers updated to current baseline; calls out
    the three commits that closed the proto3 suite (text decode,
    -0 codec, JSON strict pass).
  * M7 text-format parser entry updated: 416 ✓ / 0 failures (was
    406 / 10 expected before the codec -0 fix).
  * Added an M7 entry for the JSON strict-validation pass (six
    classes of relaxation now enforced).

docs/api-modes.md (new)
  * When to use full vs runtime (descriptor / reflect) vs lazy.
  * Concrete code shapes for each, plus what the generated
    Person_encode actually looks like in full mode.
  * Descriptor-shape contract that ties all three together.
  * Lazy: SoA index rationale, sparse-read vs dense-read trade-offs,
    cross-over points from bench numbers.

docs/codegen.md (new)
  * Pipeline diagram, CLI options, what gets emitted per .proto.
  * Walk through the inline (full) mode emission with annotated
    generated code.
  * Runtime mode: pb.finalize_message's per-field writer/reader
    closures and why they exist.
  * Hot-path rules the generated code observes (no pairs(), 64-bit
    as cdata, SoA over AoS for large index structures, keep hot
    helpers small).
  * Plugin source map; where to extend for a new wire type.
  * Proto2 deferral section: what it would take and why we punt.