~bigbes/tarantool

tarantool-protobuf

ref: 6e7835a29a3678aa86a49dfff76c55b18d091755 tarantool-protobuf/docs d---------
3042384b — Eugene Blikh 2 months ago
c-accel: arch prereqs — compat contract, C-side strategy, build scaffolding

Three companion specs under docs/specs/ formalize the boundaries
established in docs/c-accel.md, unblocking bd-mq7 (descriptor → C
plan compiler):

* c_accel_compat.md (bd-47e) — pinpoints what must stay byte-equal
  between PB_ENABLE_C unset and =1: public surface, generated
  module wrappers, 64-bit cdata, WKT shapes, unknown fields,
  extensions, errors. Calls out the lazy-view exclusion.

* c_accel_strategy.md (bd-z7x) — pb_plan struct layout, field-name
  luaL_ref caching, 4 KB stack-backed pb_buf, cached per-field
  stack indices (the 2× win from spike Phase B), sub-buffer over
  backpatching, map/oneof/unknown handling.

* c_accel_build_packaging.md (bd-wky) — where the C module lives
  (runtime/pb/c/), how it builds, what the rockspec gains, the CI
  matrix shape.

Scaffolding that lands now:

* runtime/pb/init.lua — PB_ENABLE_C=1 opt-in pcall hook; the
  loaded module (or nil) is exposed as pb.c_runtime for
  introspection. Silent fallback when the module is absent.

* Justfile — `build-c` / `clean-c` recipes (stub erroring cleanly
  until bd-ra6 lands runtime/pb/c/), new lua_cpath constant,
  LUA_CPATH wired through `test` and `test-one`.

* .builds/{pure-lua,c-enabled}.yml — sourcehut CI manifests, one
  per activation mode (sourcehut has no matrix; parallel jobs go
  in separate files). ubuntu/noble images.

* .sourcehut/conformance.yml — outside .builds/ so it doesn't
  auto-submit; trigger manually with `hut builds submit` before
  releases.

* .gitignore — runtime/pb/c_runtime.{so,dylib} and runtime/pb/c/*.o.

745/745 tests pass with PB_ENABLE_C unset and PB_ENABLE_C=1
(silent fallback verified).

Closes bd-47e, bd-z7x, bd-wky. Unblocks bd-mq7.
6e89cf99 — Eugene Blikh 3 months ago
docs: C acceleration architecture note (pf6 deliverable)

Records the architecture chosen for the C-accelerated encode/decode
path, based on the bench/c_accel/ spike results (commit d4dbd2f).
Closes the umbrella decision task pf6; the breakdown of follow-on
implementation work lives in Beads under bd-ra6 and its twelve
sub-issues (3a–3l).

Key decisions captured:

- Ship S3 (generic C runtime, descriptor-walking, one C call per
  message) as bd-ra6. Within ±15% of hand-written codegen in the
  spike, and beats it at scale on encode.
- Defer S4 (codegen-emitted C, bd-c0i) with explicit revival
  criteria — ≤15% headroom doesn't earn codegen + maintenance +
  distribution complexity.
- Drop S2 (per-primitive FFI) — structurally worse than pure Lua;
  the FFI boundary cost (~60-75 ns/call) is the same order of
  magnitude as the pure-Lua varint helpers it would replace.

User-facing contract:

- PB_ENABLE_C=1 env var is the only activation switch, default
  off. require('pb') returns the same Lua surface in both modes;
  zero behavior change for existing installs.
- No Lua-level toggle. Single knob, evaluated once at module load.

Parity verification:

- No separate Lua-vs-C diff harness. Existing test suites
  (luatest, conformance, interop fixtures) run twice in CI — once
  with PB_ENABLE_C unset and once with PB_ENABLE_C=1. Parity is
  guaranteed by transitivity through the reference outputs.

beads-tarantool-protobuf-pf6
b1a11473 — Eugene Blikh 3 months ago
docs: retire PLAN.md in favor of Beads issue tracking

Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.

References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.

The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.

Closes the goal of bd-8k2.
79efcdc9 — Eugene Blikh 3 months ago
docs: 100% proto2 + proto3 conformance

README status table: replace the four \"deferred\" rows with
explicit checkmarks for groups, extensions, closed enums; bump
both conformance lines to 2806 and 434 successes. Note MessageSet
as the only known gap (protoreflect rejects upstream, our vendored
proto2 schema has those four nested messages stripped).

docs/codegen.md proto2-support section gets three subsections —
Groups, Extensions, Closed enums — each describing the runtime
surface a user touches (decode_group, _extensions, the closed flag).
The descriptor-table example includes a group field and the new
extensions_by_id / extensions_by_full_name indices.

PLAN.md M9 closes with the conformance numbers; M5 header updated
to reflect that proto2 is no longer deferred.
cef42bf3 — Eugene Blikh 3 months ago
docs: proto2 is supported for the core surface

README's status table swaps the single \"proto2 deferred\" row for
four entries that name what works (required / optional / custom
defaults / packed semantics) and what doesn't (extend, extensions,
group, MessageSet) so users can decide before they generate.

docs/codegen.md replaces the proto2 deferral section with a
\"Proto2 support\" page: the three new descriptor attributes
(required, default_value, optional everywhere), the existing
IsPacked() trick that already produces the right unpacked default,
and a concrete list of the four features that still refuse to
generate (with the MessageSet upstream-rejection caveat).

PLAN.md gains M9 — Proto2 baseline. Wire format, codecs, parser,
dynamic, tests are checked off; extend / group / MessageSet stay
open because the conformance schema needs all three and they're
each their own slice.
a0005a05 — Eugene Blikh 3 months ago
docs: full reference + how-to set, migrate Makefile to Justfile

Documentation overhaul that adds the missing user-facing surface:
four reference pages (runtime-api, generated-api, cli, grpc-contract),
twelve how-tos walking from first-message through custom transports,
a troubleshooting page, and a docs/index map. Every how-to references
a runnable artifact under examples/, all of them verified end-to-end.

Build system migration: the Makefile is gone; the Justfile is now
the canonical entry point and absorbs every target. examples/Justfile
ships one recipe per runnable example, forwarded via top-level
'just examples <name>'. The 'examples are part of the documented
surface' convention is pinned in CLAUDE.md, alongside a dedicated
section on updating the conformance harness (PROTOBUF_TAG bumps,
libjsoncpp path drift, new test-category wiring).

Stale-number sweep across README/PLAN/CLAUDE: fixture count 18→10,
test count 613/130→639, wire.lua LOC dropped, M7 marked done.
Descriptor-shape block deduplicated against codegen.md as the
canonical source. gRPC transports spec status reframed from
'draft / decision deferred' to 'shipped contract; external
transports deferred'.

.gitignore picks up *.snap / *.xlog / *.vylog / *.run / *.pid /
512.lock so example state can't leak into the working tree.
ab214a39 — Eugene Blikh 3 months ago
codegen: emit strict <Type>_fields / _oneofs constants for lazy view

Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.

Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.

Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
4d7fda1f — Eugene Blikh 3 months ago
docs: drop text-format parser handoff brief

The pb.text.decode slice landed (7ec8f2b); the handoff brief is no
longer load-bearing. Removes the file and unlinks references from
README.md and PLAN.md.
6d06f23b — Eugene Blikh 3 months ago
docs: spec out msgpack encoding and gRPC transport story

Two design slices captured under docs/specs/ for later pickup:

- msgpack_encoding.md: proto schemas as IDL, msgpack as wire format.
  Default mp_map keyed by field number; ARRAY/MP_TUPLE opt-ins for
  box.space feeders; WKT bridging to MP_DATETIME / MP_INTERVAL.
- grpc_transports.md: Connect-JSON over HTTP/1.1 as default external
  transport, net.box tunnel for in-cluster, explicit no on HTTP/2
  termination (recommend Envoy). Conformance anchored on
  connectrpc/conformance — same framed-runner shape as the existing
  protobuf conformance harness.

PLAN.md cross-links both specs from M4, Open Questions, and Non-goals.
314f642e — Eugene Blikh 3 months ago
docs: refresh README + PLAN; add api-modes and codegen notes

README.md
  * Status table reflects current state: text-format encode + decode,
    -0.0 preservation, strict FieldMask, all-green proto3 conformance.
  * Conformance baseline jumped to 1493 / 416 / 0 failures (was
    1389 / 0 / 79 in the old table); explains the 1313+18 skipped
    tests are all TestAllTypesProto2, deferred separately.
  * Layout walks the full runtime/pb/ tree (lazy, text, json, wkt,
    grpc, parser, dynamic, fileset, descriptor_pb).
  * Generated API section calls out the three-mode design and links
    to docs/api-modes.md and docs/codegen.md.

PLAN.md
  * Section 2 ("Current state") rewritten — no longer claims M0;
    lists what's actually in the codebase.
  * M1, M2, M3 marked done with [x] checkboxes (had stale [ ]
    markers across items that have been shipping for months).
  * M5 conformance numbers updated to current baseline; calls out
    the three commits that closed the proto3 suite (text decode,
    -0 codec, JSON strict pass).
  * M7 text-format parser entry updated: 416 ✓ / 0 failures (was
    406 / 10 expected before the codec -0 fix).
  * Added an M7 entry for the JSON strict-validation pass (six
    classes of relaxation now enforced).

docs/api-modes.md (new)
  * When to use full vs runtime (descriptor / reflect) vs lazy.
  * Concrete code shapes for each, plus what the generated
    Person_encode actually looks like in full mode.
  * Descriptor-shape contract that ties all three together.
  * Lazy: SoA index rationale, sparse-read vs dense-read trade-offs,
    cross-over points from bench numbers.

docs/codegen.md (new)
  * Pipeline diagram, CLI options, what gets emitted per .proto.
  * Walk through the inline (full) mode emission with annotated
    generated code.
  * Runtime mode: pb.finalize_message's per-field writer/reader
    closures and why they exist.
  * Hot-path rules the generated code observes (no pairs(), 64-bit
    as cdata, SoA over AoS for large index structures, keep hot
    helpers small).
  * Plugin source map; where to extend for a new wire type.
  * Proto2 deferral section: what it would take and why we punt.
51f19633 — Eugene Blikh 3 months ago
docs: handoff brief for the pb.text.decode slice

Self-contained brief targeting the 408 proto3 TextFormatInput tests
still skipped after the encoder + SGROUP work. Covers existing
surface (text.lua encode-only, parser.lua schema parser, conformance
dispatch), grammar buckets the suite exercises (10 categories with
upstream test counts), files to touch, LuaJIT/cdata/box.NULL
conventions, and a verifying-locally checklist. Suggested
recursive-descent shape mirrors runtime/pb/parser.lua's cursor
mechanics.

No code changes; positions the next slice without committing to a
specific implementation schedule.