docs: retire PLAN.md in favor of Beads issue tracking
Replaces the slow-changing monolithic PLAN.md roadmap with the
Beads tracker, which is now the durable source of truth for
project state. Roadmap-and-task-list functions sit in `bd ready`
and `bd show <id>`; cross-codebase invariants stay in CLAUDE.md.
References to PLAN.md in README.md, docs/index.md, and one test
file comment now point at the bd CLI instead. A trailing stale
reference in bench/jit_trace.lua's header comment is cleaned up
in a follow-up alongside its substantive change.
The PLAN.md file's vision/architecture material is already
captured across README.md, CLAUDE.md, and docs/, so nothing
informational is lost.
Closes the goal of bd-8k2.
docs: 100% proto2 + proto3 conformance
README status table: replace the four \"deferred\" rows with
explicit checkmarks for groups, extensions, closed enums; bump
both conformance lines to 2806 and 434 successes. Note MessageSet
as the only known gap (protoreflect rejects upstream, our vendored
proto2 schema has those four nested messages stripped).
docs/codegen.md proto2-support section gets three subsections —
Groups, Extensions, Closed enums — each describing the runtime
surface a user touches (decode_group, _extensions, the closed flag).
The descriptor-table example includes a group field and the new
extensions_by_id / extensions_by_full_name indices.
PLAN.md M9 closes with the conformance numbers; M5 header updated
to reflect that proto2 is no longer deferred.
docs: proto2 is supported for the core surface
README's status table swaps the single \"proto2 deferred\" row for
four entries that name what works (required / optional / custom
defaults / packed semantics) and what doesn't (extend, extensions,
group, MessageSet) so users can decide before they generate.
docs/codegen.md replaces the proto2 deferral section with a
\"Proto2 support\" page: the three new descriptor attributes
(required, default_value, optional everywhere), the existing
IsPacked() trick that already produces the right unpacked default,
and a concrete list of the four features that still refuse to
generate (with the MessageSet upstream-rejection caveat).
PLAN.md gains M9 — Proto2 baseline. Wire format, codecs, parser,
dynamic, tests are checked off; extend / group / MessageSet stay
open because the conformance schema needs all three and they're
each their own slice.
codegen: preserve descriptor options as `options = { ... }`
Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.
The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.
The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.
UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
docs(readme): vendoring an upstream .proto schema
Consumers vendoring upstream protos (etcd, prometheus, pprof, …) hit
annotation-import resolution that protoc-gen-tarantool can't fix on
its own — mainline protoc rejects files with unresolved imports like
versionpb / google.api / gogoproto. Document the standard workaround:
strip the offending imports + their attached options, optionally
rewrite cross-package imports to a flat layout, then run protoc.
Reference implementation: tarantool-etcd's proto/_strip_annotations.py.
docs: install path, prefix= plugin param, int64 migration patterns
* README: add an "Install" section recommending `tt rocks make` from a
local clone until the canonical remote is published, plus a note in
the rockspec flagging source.url as aspirational (the URL today
resolves to "no such repository").
* README: document the `--tarantool_opt=prefix=<path>` plugin parameter
alongside `(tarantool.lua_package)` — the on-disk + require-string
semantics and how the two compose were previously only discoverable
from main.go and docs/howto/02-module-layout.md.
* README: add "Migrating from a Lua proto library that auto-down-casts
int64" listing the four places cdata int64 needs a tonumber()
wrapper (log format verbs, numeric for-loop bounds, string.format,
table keys) and the 2^53 precision caveat.
docs: full reference + how-to set, migrate Makefile to Justfile
Documentation overhaul that adds the missing user-facing surface:
four reference pages (runtime-api, generated-api, cli, grpc-contract),
twelve how-tos walking from first-message through custom transports,
a troubleshooting page, and a docs/index map. Every how-to references
a runnable artifact under examples/, all of them verified end-to-end.
Build system migration: the Makefile is gone; the Justfile is now
the canonical entry point and absorbs every target. examples/Justfile
ships one recipe per runnable example, forwarded via top-level
'just examples <name>'. The 'examples are part of the documented
surface' convention is pinned in CLAUDE.md, alongside a dedicated
section on updating the conformance harness (PROTOBUF_TAG bumps,
libjsoncpp path drift, new test-category wiring).
Stale-number sweep across README/PLAN/CLAUDE: fixture count 18→10,
test count 613/130→639, wire.lua LOC dropped, M7 marked done.
Descriptor-shape block deduplicated against codegen.md as the
canonical source. gRPC transports spec status reframed from
'draft / decision deferred' to 'shipped contract; external
transports deferred'.
.gitignore picks up *.snap / *.xlog / *.vylog / *.run / *.pid /
512.lock so example state can't leak into the working tree.
codegen: emit strict <Type>_fields / _oneofs constants for lazy view
Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.
Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.
Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
docs: drop text-format parser handoff brief
The pb.text.decode slice landed (7ec8f2b); the handoff brief is no
longer load-bearing. Removes the file and unlinks references from
README.md and PLAN.md.
docs: refresh README + PLAN; add api-modes and codegen notes
README.md
* Status table reflects current state: text-format encode + decode,
-0.0 preservation, strict FieldMask, all-green proto3 conformance.
* Conformance baseline jumped to 1493 / 416 / 0 failures (was
1389 / 0 / 79 in the old table); explains the 1313+18 skipped
tests are all TestAllTypesProto2, deferred separately.
* Layout walks the full runtime/pb/ tree (lazy, text, json, wkt,
grpc, parser, dynamic, fileset, descriptor_pb).
* Generated API section calls out the three-mode design and links
to docs/api-modes.md and docs/codegen.md.
PLAN.md
* Section 2 ("Current state") rewritten — no longer claims M0;
lists what's actually in the codebase.
* M1, M2, M3 marked done with [x] checkboxes (had stale [ ]
markers across items that have been shipping for months).
* M5 conformance numbers updated to current baseline; calls out
the three commits that closed the proto3 suite (text decode,
-0 codec, JSON strict pass).
* M7 text-format parser entry updated: 416 ✓ / 0 failures (was
406 / 10 expected before the codec -0 fix).
* Added an M7 entry for the JSON strict-validation pass (six
classes of relaxation now enforced).
docs/api-modes.md (new)
* When to use full vs runtime (descriptor / reflect) vs lazy.
* Concrete code shapes for each, plus what the generated
Person_encode actually looks like in full mode.
* Descriptor-shape contract that ties all three together.
* Lazy: SoA index rationale, sparse-read vs dense-read trade-offs,
cross-over points from bench numbers.
docs/codegen.md (new)
* Pipeline diagram, CLI options, what gets emitted per .proto.
* Walk through the inline (full) mode emission with annotated
generated code.
* Runtime mode: pb.finalize_message's per-field writer/reader
closures and why they exist.
* Hot-path rules the generated code observes (no pairs(), 64-bit
as cdata, SoA over AoS for large index structures, keep hot
helpers small).
* Plugin source map; where to extend for a new wire type.
* Proto2 deferral section: what it would take and why we punt.
license: BSD 2-Clause
Matches Tarantool's own license. Closes the "TBD" placeholder in the
README.
json: enable conformance JSON output + close 459 tests
Three coupled changes that turn the JSON output path on for the
conformance harness:
1. string_to_int64 accepts cdata: Tarantool's json.decode parses raw
JSON integer literals outside double range as int64_t/uint64_t cdata,
not Lua numbers. The decoder errored "expected JSON string or number
for int64" on any unquoted 64-bit value (Int64FieldMaxValueNotQuoted
et al). cdata is now cast through directly, preserving precision.
2. encode_message marks output as a map: an empty proto3 message
serialized as `[]` because Tarantool's json defaults empty tables to
array shape. jsoncpp's strict comparator threw Json::LogicError and
aborted the whole suite. Setting __serialize='map' on the output
gives `{}` and unblocks all JsonOutput tests.
3. PB_CONFORMANCE_SKIP_JSON gate is opt-in by default. core.lua now
matches exactly "1" (so docker -e VAR= disables it), and the
Dockerfile no longer hard-codes "=1" — JSON output runs end-to-end
for everyone unless they re-enable the gate.
Conformance moves from 930 / 1869 / 11 to 1389 / 1313 / 79
(successes / skipped / expected fails). The 75 new expected fails
are canonical-form edge cases (Duration formatting sign handling,
Timestamp out-of-range rejection, double precision digits, NaN
canonicalization, JSON-input strict rejection) — left for a follow-up.
Drops 7 entries from test/conformance/known_failures.txt that this
change closes; adds 75 newly-visible ones.
conformance: local Docker pipeline + cdata int64 map dedup
Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.
Six bugs surfaced and got fixed on the way to green:
1. conformance_test_runner uses execv (not execvp): bare `tarantool`
hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
2. The harness strips LUA_PATH from the child: the runner now
self-bootstraps package.path from debug.getinfo(1, 'S').source.
3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
full BUFSIZ before returning, deadlocking against the parent.
setvbuf('no') on stdin/stdout.
4. v34.1 fetches libjsoncpp via CMake FetchContent under
_deps/jsoncpp-build/...; the runtime image now COPYs the matching
.so* and runs ldconfig.
5. The harness's strict jsoncpp comparator crashes on our currently-
imperfect JSON output (enum numerics, map<K,V> shape, oneof
object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
set in the container ENV; host-side `make test` still exercises
the full JSON path.
6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
key map entries (per proto3's "last value wins" semantics) split
across hash buckets even though __eq matches. Codec walks the
map once on insert to find a canonical key, gated by a
precomputed `f.key_dedup` flag so the dedup only fires for
int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
same `for _k in pairs(map) do` walk only when the static key
kind is 64-bit, so string/int32-keyed map decode stays
JIT-traceable.
Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:
- Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
- Text-format suite: 0 ✓ / 430 skipped / 4 expected fails
403/403 luatest green, 19/19 jit-trace gate green.
Initial commit: protoc-gen-tarantool plugin + pb runtime
A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.
Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.