docs: proto2 is supported for the core surface
README's status table swaps the single \"proto2 deferred\" row for
four entries that name what works (required / optional / custom
defaults / packed semantics) and what doesn't (extend, extensions,
group, MessageSet) so users can decide before they generate.
docs/codegen.md replaces the proto2 deferral section with a
\"Proto2 support\" page: the three new descriptor attributes
(required, default_value, optional everywhere), the existing
IsPacked() trick that already produces the right unpacked default,
and a concrete list of the four features that still refuse to
generate (with the MessageSet upstream-rejection caveat).
PLAN.md gains M9 — Proto2 baseline. Wire format, codecs, parser,
dynamic, tests are checked off; extend / group / MessageSet stay
open because the conformance schema needs all three and they're
each their own slice.
json,text: presence-tracked fields skip proto3 default elision
Both codecs gated default-value elision on `f.optional or f.oneof` —
the proto3 implicit-presence shape. That elides proto2 `required`
fields set to zero and any other presence-tracked descriptor that
doesn't carry the proto3 explicit-optional flag.
Extend the bypass to `f.required` so a Cardinality{r=0} survives
the round-trip through pb.json.encode / pb.text.encode. 4 new
luatest cases pin the rule.
runtime: parser + dynamic accept proto2 sources
Parser now captures `required=true` and `default_value=…` from the
proto2 keywords (instead of dropping `required` silently and ignoring
field options). Dynamic descriptor builder reads `parsed.syntax`,
flips the repeated-scalar packing default for proto2, and calls
`codec.compile_writers/compile_readers` so the per-field
required-writer specialization actually fires — without that the
generic encode_field path silently elides missing required fields.
64-bit-int defaults are coerced into the appropriate cdata type
inside `coerce_default` so the codec's value-comparison rules
match what generated code emits.
Adds 7 dynamic-mode luatest cases including a static-vs-dynamic
byte-parity check. 725 tests pass.
codegen: proto2 baseline — required, optional, custom defaults
Lifts the proto3-only syntax gate in the plugin and threads three
new field-descriptor attributes through codegen and the codec:
* required=true — fields declared with the proto2 `required` keyword.
Inline codegen and the runtime codec both error when
a required field is missing on encode (vs the silent
elide that proto3 implicit-presence fields get).
* optional=true — already wired for proto3 explicit `optional`; in
proto2 every singular field carries it via the
existing HasOptionalKeyword() check, giving presence
semantics without a separate emission path.
* default_value=… — proto2 [default = X] from the field descriptor,
rendered as a Lua literal (cdata for 64-bit ints,
symbolic name for enums) so consumers can surface
it; the codec itself does not auto-materialize
defaults on decode, matching how proto3 absent
fields stay nil.
Packed-by-default already flips correctly because we ask
protoreflect's `IsPacked()`, which is syntax-aware.
Adds test/proto/proto2_basic.proto with 33 luatest cases covering
required validation, optional presence, custom defaults, the proto2
unpacked-by-default repeated rule, nested-required messages, and
full-vs-runtime mode parity. `just gen-proto2-tests` regenerates the
fixture into examples/expected/{full,runtime}/.
Out of scope: extensions, extend, group; conformance harness still
skips TestAllTypesProto2.
codegen: preserve descriptor options as `options = { ... }`
Every populated *Options message — FileOptions, MessageOptions,
FieldOptions, OneofOptions, EnumOptions, EnumValueOptions,
ServiceOptions, MethodOptions — surfaces on the generated descriptor as
a plain Lua sub-table named `options` (or `oneof_options` /
`value_options` for the per-member shapes). Standard fields use their
proto name as a bare Lua key; extensions use their fully-qualified
extension name as a bracket-quoted string key.
The walker is generic — no extension-specific code paths. Consumers
pull whatever they care about: `(google.api.http)` for REST routing,
`(versionpb.etcd_version_*)` for compatibility gates, `[deprecated =
true]` for migration tooling, and any in-house extension without pb
knowing about them. Standard fields sort alphabetically before
extensions (also alphabetical by full name) so codegen output stays
byte-identical across runs.
The `options` key is only emitted when at least one field is populated,
so option-free protos produce zero-diff output to before. Resolver
re-links *Options* messages so in-file extensions surface via the
protoreflect walker — protogen builds f.Desc before in-file extensions
are registered, and only f.Proto gets the post-pass fix-up, so we
rebuild the resolver manually.
UninterpretedOption is treated as a codegen-time error: a populated
entry means protoc couldn't resolve the extension, and emitting
opaque parser state would hide the problem.
docs(readme): vendoring an upstream .proto schema
Consumers vendoring upstream protos (etcd, prometheus, pprof, …) hit
annotation-import resolution that protoc-gen-tarantool can't fix on
its own — mainline protoc rejects files with unresolved imports like
versionpb / google.api / gogoproto. Document the standard workaround:
strip the offending imports + their attached options, optionally
rewrite cross-package imports to a flat layout, then run protoc.
Reference implementation: tarantool-etcd's proto/_strip_annotations.py.
codegen: bracket-quote Lua-keyword field names
The plugin emitted bare-identifier field names in three positions:
the `M.<Type>_fields` table key, the inline encoder's `v = t.<name>`
load, and the inline decoder's `result.<name> = ...` store. When a
proto field name collided with a Lua reserved word the generated
`*_pb.lua` failed to load with `'(' expected near '<keyword>'`.
The real-world hit is pprof's profile.proto, which declares
`repeated Function function = 5` — that breaks all three emit sites.
Route every user-named identifier through new luaTableKey /
luaFieldAccess helpers that bracket-quote reserved words. Covers
field_names + oneof descriptors, oneof presence pre-pass, inline
encode/decode field bodies (including repeated and map paths), and
optional has/clear accessors.
docs: install path, prefix= plugin param, int64 migration patterns
* README: add an "Install" section recommending `tt rocks make` from a
local clone until the canonical remote is published, plus a note in
the rockspec flagging source.url as aspirational (the URL today
resolves to "no such repository").
* README: document the `--tarantool_opt=prefix=<path>` plugin parameter
alongside `(tarantool.lua_package)` — the on-disk + require-string
semantics and how the two compose were previously only discoverable
from main.go and docs/howto/02-module-layout.md.
* README: add "Migrating from a Lua proto library that auto-down-casts
int64" listing the four places cdata int64 needs a tonumber()
wrapper (log format verbs, numeric for-loop bounds, string.format,
table keys) and the 2^53 precision caveat.
json: M.encode(desc, t, opts) for use_proto_names / emit_defaults / indent
Canonical protojson options were silently ignored. Plumb a single opts
table through encode_message via a CURRENT_ENCODE_OPTS state (mirroring
the decode-side CURRENT_OPTS):
* use_proto_names — emit snake_case field names (proto wire
names) instead of the spec-default
lowerCamelCase.
* emit_defaults — emit zero-valued implicit-presence scalars,
(alias: always_emit_zero_value) empty repeated lists, and empty
maps. Explicit-presence fields (optional /
oneof) and singular message fields remain
absent — matches protobuf-go's behavior.
* indent — pretty-print with the given indent string;
empty arrays/objects stay compact.
Tests parameterize over both codegen modes (descriptor-table contract is
shared between full and runtime) plus a dedicated parity group that
asserts byte-identical JSON across modes for each option.
codegen: resolve (tarantool.lua_package) via global type registry
The custom file option was being dropped silently: protoc encoded it
correctly into the FileDescriptorProto, but protobuf-go parked the
unknown extension in the message's unknown-fields tail because
E_LuaPackage was never registered with protoregistry.GlobalTypes. As a
result proto.GetExtension returned "" and every override the README
documented was a no-op — every caller fell through to the default
"<pkg>.<file>_pb" path resolution.
Register E_LuaPackage in init() and add a luatest regression that
asserts both the output path and the cross-file require strings honor
the option, parameterized over both codegen modes.
json: emit unbroken base64 for bytes fields ({nowrap = true})
Canonical proto3 JSON expects RFC 4648 unwrapped base64, but Tarantool's
`digest.base64_encode` defaults to RFC 2045 MIME-style 76-char line
wrapping. Any `bytes` payload past ~57 bytes used to land in the JSON
string with an embedded `\n`, which breaks every spec-compliant
consumer (grpc-gateway, protojson, protobuf-go's JSON, ...).
Pass `{nowrap = true}` at the two encode sites: the per-field bytes
encoder and the `google.protobuf.Any` opaque-fallback `value` encoder.
Surfaced by tarantool-etcd's `TestJSONGatewayBytesUnwrapped` over Range
responses whose `value` is >=57 bytes; its reference grpc-gateway never
emits the wrapped form.
build: add scm-1 rockspec for the pb runtime
Installs the 12 modules under pb.* from runtime/pb/ via builtin build
type; lua >= 5.1 is the only declared dependency since every other
import (bit, datetime, digest, ffi, fiber, json, protobuf, utf8) is a
Tarantool builtin. Plugin stays out of scope — it's a Go binary built
via the Justfile, not a luarocks artifact. Source URL points at the
planned sourcecraft.dev home that matches go.mod's module path.
docs: track protobuf editions support as a deferred non-goal
PLAN gained a §4.6 design note: what FEATURE_SUPPORTS_EDITIONS would
buy us (per-field packed opt-out is the only proto3 gap), what it
would cost across plugin + parser + descriptor contract, and the
conditions that would force a revisit. Section 8 non-goals now points
at the analysis instead of the bare bullet.
docs: full reference + how-to set, migrate Makefile to Justfile
Documentation overhaul that adds the missing user-facing surface:
four reference pages (runtime-api, generated-api, cli, grpc-contract),
twelve how-tos walking from first-message through custom transports,
a troubleshooting page, and a docs/index map. Every how-to references
a runnable artifact under examples/, all of them verified end-to-end.
Build system migration: the Makefile is gone; the Justfile is now
the canonical entry point and absorbs every target. examples/Justfile
ships one recipe per runnable example, forwarded via top-level
'just examples <name>'. The 'examples are part of the documented
surface' convention is pinned in CLAUDE.md, alongside a dedicated
section on updating the conformance harness (PROTOBUF_TAG bumps,
libjsoncpp path drift, new test-category wiring).
Stale-number sweep across README/PLAN/CLAUDE: fixture count 18→10,
test count 613/130→639, wire.lua LOC dropped, M7 marked done.
Descriptor-shape block deduplicated against codegen.md as the
canonical source. gRPC transports spec status reframed from
'draft / decision deferred' to 'shipped contract; external
transports deferred'.
.gitignore picks up *.snap / *.xlog / *.vylog / *.run / *.pid /
512.lock so example state can't leak into the working tree.
codegen: emit strict <Type>_fields / _oneofs constants for lazy view
Lazy-view callers passing a typo'd field name to :get / :has / :set /
:clear / :which got `nil` back, indistinguishable from a legitimately-
absent optional field. Failures surfaced as missing data downstream.
Each generated message now exports a M.<Type>_fields table mapping
each field name to itself (and M.<Type>_oneofs for oneof groups),
wrapped by a new pb.field_names() helper that errors on unknown-key
reads and on any write. Routing field-name arguments through these
tables turns a typo into a load-time error at the read site.
Eager _encode / _decode keep round-tripping plain Lua tables — the
constants table is a lazy-view contract, documented in
docs/api-modes.md. README and lazy_test.lua converted to the new
pattern; three new tests cover typo / read-only / oneof-typo errors.
codegen: propagate proto comments into generated _pb.lua
Leading `//` comments on messages, enums, enum values, fields, services,
and RPC methods now surface in the generated Lua:
- Message / enum descriptions become `---` blocks above `---@class` /
`---@alias`, so LuaLS shows them on hover.
- Per-field comments collapse onto a trailing `@ description` on the
matching `---@field` line.
- Enum values, service banners, and per-method entries get plain `--`
lines inside the table literals so readers without an LSP still see
the proto-side context.
Covers both `mode=full` and `mode=runtime` (emission is mode-independent
but the new luatest group runs against both).
docs: drop text-format parser handoff brief
The pb.text.decode slice landed (7ec8f2b); the handoff brief is no
longer load-bearing. Removes the file and unlinks references from
README.md and PLAN.md.
docs: spec out msgpack encoding and gRPC transport story
Two design slices captured under docs/specs/ for later pickup:
- msgpack_encoding.md: proto schemas as IDL, msgpack as wire format.
Default mp_map keyed by field number; ARRAY/MP_TUPLE opt-ins for
box.space feeders; WKT bridging to MP_DATETIME / MP_INTERVAL.
- grpc_transports.md: Connect-JSON over HTTP/1.1 as default external
transport, net.box tunnel for in-cluster, explicit no on HTTP/2
termination (recommend Envoy). Conformance anchored on
connectrpc/conformance — same framed-runner shape as the existing
protobuf conformance harness.
PLAN.md cross-links both specs from M4, Open Questions, and Non-goals.
docs: refresh README + PLAN; add api-modes and codegen notes
README.md
* Status table reflects current state: text-format encode + decode,
-0.0 preservation, strict FieldMask, all-green proto3 conformance.
* Conformance baseline jumped to 1493 / 416 / 0 failures (was
1389 / 0 / 79 in the old table); explains the 1313+18 skipped
tests are all TestAllTypesProto2, deferred separately.
* Layout walks the full runtime/pb/ tree (lazy, text, json, wkt,
grpc, parser, dynamic, fileset, descriptor_pb).
* Generated API section calls out the three-mode design and links
to docs/api-modes.md and docs/codegen.md.
PLAN.md
* Section 2 ("Current state") rewritten — no longer claims M0;
lists what's actually in the codebase.
* M1, M2, M3 marked done with [x] checkboxes (had stale [ ]
markers across items that have been shipping for months).
* M5 conformance numbers updated to current baseline; calls out
the three commits that closed the proto3 suite (text decode,
-0 codec, JSON strict pass).
* M7 text-format parser entry updated: 416 ✓ / 0 failures (was
406 / 10 expected before the codec -0 fix).
* Added an M7 entry for the JSON strict-validation pass (six
classes of relaxation now enforced).
docs/api-modes.md (new)
* When to use full vs runtime (descriptor / reflect) vs lazy.
* Concrete code shapes for each, plus what the generated
Person_encode actually looks like in full mode.
* Descriptor-shape contract that ties all three together.
* Lazy: SoA index rationale, sparse-read vs dense-read trade-offs,
cross-over points from bench numbers.
docs/codegen.md (new)
* Pipeline diagram, CLI options, what gets emitted per .proto.
* Walk through the inline (full) mode emission with annotated
generated code.
* Runtime mode: pb.finalize_message's per-field writer/reader
closures and why they exist.
* Hot-path rules the generated code observes (no pairs(), 64-bit
as cdata, SoA over AoS for large index structures, keep hot
helpers small).
* Plugin source map; where to extend for a new wire type.
* Proto2 deferral section: what it would take and why we punt.
json: strict validation pass closes the proto3 conformance suite
Six classes of relaxation that the proto3 JSON conformance corpus
flagged are now enforced. All as Recommended.* tests; combined with
the -0.0 codec fix this empties known_failures.txt and brings the
proto3 binary+JSON suite to 1493 ✓ / 1313 skipped / 0 expected
failures / 0 unexpected failures.
1. Duplicate JSON keys. Tarantool's json.decode is hash-backed and
silently collapses `{"foo":1,"foo":2}` to one entry. A small
byte-walker `find_duplicate_json_keys` runs before json.decode,
tracks per-object brace frames and key sets, errors on the
second occurrence. Closes Recommended.FieldNameDuplicate.
2. camelCase / snake_case aliases of the same proto field appearing
side-by-side. Detected inside decode_message via a `field_seen`
set keyed by proto-name; second hit errors. Closes
FieldNameDuplicateDifferentCasing{1,2}.
3. JSON null inside repeated arrays and map values. Previously
silently dropped; now errors before decode_field_value. Closes
RepeatedField{Message,Primitive}ElementIsNull and
MapFieldValueIsNull.
4. Unknown enum *names* (not integers). decode_enum used to return
nil so callers silently dropped them; now raises by default and
returns nil only when M.decode's `ignore_unknown_fields=true`
opt is set. Conformance dispatch in cmd/conformance/core.lua
forwards this flag when req.test_category ==
JSON_IGNORE_UNKNOWN_PARSING_TEST. Closes
RejectUnknownEnumStringValueIn{Optional,Repeated,Map} and the
paired IgnoreUnknownEnumStringValueIn* tests.
5. google.protobuf.NullValue JSON canonical form. The single enum
value renders as the literal JSON `null` (not the string
"NULL_VALUE"); decode accepts either, encode emits null. The
decode_field_value null-handling path also treats a JSON null
on a NullValue-typed field as "set" rather than "absent" so a
oneof gets marked active. Closes
NullValueInOtherOneof{New,Old}Format.Validator.
6. FieldMask strict round-trip. Path validity is checked on both
sides: the snake_case wire form rejects uppercase letters,
consecutive underscores, trailing underscore, and underscore
followed by anything other than a lowercase letter — these
break the snake↔camel round-trip. The JSON form rejects any
underscore in the input (must be lowerCamelCase). Closes
FieldMask{TooManyUnderscore,PathsDontRoundTrip,
NumbersDontRoundTrip}.JsonOutput and JsonInput.FieldMaskInvalidCharacter.
The pre-existing "drop unknown enum strings" unit regressions in
test/conformance_test.lua were inverted to assert the new error
shape. New strict-validation regressions in test/json_test.lua pin
all six categories so they don't regress; the `json.strict` group
runs across both codegen modes via the shared descriptor table.