text: wire pb.text into conformance runner TEXT_FORMAT output
The runner short-circuited every TEXT_FORMAT request to `skipped`, even
though pb.text.encode has been encode-capable since M7. Plug it into
cmd/conformance/core.lua so protobuf/JSON input → text output exercises
the existing encoder end-to-end. Text-format input remains deferred
(pb.text is encode-only).
Text-format suite: 0 ✓ / 430 skipped / 4 expected fails →
2 ✓ / 426 skipped / 6 expected fails (Scalar/Message
*_Print added — unknown-field rendering still missing).
json: enable conformance JSON output + close 459 tests
Three coupled changes that turn the JSON output path on for the
conformance harness:
1. string_to_int64 accepts cdata: Tarantool's json.decode parses raw
JSON integer literals outside double range as int64_t/uint64_t cdata,
not Lua numbers. The decoder errored "expected JSON string or number
for int64" on any unquoted 64-bit value (Int64FieldMaxValueNotQuoted
et al). cdata is now cast through directly, preserving precision.
2. encode_message marks output as a map: an empty proto3 message
serialized as `[]` because Tarantool's json defaults empty tables to
array shape. jsoncpp's strict comparator threw Json::LogicError and
aborted the whole suite. Setting __serialize='map' on the output
gives `{}` and unblocks all JsonOutput tests.
3. PB_CONFORMANCE_SKIP_JSON gate is opt-in by default. core.lua now
matches exactly "1" (so docker -e VAR= disables it), and the
Dockerfile no longer hard-codes "=1" — JSON output runs end-to-end
for everyone unless they re-enable the gate.
Conformance moves from 930 / 1869 / 11 to 1389 / 1313 / 79
(successes / skipped / expected fails). The 75 new expected fails
are canonical-form edge cases (Duration formatting sign handling,
Timestamp out-of-range rejection, double precision digits, NaN
canonicalization, JSON-input strict rejection) — left for a follow-up.
Drops 7 entries from test/conformance/known_failures.txt that this
change closes; adds 75 newly-visible ones.
conformance: local Docker pipeline + cdata int64 map dedup
Wires up the Google protobuf conformance harness as a local target.
docker/conformance.Dockerfile builds conformance_test_runner from
upstream protobuf v34.1 source (matching the host's libprotoc 34.1)
and bundles Tarantool 3 from the official installer. `just conformance`
regenerates Lua, then runs the harness against cmd/conformance-runner.lua
with the repo mounted as a volume.
Six bugs surfaced and got fixed on the way to green:
1. conformance_test_runner uses execv (not execvp): bare `tarantool`
hits ENOENT. Pass /usr/bin/tarantool in CMD and Justfile.
2. The harness strips LUA_PATH from the child: the runner now
self-bootstraps package.path from debug.getinfo(1, 'S').source.
3. C-stdio buffering on pipe stdin made io.stdin:read(n) wait for a
full BUFSIZ before returning, deadlocking against the parent.
setvbuf('no') on stdin/stdout.
4. v34.1 fetches libjsoncpp via CMake FetchContent under
_deps/jsoncpp-build/...; the runtime image now COPYs the matching
.so* and runs ldconfig.
5. The harness's strict jsoncpp comparator crashes on our currently-
imperfect JSON output (enum numerics, map<K,V> shape, oneof
object form). Gate JSON output behind PB_CONFORMANCE_SKIP_JSON=1,
set in the container ENV; host-side `make test` still exercises
the full JSON path.
6. Codec bug — LuaJIT hashes cdata int64 by pointer, so duplicate-
key map entries (per proto3's "last value wins" semantics) split
across hash buckets even though __eq matches. Codec walks the
map once on insert to find a canonical key, gated by a
precomputed `f.key_dedup` flag so the dedup only fires for
int64/uint64/sint64/fixed64/sfixed64 keys. inline.go emits the
same `for _k in pairs(map) do` walk only when the static key
kind is 64-bit, so string/int32-keyed map decode stays
JIT-traceable.
Watchlists at test/conformance/known_failures.txt (binary + JSON) and
test/conformance/known_failures_text.txt (text-format) hold the
deferred failures. Current baseline:
- Binary + JSON suite: 803 ✓ / 1864 skipped / 139 expected fails
- Text-format suite: 0 ✓ / 430 skipped / 4 expected fails
403/403 luatest green, 19/19 jit-trace gate green.
codegen: EmmyLua type annotations for messages, enums, wrappers
Generated Lua modules now carry lua-language-server type annotations:
---@alias <full.Enum> integer per enum
---@class <full.Message> per message
---@field <name> <type> per field
---@param / ---@return per wrapper
Mappings:
bool -> boolean
string / bytes -> string
float / double -> number
all int kinds -> integer (64-bit cdata typed as integer;
LSP has no cdata model)
enum / message -> <full.Name> (resolves to declared alias/class)
repeated T -> T[]
map<K,V> -> table<K, V>
Presence markers (trailing `?` on field name):
proto3 explicit optional
oneof branches (only one is set at a time)
Wrapper signatures cover _new / _encode / _decode / _decode_lazy plus
_has_<field> / _clear_<field> on optional fields. _decode_lazy returns
pb.MessageView, which is declared inline in runtime/pb/lazy.lua along
with pb.ArrayView and pb.MapView so cross-file references resolve in
any project that requires('pb.lazy').
Class identifiers use proto full names verbatim (e.g. `hello.Person`)
so cross-file imports and WKT references both resolve to a single
declared `---@class` block — no per-module renaming needed.
Pure comment addition: 300/300 luatest + 19/19 jit-trace gate stay
green. Generated examples regenerated and committed.
docs: PLAN.md lazy perf numbers after SoA refactor
The lazy entry's perf claims were written against the per-segment-table
implementation. Update with the post-SoA numbers — passthrough 1.6–1.9×,
sparse read 0.90–1.16×, mutate+reencode 1.09–1.26× — and drop the
caveat about sparse-read losing on flat shapes (no longer true).
docs: PLAN.md M6 lazy view entry
Marks the M6 "lazy view" bullet done with the API surface, conformance
claim (interop byte-equal through decode_lazy:encode), trace stability
(make jit-trace 19/19), and the honest perf characteristic from
bench/lazy_bench.lua — lazy wins passthrough re-encode (1.0–1.5×),
loses sparse-read 0.60–0.77× on the flat shape because per-segment
table allocation dominates index cost. Mutate-then-reencode is
roughly break-even.
Lazy is a byte-passthrough optimization, not a universal speedup.
M6: trace stability gate + two fixes
Add `make jit-trace` (`bench/jit_trace.lua`) — a standalone tarantool
script that attaches a `jit.attach('trace')` listener over each hot
encode/decode path and asserts no aborts in our source files fall into
the fatal set (NYI bytecode, blacklisting, persistent type instability).
Runs outside luatest because on macOS arm64 the test framework exhausts
JIT mcode pages before the test body runs, masking real abort reasons.
Two fixes shipped to make all 13 scenarios pass:
- `decode_varint` grew a 1-byte fast path. Before, calling it from a
hot decode loop pulled an inner `while true do` into the caller's
root trace, which got blacklisted after enough retries.
- `pb.finalize_message` now precomputes `desc.oneofs_list` (array
form) and the runtime-mode codec iterates it with ipairs instead
of `pairs(desc.oneofs)`. `pairs()` over a hash-keyed table compiles
to bytecode ISNEXT, which is NYI in LuaJIT 2.1.
The gate also reports interpreter-bridge counts as a benchmark-quality
metric. Decoders show 0-4 bridges per run depending on JIT timing —
caused by side traces returning from inlined `decode_varint` calls,
which LuaJIT 2.1 can't stitch back cleanly. Small per-call overhead on
the multi-byte slow path, structural to the engine.
Scope caveat: map fields encode via `pairs()` and remain off-trace —
pinned by the gate's last scenario so we notice if upstream lifts the
restriction.
Initial commit: protoc-gen-tarantool plugin + pb runtime
A protoc plugin (Go) and a pure-Lua + LuaJIT-FFI runtime that give
Tarantool a complete proto3 + gRPC stack. Two codegen modes (full
inline / runtime descriptor), 226-test luatest suite, 18-fixture
mainline-protoc interop corpus, JSON codec, well-known types,
gRPC client/server factories, runtime .proto parser, microbench
harness with allocation regression gate.
Covers PLAN.md M1-M5. Module is `pb` (not `protobuf`) to avoid
colliding with Tarantool's built-in encode-only `protobuf` module.