#Text-format parser brief — pb.text.decode

Handoff doc for the next slice of text-format conformance work. Goal: turn the 408 proto3 TextFormatInput.* tests we currently skipped into passes. Read this end-to-end before opening files.

#What exists

• runtime/pb/text.lua — encode-only proto3 text-format printer. Walks descriptor fields and emits name: value / name { ... }, honors _unknown_fields under opts.print_unknown_fields. WKT emitters for Timestamp/Duration/Struct/Value/ListValue/Any/FieldMask/ Empty/Wrappers all hook in via desc.text(t, buf, depth).
• cmd/conformance/core.lua — TEXT input still returns {skipped = 'text-format input not supported'}. PB and JSON inputs dispatch into pb.decode / pb.json.decode. TEXT output is wired.
• runtime/pb/parser.lua — pure-Lua proto3 schema parser (different artifact; parses .proto files into AST). Useful only as a style reference for hand-written Lua recursive-descent.
• test/text_test.lua — encoder tests parameterized over both codegen modes (full, runtime). Pattern to mirror for decoder tests.
• test/conformance_test.lua — unit regressions that pin each upstream conformance case to a core.handle_request call. Cheaper than the Docker round-trip for the inner dev loop.

#Goal

Add pb.text.decode(desc, text) -> table (mirroring pb.text.encode(desc, t, opts)) so:

1. cmd/conformance/core.lua stops skipping text_payload inputs, instead calling pb.text.decode(desc, req.text_payload) and feeding the result into the existing output dispatch.
2. The proto3 text suite under test/conformance/known_failures_text.txt is empty after parser-level conformance is closed (some Recommended-only edge cases may legitimately fail and need an entry — fine, mark them).
3. test/conformance_test.lua grows one regression per scenario bucket (see "Grammar coverage" below) so the inner loop doesn't require Docker.

Encoder is already complete for the conformance corpus; do not expand encoder scope unless the conformance probe shows a specific gap. The only encoder polish in scope: emit textproto-style list short-form [a, b, c] for repeated scalars when opts.list_short = true, so decoded text round-trips cleanly through encode. Default stays one-line-per-element (the form the current corpus uses).

#Acceptance criteria

• make test stays green; the existing 509 unit tests don't regress.
• New unit tests under test/text_decode_test.lua cover each grammar item in the table below, parameterized over both full and runtime modes against the hello.Person and proto3.TestAllTypes descriptors.
• New regressions under test/conformance_test.lua in a core.text_input_* cluster, one per scenario bucket, pinned with byte-exact text_payload strings cribbed from the upstream conformance source (see "How to extract test payloads" below).
• Docker conformance under default just conformance settings: binary+JSON suite stays at 1478 ✓; text-format suite climbs from 8 ✓ / 426 skipped → ~416 ✓ / ~18 skipped (the residual ~18 are proto2 message-type skips, unaffected by this work).
• runtime/pb/text.lua decoder respects the same conventions as the rest of the runtime: LuaJIT int64_t/uint64_t cdata for 64-bit ints; box.NULL for the Value WKT's null_value; ipairs / for i = 1, #t on any hot path (pairs() over a hash NYIs the JIT trace per CLAUDE.md).

#Files to touch

runtime/pb/text.lua             — add decode() and helpers; keep encode untouched
cmd/conformance/core.lua        — drop the text_payload skip, call pb.text.decode
test/text_decode_test.lua       — new, parameterized over modes
test/conformance_test.lua       — add a core.text_input_<bucket> cluster
test/conformance/known_failures_text.txt
                                — empty (or only Recommended residuals)
PLAN.md                         — bump M7 baseline: 8 ✓ → ~416 ✓
runtime/pb/init.lua             — no surface change; pb.text already exposed

No codegen-side changes (cmd/protoc-gen-tarantool/...) — decode is descriptor-driven, not inlined.

#Grammar coverage (from a --verbose skip-name extract)

The conformance suite stresses these grammar features. Cover each under both the unit-test set and the conformance regression cluster. Counts are upstream-test pair counts (input is duplicated × output formats).

A scenario worth flagging: ListSeparatorMissingIsOneValue_* — field: [123] followed by a separate field: [456] produces a two-element repeated list. Don't collapse the two entries; they're distinct list-shorts that each append.

#How to extract test payloads

For unit-test regressions, paste the byte-exact text_payload from the upstream conformance source rather than reconstructing:

# All proto3 text-input test names + their payloads:
docker run --rm -v "$(pwd):/work" -w /work \
    --entrypoint conformance_test_runner <image> \
    --enforce_recommended --verbose \
    --failure_list test/conformance/known_failures.txt \
    --text_format_failure_list test/conformance/known_failures_text.txt \
    /usr/bin/tarantool cmd/conformance-runner.lua 2>&1 \
    | grep '^SKIPPED.*Proto3.*TextFormatInput'

Each line carries test=<name>, request=goo.gle/debugproto <serialized ConformanceRequest including text_payload>. The payload after text_payload: " is the input text (C-escaped); unescape it for the Lua string literal.

Alternatively, --test <name> --debug dumps the octal-escaped request for a single test, which is easier to copy verbatim.

#Constraints / conventions

• No pairs() on hot paths — the proto3 spec lets fields appear in any order, but the parser's outer loop is while pos < len do over the input string, not iteration over a hash. Field lookup inside the loop must be field_by_name[…] (already O(1) hash lookup, but only one per field-occurrence, not per byte). Map fields are the one accepted exception per CLAUDE.md.
• int64 family stays cdata. decode_uint64_token returns ffi.cast('uint64_t', …), not a Lua number. The runtime's wire.lua has to_uint64 / to_int64 helpers — use them for big integer literals (esp. 0xFFFFFFFFFFFFFFFF and 18446744073709551615).
• box.NULL for nulls. When the parser encounters null_value: NULL_VALUE inside a Value WKT, emit box.NULL, not nil / not a custom sentinel — runtime/pb/wkt.lua already does this convention.
• Numeric-field-id mode is optional — protoc's AllowFieldNumber(true) is what lets the harness re-parse our output. For the parser, accept numeric field IDs by default since the conformance harness emits some scenarios that way (e.g. unknown-field text round-trips). Skip silently if the ID isn't in the schema; same shape as reserved handling.
• Token-recursion depth limits: textproto recommends rejecting deeply nested messages. A 100-level cap mirrors mainline protoc and stops pathological inputs from blowing the Lua stack.
• Reuse runtime/pb/wire.lua numeric edge logic. Don't re-derive int32-overflow rules in the parser — produce the value as a uint64 cdata and feed through the same truncation helpers the binary decoder uses (wire.varint_to_int32, etc.).

#Suggested implementation outline

Recommend recursive-descent, not a generated parser. The grammar is small enough that a hand-written tokenizer + parser stays under ~600 LOC. Mirror runtime/pb/parser.lua (the .proto schema parser) for buffer/cursor mechanics: a lexer table with pos, src, peek(), expect(tok), plus skip-whitespace-and-comments helpers (textproto supports # line comments; the conformance corpus tests at least one comment scenario).

Top-down phases:

1. Lexer — number, string-literal (with all escape variants), identifier, bracketed-Any URL, :, {, }, <, >, [, ], ,, ;. Single function returning (token_kind, token_value) and advancing pos. Skips whitespace + #...\n comments.
2. Value parser — given a field's kind + proto_type, consume one value from the lexer. For scalars: type-specific token form validation (e.g. floats accept inf/nan, ints don't). For messages: open { or <, recurse, close.
3. Field parser — consume identifier, optional :, value, optional trailing ; or ,. Look up by name in desc.field_by_name. If field is reserved (need to thread reserved set into descriptors; currently absent — see "Schema-side prep" below), parse-and-drop. If unknown name and opts.allow_unknown_fields = false, error; otherwise drop.
4. Top-level entry — M.decode(desc, text) constructs an empty result, loops field_parser until input exhausted.

WKTs need explicit overrides on the decode path too (Timestamp parses RFC3339-ish forms when given as seconds: N nanos: M, Duration similar, Value parses the oneof from token shape: string_value: "..." vs number_value: 1.5 vs bool_value: true vs list_value { ... } vs struct_value { ... } vs null_value: NULL_VALUE).

#Schema-side prep

To pass ReservedFieldName we need to know reserved names. They're in FileDescriptorProto.MessageType[].ReservedName (proto descriptor), which our cmd/protoc-gen-tarantool plugin currently ignores. Plumb reserved_names = {…} onto generated descriptors (both modes) and expose desc.is_reserved(name) -> bool for the parser. Plugin change

• regen via make gen.

#Out of scope

• Proto2 / editions message types. Even after this lands, ~1331 proto2 tests stay skipped because we don't generate Lua for TestAllTypesProto2. That's a separate axis: generate proto2 bindings, register the descriptor in MESSAGE_REGISTRY, then ~1331 more tests light up (most will likely pass since the wire format is unchanged proto3-shape, but some proto2-specific shapes — extensions, groups as a first-class feature, required-field rules — need work).
• JSPB. Google-internal; nobody implements.
• TextFormat printer changes beyond the optional list_short opt.
• Streaming / incremental parse. The conformance corpus payloads are small; one-shot string decode is fine.

#Verifying locally

Inner loop (no Docker):

make gen && make test         # full luatest suite, ~0.5s
# Spot-check one conformance test:
LUA_PATH="./runtime/?/init.lua;./runtime/?.lua;./examples/expected/?.lua;./examples/expected/?/init.lua;./cmd/?.lua;./cmd/?/init.lua;./test/?.lua;;" \
    .rocks/bin/luatest -v test/text_decode_test.lua

Full conformance (5–8 min):

just conformance              # both suites, both failure lists, --enforce_recommended

Single test (fast triage):

docker run --rm -v "$(pwd):/work" -w /work \
    --entrypoint conformance_test_runner \
    tarantool-protobuf-conformance:latest --enforce_recommended \
    --test Required.Proto3.TextFormatInput.ReservedFieldName \
    --failure_list test/conformance/known_failures.txt \
    --text_format_failure_list test/conformance/known_failures_text.txt \
    /usr/bin/tarantool cmd/conformance-runner.lua

When the unit + parity tests are green and just conformance exits zero, update PLAN.md with the new baseline and commit on top of this branch (text-conformance-output).