# Text-format parser brief — `pb.text.decode`

Handoff doc for the next slice of text-format conformance work. Goal:
turn the 408 proto3 `TextFormatInput.*` tests we currently `skipped`
into passes. Read this end-to-end before opening files.

## What exists

- `runtime/pb/text.lua` — encode-only proto3 text-format printer.
  Walks descriptor fields and emits `name: value` / `name { ... }`,
  honors `_unknown_fields` under `opts.print_unknown_fields`. WKT
  emitters for Timestamp/Duration/Struct/Value/ListValue/Any/FieldMask/
  Empty/Wrappers all hook in via `desc.text(t, buf, depth)`.
- `cmd/conformance/core.lua` — TEXT input still returns
  `{skipped = 'text-format input not supported'}`. PB and JSON inputs
  dispatch into `pb.decode` / `pb.json.decode`. TEXT output is wired.
- `runtime/pb/parser.lua` — pure-Lua proto3 *schema* parser (different
  artifact; parses `.proto` files into AST). Useful only as a style
  reference for hand-written Lua recursive-descent.
- `test/text_test.lua` — encoder tests parameterized over both codegen
  modes (`full`, `runtime`). Pattern to mirror for decoder tests.
- `test/conformance_test.lua` — unit regressions that pin each upstream
  conformance case to a `core.handle_request` call. Cheaper than the
  Docker round-trip for the inner dev loop.

## Goal

Add `pb.text.decode(desc, text) -> table` (mirroring
`pb.text.encode(desc, t, opts)`) so:

1. `cmd/conformance/core.lua` stops skipping `text_payload` inputs,
   instead calling `pb.text.decode(desc, req.text_payload)` and feeding
   the result into the existing output dispatch.
2. The proto3 text suite under
   `test/conformance/known_failures_text.txt` is empty *after*
   parser-level conformance is closed (some Recommended-only edge
   cases may legitimately fail and need an entry — fine, mark them).
3. `test/conformance_test.lua` grows one regression per scenario
   bucket (see "Grammar coverage" below) so the inner loop doesn't
   require Docker.

Encoder is already complete for the conformance corpus; do **not**
expand encoder scope unless the conformance probe shows a specific
gap. The only encoder polish in scope: emit textproto-style list
short-form `[a, b, c]` for repeated scalars when `opts.list_short =
true`, so decoded text round-trips cleanly through encode. Default
stays one-line-per-element (the form the current corpus uses).

## Acceptance criteria

- `make test` stays green; the existing 509 unit tests don't regress.
- New unit tests under `test/text_decode_test.lua` cover each grammar
  item in the table below, parameterized over both `full` and
  `runtime` modes against the `hello.Person` and `proto3.TestAllTypes`
  descriptors.
- New regressions under `test/conformance_test.lua` in a
  `core.text_input_*` cluster, one per scenario bucket, pinned with
  byte-exact `text_payload` strings cribbed from the upstream
  conformance source (see "How to extract test payloads" below).
- Docker conformance under default `just conformance` settings:
  binary+JSON suite stays at 1478 ✓; text-format suite climbs from
  8 ✓ / 426 skipped → ~416 ✓ / ~18 skipped (the residual ~18 are
  proto2 message-type skips, unaffected by this work).
- `runtime/pb/text.lua` decoder respects the same conventions as the
  rest of the runtime: LuaJIT `int64_t`/`uint64_t` cdata for 64-bit
  ints; `box.NULL` for the `Value` WKT's `null_value`; `ipairs` /
  `for i = 1, #t` on any hot path (`pairs()` over a hash NYIs the JIT
  trace per CLAUDE.md).

## Files to touch

```
runtime/pb/text.lua             — add decode() and helpers; keep encode untouched
cmd/conformance/core.lua        — drop the text_payload skip, call pb.text.decode
test/text_decode_test.lua       — new, parameterized over modes
test/conformance_test.lua       — add a core.text_input_<bucket> cluster
test/conformance/known_failures_text.txt
                                — empty (or only Recommended residuals)
PLAN.md                         — bump M7 baseline: 8 ✓ → ~416 ✓
runtime/pb/init.lua             — no surface change; pb.text already exposed
```

No codegen-side changes (`cmd/protoc-gen-tarantool/...`) — decode is
descriptor-driven, not inlined.

## Grammar coverage (from a `--verbose` skip-name extract)

The conformance suite stresses these grammar features. Cover each
under both the unit-test set and the conformance regression cluster.
Counts are upstream-test pair counts (input is duplicated × output
formats).

| Bucket | Count | Grammar feature |
|---|---:|---|
| Number literals | ~24 | decimal, hex (`0x…`), octal (`0…` prefix), trailing `f`/`F` for floats, signed exponents (`1e-50`, `1e9999`); `Int{32,64}/Uint{32,64}FieldMaxValue{Octal,Hex,}` all radixes |
| Float specials | ~50 | `inf`, `infinity`, `nan` in arbitrary casings (`NaN`, `iNF`, `INFINITY`); oversize exponents parse to ±inf; `Neg{Float,Double}LargeNegativeExponentParsesAsNegZero` (`-1e-50` → `-0.0`) |
| String/bytes literals | ~24 | `"…"` and `'…'`; escapes `\a \b \f \n \r \t \v \? \\ \' \"`; octal `\341`, hex `\xc0`, short-unicode `ሴ`, long-unicode `\U00010437`; adjacent-literal concat (`"a" "b"` → `"ab"`); UTF-8 validation for `string` fields (reject surrogates, overlongs); `bytes` accepts any byte |
| Aggregates | ~30 | `field: { ... }` and `field: < ... >` (legacy angle-bracket); fields separated by `;`, `,`, or whitespace; trailing `;` accepted; **multiple trailing** `;;` accepted (a corner case the suite tests) |
| Repeated short-form lists | ~30 | `repeated_int32: [123, 456]`, with `,` separator (and trailing `;` after the list). `[a b]` space-only is **invalid**; `[a; b]` semicolons inside lists is **invalid** |
| ReservedFieldName | 20 | any value shape after a reserved-field name (literal, list, message, group-style) must parse and be **silently dropped** — not produce parse_error |
| Map entries | ~8 | `map_<k>_<v> { key: K value: V }` aggregate form, one entry per occurrence |
| Any-typed inline | ~12 | `[type.googleapis.com/<full.name>] { ... }` resolves to an Any with the inner message bytes; the URL syntax inside square brackets is the type-tag |
| Enums | ~4 | accept enum-by-name and enum-by-integer; unknown enum *name* must reject (proto3 rule) |
| Map key types | ~3 | bool keys (`key: true`), int keys, string keys |

A scenario worth flagging: `ListSeparatorMissingIsOneValue_*` —
`field: [123]` followed by a *separate* `field: [456]` produces a
two-element repeated list. Don't collapse the two entries; they're
distinct list-shorts that each append.

## How to extract test payloads

For unit-test regressions, paste the byte-exact `text_payload` from
the upstream conformance source rather than reconstructing:

```bash
# All proto3 text-input test names + their payloads:
docker run --rm -v "$(pwd):/work" -w /work \
    --entrypoint conformance_test_runner <image> \
    --enforce_recommended --verbose \
    --failure_list test/conformance/known_failures.txt \
    --text_format_failure_list test/conformance/known_failures_text.txt \
    /usr/bin/tarantool cmd/conformance-runner.lua 2>&1 \
    | grep '^SKIPPED.*Proto3.*TextFormatInput'
```

Each line carries `test=<name>, request=goo.gle/debugproto <serialized
ConformanceRequest including text_payload>`. The payload after
`text_payload: "` is the input text (C-escaped); unescape it for the
Lua string literal.

Alternatively, `--test <name> --debug` dumps the octal-escaped request
for a single test, which is easier to copy verbatim.

## Constraints / conventions

- **No `pairs()` on hot paths** — the proto3 spec lets fields appear
  in any order, but the parser's outer loop is `while pos < len do`
  over the input string, not iteration over a hash. Field lookup
  inside the loop must be `field_by_name[…]` (already O(1) hash
  lookup, but only one per field-occurrence, not per byte). Map
  fields are the one accepted exception per CLAUDE.md.
- **int64 family stays cdata.** `decode_uint64_token` returns
  `ffi.cast('uint64_t', …)`, not a Lua number. The runtime's `wire.lua`
  has `to_uint64` / `to_int64` helpers — use them for big integer
  literals (esp. `0xFFFFFFFFFFFFFFFF` and `18446744073709551615`).
- **`box.NULL` for nulls.** When the parser encounters `null_value:
  NULL_VALUE` inside a Value WKT, emit `box.NULL`, not `nil` / not a
  custom sentinel — `runtime/pb/wkt.lua` already does this convention.
- **Numeric-field-id mode is optional** — protoc's
  `AllowFieldNumber(true)` is what lets the harness re-parse our
  output. For the parser, accept numeric field IDs by default since
  the conformance harness emits some scenarios that way (e.g.
  unknown-field text round-trips). Skip silently if the ID isn't in
  the schema; same shape as `reserved` handling.
- **Token-recursion depth limits**: textproto recommends rejecting
  deeply nested messages. A 100-level cap mirrors mainline protoc
  and stops pathological inputs from blowing the Lua stack.
- **Reuse `runtime/pb/wire.lua` numeric edge logic.** Don't
  re-derive int32-overflow rules in the parser — produce the value
  as a uint64 cdata and feed through the same truncation helpers the
  binary decoder uses (`wire.varint_to_int32`, etc.).

## Suggested implementation outline

Recommend recursive-descent, not a generated parser. The grammar is
small enough that a hand-written tokenizer + parser stays under ~600
LOC. Mirror `runtime/pb/parser.lua` (the `.proto` schema parser) for
buffer/cursor mechanics: a `lexer` table with `pos`, `src`, `peek()`,
`expect(tok)`, plus skip-whitespace-and-comments helpers (textproto
supports `#` line comments; the conformance corpus tests at least one
comment scenario).

Top-down phases:

1. **Lexer** — number, string-literal (with all escape variants),
   identifier, bracketed-Any URL, `:`, `{`, `}`, `<`, `>`, `[`, `]`,
   `,`, `;`. Single function returning `(token_kind, token_value)`
   and advancing `pos`. Skips whitespace + `#...\n` comments.
2. **Value parser** — given a field's `kind` + `proto_type`, consume
   one value from the lexer. For scalars: type-specific token form
   validation (e.g. floats accept `inf`/`nan`, ints don't). For
   messages: open `{` or `<`, recurse, close.
3. **Field parser** — consume identifier, optional `:`, value, optional
   trailing `;` or `,`. Look up by name in `desc.field_by_name`. If
   field is `reserved` (need to thread reserved set into descriptors;
   currently absent — see "Schema-side prep" below), parse-and-drop.
   If unknown name and `opts.allow_unknown_fields = false`, error;
   otherwise drop.
4. **Top-level entry** — `M.decode(desc, text)` constructs an empty
   result, loops `field_parser` until input exhausted.

WKTs need explicit overrides on the decode path too (Timestamp parses
RFC3339-ish forms when given as `seconds: N nanos: M`, Duration similar,
Value parses the oneof from token shape: `string_value: "..."` vs
`number_value: 1.5` vs `bool_value: true` vs `list_value { ... }` vs
`struct_value { ... }` vs `null_value: NULL_VALUE`).

### Schema-side prep

To pass `ReservedFieldName` we need to know reserved names. They're in
`FileDescriptorProto.MessageType[].ReservedName` (proto descriptor),
which our `cmd/protoc-gen-tarantool` plugin currently ignores. Plumb
`reserved_names = {…}` onto generated descriptors (both modes) and
expose `desc.is_reserved(name) -> bool` for the parser. Plugin change
+ regen via `make gen`.

## Out of scope

- Proto2 / editions message types. Even after this lands, ~1331
  proto2 tests stay skipped because we don't generate Lua for
  `TestAllTypesProto2`. That's a separate axis: generate proto2
  bindings, register the descriptor in `MESSAGE_REGISTRY`, then ~1331
  more tests light up (most will likely pass since the wire format is
  unchanged proto3-shape, but some proto2-specific shapes —
  extensions, groups as a *first-class* feature, required-field rules
  — need work).
- JSPB. Google-internal; nobody implements.
- TextFormat printer changes beyond the optional `list_short` opt.
- Streaming / incremental parse. The conformance corpus payloads are
  small; one-shot string decode is fine.

## Verifying locally

Inner loop (no Docker):

```bash
make gen && make test         # full luatest suite, ~0.5s
# Spot-check one conformance test:
LUA_PATH="./runtime/?/init.lua;./runtime/?.lua;./examples/expected/?.lua;./examples/expected/?/init.lua;./cmd/?.lua;./cmd/?/init.lua;./test/?.lua;;" \
    .rocks/bin/luatest -v test/text_decode_test.lua
```

Full conformance (5–8 min):

```bash
just conformance              # both suites, both failure lists, --enforce_recommended
```

Single test (fast triage):

```bash
docker run --rm -v "$(pwd):/work" -w /work \
    --entrypoint conformance_test_runner \
    tarantool-protobuf-conformance:latest --enforce_recommended \
    --test Required.Proto3.TextFormatInput.ReservedFieldName \
    --failure_list test/conformance/known_failures.txt \
    --text_format_failure_list test/conformance/known_failures_text.txt \
    /usr/bin/tarantool cmd/conformance-runner.lua
```

When the unit + parity tests are green and `just conformance` exits
zero, update `PLAN.md` with the new baseline and commit on top of
this branch (`text-conformance-output`).