~bigbes/confluence-md-utilities

f6c846a1f16d1f6d0045c92a1c0f24429e6b901f — Eugene Blikh a month ago 442bf59
fix: preserve inline spacing in fmt and code-block round-trip

Three related round-trip/formatting bugs, each pinned by a regression test
that fails on the prior code:

- fmt dropped the significant space between two inline elements
  (</strong> <code>): a lone-space text node was always treated as
  indentation. Keep it mid-line; drop it only at a block boundary.
- fmt's line-wrapper dropped a word's trailing space before a tag
  (strings.Fields discards it) and broke inside <code>...</code>,
  injecting indent that xml2md collapses into a spurious in-span space.
  Rewrote wrapLine around an atom model with a per-atom spaceBefore flag
  and keeps an inline code span whole.
- verify.normalizeForVerify's reBareTextAfterHeading used \S for its
  "visible char" token; \S matches '<', so the capture swallowed a
  following <ac:structured-macro> and wrapped it in <p>...</p>, splitting
  a code macro from its body and breaking round-trip for a no-language
  fenced code block after a heading. Use [^\s<] instead.

Full RFC-0010 document now verifies as lossless.
5 files changed, 204 insertions(+), 68 deletions(-)

M CLAUDE.md
M cmd/mdcx/verify.go
M cmd/mdcx/verify_test.go
M format/pretty.go
M format/pretty_test.go
M CLAUDE.md => CLAUDE.md +7 -0
@@ 12,6 12,8 @@ just test-v         # verbose tests
go test ./converter/... -run TestRoundTrip  # run specific tests
```

**Every bug fix gets a regression test.** Whenever you fix a bug, add a test that reproduces the exact case and fails on the original code (verify it fails before the fix, passes after). This codebase is a web of subtle round-trip traps — see "Round-trip gotchas" below; each was re-broken at least once before a test pinned it.

## Architecture

**mdcx** converts Markdown ↔ Confluence storage-format XML with round-trip fidelity. The binary lives at `cmd/mdcx/`.


@@ 63,6 65,11 @@ These are subtle traps the round-trip has been bitten by; touching the related c
- **Nested-list indent depends on the *parent* list type** — CommonMark places content of `1. ` at column 4 and `- ` at column 3, so a nested item must be indented by ≥ the parent marker's width. `xmlConverter.listIndents` is a stack of per-level widths (3 for `<ol>`, 2 for `<ul>` / task-list); `itemIndent()` sums all *ancestor* contributions. Using a fixed `2*(depth-1)` collapses nested ordered lists into siblings. *Tests:* `TestRoundTrip_NestedOrderedList`, `TestRoundTrip_NestedMixedList`, `TestRoundTrip_DoubleNestedUnorderedList`.
- **Tight lists are one stray `\n` away from loose** — a blank line anywhere in a list turns it loose, which makes goldmark wrap every `<li>` in `<p>`. `ensureSingleNewline()` writes `\n` only when the buffer doesn't already end in one, preventing the double newline that appeared when a `<li>`'s body ended with a nested list (which itself terminated in `\n`). Always use it instead of `c.buf.WriteString("\n")` after walking list-item content.

These two live in the formatter / verify layer rather than the converter, but bite the same way:

- **A "visible char" class in regexes must exclude `<`, not just whitespace** — `verify.normalizeForVerify`'s `reBareTextAfterHeading` wraps bare text between a heading and a following block in `<p>…</p>`. Its "≥1 visible char" token must be `[^\s<]`, never `\S`: `\S` matches `<`, so the capture swallows the next tag's opening bracket (e.g. `<ac:structured-macro>`) and the inserted `</p>` lands between a code macro's open tag and its body — destroying round-trip for a no-language fenced code block after a heading. *Tests:* `TestNormalizeForVerify_BareTextAfterHeading`, `TestVerifyRoundTrip_CodeBlockAfterHeading`.
- **`fmt` line-wrapping must keep `<code>…</code>` whole** — breaking a line inside an inline code span injects the continuation indent into the span's text, which `xml2md` collapses into a spurious space *inside* the span (`` ` app.manifest.lock` ``). `format.splitAtoms` emits an entire `<code>…</code>` as one unbreakable atom. Significant whitespace between adjacent inline elements (`</strong> <code>`) is carried on each atom's `spaceBefore` flag — never via `strings.Fields`, which drops trailing space. *Tests:* `TestPrettyXML_WrapKeepsCodeSpanWhole`, `TestPrettyXML_WrapPreservesSpaceBeforeCodeAfterWord`, `TestPrettyXML_SpaceBetweenInlineElements`.

### Verifying real documents

`mdcx verify <file.xml>` formats the input, round-trips it through `xml2md → md2xml`, formats the output, and diffs them. The justfile in `~/data/1` runs this against five real Confluence pages — exercise it after any change to converter or renderer code, since the unit tests can miss multi-feature interactions (e.g. a task list inside a `<div class="content-wrapper">` inside a table cell).

M cmd/mdcx/verify.go => cmd/mdcx/verify.go +6 -1
@@ 124,7 124,12 @@ var (
	// reBareTextAfterHeading wraps bare text appearing directly between a
	// heading close and a following block tag in <p>...</p>. Confluence accepts
	// both forms but markdown round-trip always paragraphs it.
	reBareTextAfterHeading = regexp.MustCompile(`(?s)(</h[1-6]>)([^<]*?\S[^<]*?)(\s*<(?:table|ul|ol|h[1-6]|ac:|p|hr))`)
	//
	// The "at least one visible char" token is [^\s<], not \S: \S also matches
	// '<', so it would let the group swallow a following tag's opening bracket
	// (e.g. <ac:structured-macro>) and wrap a structural element in <p>...</p> —
	// which splits a code macro open from its body and destroys round-trip.
	reBareTextAfterHeading = regexp.MustCompile(`(?s)(</h[1-6]>)([^<]*?[^\s<][^<]*?)(\s*<(?:table|ul|ol|h[1-6]|ac:|p|hr))`)
)

// normalizeForVerify strips XML patterns that cannot survive a round-trip

M cmd/mdcx/verify_test.go => cmd/mdcx/verify_test.go +54 -0
@@ 6,6 6,9 @@ import (

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"

	"go.bigb.es/confluence-md-utilities/converter"
	"go.bigb.es/confluence-md-utilities/format"
)

// --- normalizeForVerify ---


@@ 117,6 120,57 @@ func TestNormalizeForVerify_AdjacentCodeMerged(t *testing.T) {
	}
}

func TestNormalizeForVerify_BareTextAfterHeading(t *testing.T) {
	tests := []struct {
		name  string
		input string
		want  string
	}{
		{
			name:  "genuine bare text is wrapped",
			input: `<h2>Title</h2>bare text<table><tr><td>x</td></tr></table>`,
			want:  `<h2>Title</h2><p>bare text</p><table><tr><td>x</td></tr></table>`,
		},
		{
			// Regression: \S used to match the '<' of the following tag, so the
			// capture swallowed <ac:structured-macro> and wrapped it in <p>...</p>,
			// splitting a code macro from its body and breaking round-trip.
			name:  "structured-macro after heading is left intact",
			input: "<h3>Heading</h3>\n<ac:structured-macro ac:name=\"code\"><ac:plain-text-body><![CDATA[x]]></ac:plain-text-body></ac:structured-macro>",
			want:  "<h3>Heading</h3>\n<ac:structured-macro ac:name=\"code\"><ac:plain-text-body><![CDATA[x]]></ac:plain-text-body></ac:structured-macro>",
		},
		{
			name:  "paragraph after heading is left intact",
			input: "<h1>T</h1>\n<p>already a paragraph</p>",
			want:  "<h1>T</h1>\n<p>already a paragraph</p>",
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			assert.Equal(t, tt.want, normalizeForVerify(tt.input))
		})
	}
}

// TestVerifyRoundTrip_CodeBlockAfterHeading is the end-to-end regression for the
// no-language fenced code block immediately after a heading: it must survive
// xml2md → md2xml with its body intact.
func TestVerifyRoundTrip_CodeBlockAfterHeading(t *testing.T) {
	xml := "<h3>Heading</h3>\n" +
		`<ac:structured-macro ac:name="code" ac:schema-version="1">` +
		`<ac:plain-text-body><![CDATA[tt run --config etc/config.yaml]]></ac:plain-text-body>` +
		`</ac:structured-macro>` + "\n<p>After.</p>"
	norm := normalizeForVerify(xml)
	md, err := converter.ConfluenceToMarkdown(norm)
	require.NoError(t, err)
	rt, err := converter.MarkdownToConfluence([]byte(md))
	require.NoError(t, err)
	assert.Equal(t,
		format.PrettyXML(norm, "  "),
		format.PrettyXML(normalizeForVerify(rt), "  "),
		"code-block body must survive the round-trip")
}

// --- computeDiffOps ---

func TestComputeDiffOps_IdenticalInputs(t *testing.T) {

M format/pretty.go => format/pretty.go +76 -67
@@ 153,10 153,13 @@ func PrettyXML(input string, indent string) string {
				continue
			}
			text := collapseWS(tok.raw)
			if text == "" || text == " " {
			if text == "" {
				i++
				continue
			}
			// A lone space is an indentation artifact only at a block boundary
			// (line start); mid-line it sits between two inline elements
			// (e.g. </strong> <code>) and is a significant space — keep it.
			if atLineStart {
				text = strings.TrimLeft(text, " ")
				if text == "" {


@@ 273,63 276,38 @@ func wrapLine(line string, maxWidth int) []string {
	indentStr := line[:len(line)-len(trimmed)]
	contIndent := indentStr + "  " // continuation lines get extra indent

	// Split into segments: tags (unsplittable) and text (splittable at spaces)
	segments := splitSegments(trimmed)
	// Split into atoms: tags and words, never broken internally. Whitespace
	// between atoms is collapsed to a single significant space recorded in
	// spaceBefore — including space between a word and an adjacent tag
	// (e.g. "</strong> <code>"), which must survive wrapping.
	atoms := splitAtoms(trimmed)

	var lines []string
	var cur strings.Builder
	cur.WriteString(indentStr)
	curWidth := runeWidth(indentStr)

	for _, seg := range segments {
		segW := runeWidth(seg)

		if seg == "" {
			continue
	for _, a := range atoms {
		atomW := runeWidth(a.text)
		spaceW := 0
		if a.spaceBefore {
			spaceW = 1
		}

		// Tags and non-space text: never break inside
		if strings.HasPrefix(seg, "<") {
			// If adding this tag exceeds limit and we have content, wrap
			if curWidth+segW > maxWidth && curWidth > runeWidth(indentStr) {
				lines = append(lines, strings.TrimRight(cur.String(), " "))
				cur.Reset()
				cur.WriteString(contIndent)
				curWidth = runeWidth(contIndent)
			}
			cur.WriteString(seg)
			curWidth += segW
			continue
		// Wrap before this atom if it would overflow and the current line
		// already holds content beyond its indentation.
		if curWidth+spaceW+atomW > maxWidth && curWidth > runeWidth(contIndent) {
			lines = append(lines, strings.TrimRight(cur.String(), " "))
			cur.Reset()
			cur.WriteString(contIndent)
			curWidth = runeWidth(contIndent)
		} else if a.spaceBefore && curWidth > runeWidth(indentStr) {
			cur.WriteByte(' ')
			curWidth++
		}

		// Text segment: split at word boundaries
		words := strings.Fields(seg)
		// Preserve leading space if original had one
		needSpace := len(seg) > 0 && seg[0] == ' '

		for _, word := range words {
			wordW := runeWidth(word)
			spaceW := 0
			if needSpace {
				spaceW = 1
			}

			if curWidth+spaceW+wordW > maxWidth && curWidth > runeWidth(contIndent) {
				lines = append(lines, strings.TrimRight(cur.String(), " "))
				cur.Reset()
				cur.WriteString(contIndent)
				curWidth = runeWidth(contIndent)
				needSpace = false
			}

			if needSpace {
				cur.WriteByte(' ')
				curWidth++
			}
			cur.WriteString(word)
			curWidth += wordW
			needSpace = true
		}
		cur.WriteString(a.text)
		curWidth += atomW
	}

	if cur.Len() > 0 {


@@ 345,28 323,59 @@ func wrapLine(line string, maxWidth int) []string {
	return lines
}

// splitSegments breaks text into alternating tag and text segments.
// E.g. "Hello <strong>world</strong> end" -> ["Hello ", "<strong>", "world", "</strong>", " end"]
func splitSegments(s string) []string {
	var segs []string
// atom is an unbreakable unit of a line — a tag (<...>) or a word — together
// with whether whitespace separated it from the previous atom.
type atom struct {
	text        string
	spaceBefore bool
}

// splitAtoms breaks a line into tags and words, recording the inter-atom
// whitespace as a single space flag. E.g. "Hello <strong>world</strong> end"
// -> {Hello}, { <strong>}, {world}, {</strong>}, { end}.
func splitAtoms(s string) []atom {
	var atoms []atom
	pendingSpace := false
	for len(s) > 0 {
		lt := strings.Index(s, "<")
		if lt == -1 {
			segs = append(segs, s)
			break
		}
		if lt > 0 {
			segs = append(segs, s[:lt])
		}
		gt := strings.Index(s[lt:], ">")
		if gt == -1 {
			segs = append(segs, s[lt:])
			break
		r := s[0]
		switch {
		case r == ' ' || r == '\t' || r == '\n' || r == '\r':
			pendingSpace = true
			s = s[1:]
		case r == '<':
			end := strings.Index(s, ">")
			if end == -1 {
				atoms = append(atoms, atom{s, pendingSpace})
				return atoms
			}
			// Keep an inline code span whole. A line break inside
			// <code>...</code> would inject the continuation indent into the
			// code's text content, which xml2md then collapses into a spurious
			// space inside the span (e.g. "` app.manifest.lock`").
			if strings.HasPrefix(s, "<code>") || strings.HasPrefix(s, "<code ") {
				if c := strings.Index(s, "</code>"); c != -1 {
					spanEnd := c + len("</code>")
					atoms = append(atoms, atom{s[:spanEnd], pendingSpace})
					pendingSpace = false
					s = s[spanEnd:]
					continue
				}
			}
			atoms = append(atoms, atom{s[:end+1], pendingSpace})
			pendingSpace = false
			s = s[end+1:]
		default:
			// A word runs until the next space or tag start.
			i := strings.IndexAny(s, " \t\n\r<")
			if i == -1 {
				i = len(s)
			}
			atoms = append(atoms, atom{s[:i], pendingSpace})
			pendingSpace = false
			s = s[i:]
		}
		segs = append(segs, s[lt:lt+gt+1])
		s = s[lt+gt+1:]
	}
	return segs
	return atoms
}

func runeWidth(s string) int {

M format/pretty_test.go => format/pretty_test.go +61 -0
@@ 161,3 161,64 @@ func TestPrettyXML_LongLinePreservesTagIntegrity(t *testing.T) {
	assert.NotContains(t, result, "<strong\n")
	assert.NotContains(t, result, "</strong\n")
}

// A single space between two adjacent inline elements is significant and must
// survive formatting. It used to be dropped as a presumed indentation artifact,
// gluing the elements together (e.g. "</strong><code>"). Regression for the
// "missing space before inline code" report.
func TestPrettyXML_SpaceBetweenInlineElements(t *testing.T) {
	cases := []struct{ input, want string }{
		{`<p>Bold <strong>word</strong> <code>code</code> here.</p>`, "</strong> <code>"},
		{`<p>Link <a href="http://x">t</a> <code>code</code> here.</p>`, "</a> <code>"},
		{`<p>Word <em>em</em> <code>code</code>.</p>`, "</em> <code>"},
	}
	for _, tc := range cases {
		result := PrettyXML(tc.input, "  ")
		assert.Contains(t, result, tc.want, tc.input)
	}
}

// The same significant space must also survive when the line is long enough to
// be wrapped: wrapLine used to drop whitespace that sat between a word/tag and
// an adjacent tag, because strings.Fields discarded trailing space and tags
// ignored any pending space.
func TestPrettyXML_WrapPreservesSpaceBeforeTag(t *testing.T) {
	long := strings.Repeat("слово ", 25) // push past the 120-rune wrap width
	input := `<p>` + long + `<strong>жирный</strong> <code>код</code> хвост предложения.</p>`
	result := PrettyXML(input, "  ")
	assert.NotContains(t, result, "</strong><code>")
	assert.Contains(t, result, "</strong> <code>")
}

// The literal reported pattern: a plain word followed by inline code on a line
// long enough to wrap. wrapLine's strings.Fields dropped the word's trailing
// space, gluing it to <code> ("манифеста<code>" instead of "манифеста <code>").
func TestPrettyXML_WrapPreservesSpaceBeforeCodeAfterWord(t *testing.T) {
	long := strings.Repeat("текст ", 25) // push past the 120-rune wrap width
	input := `<p>` + long + `манифеста <code>app.manifest.toml</code> и далее.</p>`
	result := PrettyXML(input, "  ")
	assert.NotContains(t, result, "манифеста<code>")
	assert.Contains(t, result, "манифеста <code>")
}

// A wrap must never fall inside <code>...</code>: doing so injects the
// continuation indent into the code text, which round-trips as a spurious space
// inside the span. The whole span is kept on one line regardless of where the
// boundary lands. Probes several lengths so the wrap point sweeps across the
// code span's open tag.
func TestPrettyXML_WrapKeepsCodeSpanWhole(t *testing.T) {
	for n := 18; n <= 24; n++ {
		input := `<p>` + strings.Repeat("word ", n) + `tail <code>app.manifest.lock</code> rest of it.</p>`
		result := PrettyXML(input, "  ")
		for _, line := range strings.Split(result, "\n") {
			trimmed := strings.TrimSpace(line)
			// No line may start or end mid-span (open without close, or vice versa).
			assert.False(t, strings.HasPrefix(trimmed, "app.manifest.lock"),
				"code content split onto its own line at n=%d: %q", n, line)
			if strings.Contains(line, "<code>") {
				assert.Contains(t, line, "</code>",
					"<code> opened but not closed on same line at n=%d: %q", n, line)
			}
		}
	}
}