~bigbes/confluence-md-utilities

3e8e97f02c9e39714adc195299d8499a5b5f5169 — Eugene Blikh a month ago 3c0272a master
fix: round-trip fidelity for inline TOC macros, nbsp emphasis, and strikethrough

Surfaced by `verify` on outside-doc-core-v2.xml (real Confluence XML).

- A TOC macro inside a heading round-trips as `# <!-- ac:toc ... -->`, where
  goldmark parses the comment as inline RawHTML. renderRawHTML now calls
  convertComment (like the block path) so the macro is reconstructed instead of
  leaking back out as a literal comment.
- Confluence stores structured-macro attributes in no fixed order (TOC is
  macro-id,name,schema-version on one page, name,schema-version,macro-id on
  another) and the verify formatter preserves order, so no reconstruction order
  matches everything; sortMacroAttrs canonicalizes them on both sides.
- Bold applied to a lone nbsp (`<strong> </strong>`) emitted `** **`: the
  writeEmphasis trim cutset now includes U+00A0, and normalizeForVerify drops
  whitespace-only emphasis.
- `<s>` and `<del>` (both strikethrough) are canonicalized to `<del>`.

Adds regression tests for each and documents the gotchas in CLAUDE.md.
M CLAUDE.md => CLAUDE.md +3 -1
@@ 65,11 65,13 @@ These are subtle traps the round-trip has been bitten by; touching the related c
- **Nested-list indent depends on the *parent* list type** — CommonMark places content of `1. ` at column 4 and `- ` at column 3, so a nested item must be indented by ≥ the parent marker's width. `xmlConverter.listIndents` is a stack of per-level widths (3 for `<ol>`, 2 for `<ul>` / task-list); `itemIndent()` sums all *ancestor* contributions. Using a fixed `2*(depth-1)` collapses nested ordered lists into siblings. *Tests:* `TestRoundTrip_NestedOrderedList`, `TestRoundTrip_NestedMixedList`, `TestRoundTrip_DoubleNestedUnorderedList`.
- **Tight lists are one stray `\n` away from loose** — a blank line anywhere in a list turns it loose, which makes goldmark wrap every `<li>` in `<p>`. `ensureSingleNewline()` writes `\n` only when the buffer doesn't already end in one, preventing the double newline that appeared when a `<li>`'s body ended with a nested list (which itself terminated in `\n`). Always use it instead of `c.buf.WriteString("\n")` after walking list-item content.
- **Literal angle-bracket text turns into phantom elements** — text stored as `&lt;table&gt;` (a function signature like `box.backup.start(<table>)`) decodes to `<table>` in the text node; emitted raw into Markdown, goldmark parses it as an HTML table and the phantom element swallows the rest of the document. `xml2md.escapeMarkdownAngle` backslash-escapes a *tag-like* `<` (followed by a letter, `/`, `!`, `?`) in **both** text paths — the `walk` TextNode branch *and* `walkChildrenInline` (used for `<li>` bodies, a separate branch that's easy to miss). The matching unescape is in `confluence.Renderer.renderText`: it strips only `\<` (goldmark keeps a backslash-escaped run as one Text node with the backslash intact, and this renderer emits the raw segment) — **never** a blanket `util.UnescapePunctuations`, which would also eat the deliberate `\|` table-pipe escaping and re-break `base.xml`. A non-tag `<` (`a < b`, `<=`) is left alone. *Test:* `TestRoundTrip_LiteralAngleBracketText`.
- **Emphasis can't be flanked by whitespace** — `<strong> x </strong>` (space just inside the tag) renders as literal `** x **` because CommonMark requires the run to be flanked by non-whitespace. `xml2md.writeEmphasis` (used for strong/em/del) captures the children into a scratch buffer and moves leading/trailing whitespace *outside* the markers (` **x** `). The stored and round-tripped forms then differ only in space placement, which `normalizeForVerify` reconciles (`reEmphLeadWS`/`reEmphTrailWS`). *Test:* `TestRoundTrip_EmphasisInteriorSpaces`.
- **Emphasis can't be flanked by whitespace** — `<strong> x </strong>` (space just inside the tag) renders as literal `** x **` because CommonMark requires the run to be flanked by non-whitespace. `xml2md.writeEmphasis` (used for strong/em/del) captures the children into a scratch buffer and moves leading/trailing whitespace *outside* the markers (` **x** `). Its trim cutset **must include U+00A0** — Confluence stores stray bold applied to a lone nbsp (`<strong> </strong>`), and an nbsp left as content emits `** **`, which round-trips as literal `** **`; `normalizeForVerify` drops the empty emphasis (`reEmptyStrong`/`reEmptyEm`/`reEmptyDel`) and reconciles the surviving space placement (`reEmphLeadWS`/`reEmphTrailWS`). *Tests:* `TestRoundTrip_EmphasisInteriorSpaces`, `TestRoundTrip_EmphasisNbspOnly`.
- **A macro marker comment can be inline, not just block** — a TOC macro that lived inside a heading (`<h1><ac:structured-macro ac:name="toc"/></h1>`) round-trips as `# <!-- ac:toc … -->`, where goldmark parses the comment as *inline* RawHTML. `confluence.Renderer.renderRawHTML` must therefore call `convertComment` (the same reconstruction the block path `renderHTMLBlock` uses) before falling back to `convertRawSpan`, or the macro leaks back out as a literal comment. *Test:* `TestRoundTrip_TocMacroInHeading`.

These live in the formatter / verify layer rather than the converter, but bite the same way:

- **Some XML carries values Markdown has no slot for** — `<ac:task-id>` (Confluence renumbers them every save), `<ul style="list-style-type: square;">` bullet styling, and a stray trailing nbsp before `</ac:task-body>` cannot survive a Markdown round-trip and aren't meaningful content. `normalizeForVerify` canonicalizes them on **both** sides (`reTaskID` zeroes the id, `reListStyleAttr` drops `<ul>/<ol>` attributes, `reTrailWSTaskBody` strips the trailing space — note the whitespace class must include `\x{00a0}`, the actual byte stored). *Tests:* `TestNormalizeForVerify_TaskID`, `TestNormalizeForVerify_ListStyleAttr`, `TestNormalizeForVerify_TaskBodyTrailingWS`, `TestNormalizeForVerify_EmphasisWhitespace`.
- **Structured-macro attribute order isn't canonical** — Confluence emits a TOC macro's attributes as `macro-id,name,schema-version` on one page and `name,schema-version,macro-id` on another, and the verify formatter preserves order, so no fixed reconstruction order matches everything. `normalizeForVerify.sortMacroAttrs` sorts the attributes of every `<ac:structured-macro …>` opener on both sides. Likewise `<s>` ↔ `<del>` (both strikethrough — xml2md emits `~~…~~`, md2xml renders `<del>`) is canonicalized via `reStrikeOpen`/`reStrikeClose`. *Tests:* `TestNormalizeForVerify_MacroAttrOrder`, `TestNormalizeForVerify_StrikeTag`, `TestNormalizeForVerify_EmptyEmphasis`.

- **A "visible char" class in regexes must exclude `<`, not just whitespace** — `verify.normalizeForVerify`'s `reBareTextAfterHeading` wraps bare text between a heading and a following block in `<p>…</p>`. Its "≥1 visible char" token must be `[^\s<]`, never `\S`: `\S` matches `<`, so the capture swallows the next tag's opening bracket (e.g. `<ac:structured-macro>`) and the inserted `</p>` lands between a code macro's open tag and its body — destroying round-trip for a no-language fenced code block after a heading. *Tests:* `TestNormalizeForVerify_BareTextAfterHeading`, `TestVerifyRoundTrip_CodeBlockAfterHeading`.
- **`fmt` line-wrapping must keep `<code>…</code>` whole** — breaking a line inside an inline code span injects the continuation indent into the span's text, which `xml2md` collapses into a spurious space *inside* the span (`` ` app.manifest.lock` ``). `format.splitAtoms` emits an entire `<code>…</code>` as one unbreakable atom. Significant whitespace between adjacent inline elements (`</strong> <code>`) is carried on each atom's `spaceBefore` flag — never via `strings.Fields`, which drops trailing space. *Tests:* `TestPrettyXML_WrapKeepsCodeSpanWhole`, `TestPrettyXML_WrapPreservesSpaceBeforeCodeAfterWord`, `TestPrettyXML_SpaceBetweenInlineElements`.

M cmd/mdcx/verify.go => cmd/mdcx/verify.go +50 -0
@@ 5,6 5,7 @@ import (
	"io"
	"os"
	"regexp"
	"sort"
	"strings"

	"github.com/spf13/cobra"


@@ 149,8 150,48 @@ var (
	// Markdown emphasis cannot be flanked by whitespace) to one shape.
	reEmphLeadWS  = regexp.MustCompile(`<(strong|em|del)>([ \t]+)`)
	reEmphTrailWS = regexp.MustCompile(`([ \t]+)</(strong|em|del)>`)
	// reStrikeTag canonicalizes `<s>`/`</s>` to `<del>`/`</del>`. Both are
	// strikethrough; xml2md emits `~~…~~`, which md2xml renders as <del>, so the
	// stored `<s>` and the round-tripped `<del>` must compare equal.
	reStrikeOpen  = regexp.MustCompile(`<s>`)
	reStrikeClose = regexp.MustCompile(`</s>`)
	// reEmptyEmphasis drops an emphasis tag whose entire body is whitespace
	// (including nbsp) — Confluence stores stray bold/italic applied to a lone
	// space, which carries no content and round-trips as bare whitespace.
	reEmptyStrong = regexp.MustCompile(`<strong>([ \t\x{00a0}]*)</strong>`)
	reEmptyEm     = regexp.MustCompile(`<em>([ \t\x{00a0}]*)</em>`)
	reEmptyDel    = regexp.MustCompile(`<del>([ \t\x{00a0}]*)</del>`)
	// reStructMacroOpen matches a `<ac:structured-macro …>` opening tag (self-
	// closing or not). Confluence does not store its attributes in a fixed order
	// — different pages emit a TOC macro as `macro-id,name,schema-version` or
	// `name,schema-version,macro-id` — so the round-trip can't reproduce the
	// original order. sortMacroAttrs canonicalizes it on both sides instead.
	reStructMacroOpen = regexp.MustCompile(`<ac:structured-macro\s+[^>]*?\s*/?>`)
	reTagAttr         = regexp.MustCompile(`[\w:-]+="[^"]*"`)
)

// sortMacroAttrs rewrites every <ac:structured-macro …> opener with its
// attributes sorted, so attribute order never affects the verify comparison.
func sortMacroAttrs(xml string) string {
	return reStructMacroOpen.ReplaceAllStringFunc(xml, func(tag string) string {
		selfClose := strings.HasSuffix(tag, "/>")
		attrs := reTagAttr.FindAllString(tag, -1)
		sort.Strings(attrs)
		var b strings.Builder
		b.WriteString("<ac:structured-macro")
		for _, a := range attrs {
			b.WriteString(" ")
			b.WriteString(a)
		}
		if selfClose {
			b.WriteString("/>")
		} else {
			b.WriteString(">")
		}
		return b.String()
	})
}

// normalizeForVerify strips XML patterns that cannot survive a round-trip
// through Markdown, so verify compares only what the converter can preserve.
// This is applied to BOTH the original input and the round-trip output so that


@@ 190,6 231,15 @@ func normalizeForVerify(xml string) string {
	xml = reListStyleAttr.ReplaceAllString(xml, "<$1>")
	// Strip trailing whitespace before a task-body close.
	xml = reTrailWSTaskBody.ReplaceAllString(xml, "$1")
	// Canonicalize <ac:structured-macro> attribute order.
	xml = sortMacroAttrs(xml)
	// Canonicalize <s> strikethrough to <del>.
	xml = reStrikeOpen.ReplaceAllString(xml, "<del>")
	xml = reStrikeClose.ReplaceAllString(xml, "</del>")
	// Drop emphasis tags whose body is only whitespace.
	xml = reEmptyStrong.ReplaceAllString(xml, "$1")
	xml = reEmptyEm.ReplaceAllString(xml, "$1")
	xml = reEmptyDel.ReplaceAllString(xml, "$1")
	// Move whitespace from just inside an emphasis tag to just outside it.
	xml = reEmphLeadWS.ReplaceAllString(xml, "$2<$1>")
	xml = reEmphTrailWS.ReplaceAllString(xml, "</$2>$1")

M cmd/mdcx/verify_test.go => cmd/mdcx/verify_test.go +19 -0
@@ 208,6 208,25 @@ func TestNormalizeForVerify_EmphasisWhitespace(t *testing.T) {
	assert.Equal(t, `a <strong>x</strong> b`, normalizeForVerify(stored))
}

func TestNormalizeForVerify_MacroAttrOrder(t *testing.T) {
	// Confluence stores TOC-macro attributes in different orders on different
	// pages; sorting them on both sides makes order irrelevant.
	a := `<ac:structured-macro ac:macro-id="X" ac:name="toc" ac:schema-version="1" />`
	b := `<ac:structured-macro ac:name="toc" ac:schema-version="1" ac:macro-id="X" />`
	assert.Equal(t, normalizeForVerify(a), normalizeForVerify(b))
}

func TestNormalizeForVerify_StrikeTag(t *testing.T) {
	// <s> and <del> are both strikethrough; canonicalize to <del>.
	assert.Equal(t, `<del>x</del>`, normalizeForVerify(`<s>x</s>`))
}

func TestNormalizeForVerify_EmptyEmphasis(t *testing.T) {
	// Whitespace-only emphasis (incl. nbsp) is dropped to its bare content.
	assert.Equal(t, "a b", normalizeForVerify("a<strong> </strong>b"))
	assert.Equal(t, "a\u00a0b", normalizeForVerify("a<strong>\u00a0</strong>b"))
}

// --- computeDiffOps ---

func TestComputeDiffOps_IdenticalInputs(t *testing.T) {

M confluence/renderer.go => confluence/renderer.go +24 -14
@@ 300,6 300,19 @@ func (r *Renderer) renderHTMLBlock(w util.BufWriter, source []byte, node ast.Nod
	return ast.WalkSkipChildren, nil
}

// tocMacro builds a TOC structured-macro, emitting attributes in Confluence's
// stored order (macro-id, name, schema-version) so a round-tripped macro matches
// the original byte-for-byte under the verify formatter, which preserves order.
func tocMacro(macroID string) string {
	var b strings.Builder
	b.WriteString("<ac:structured-macro")
	if macroID != "" {
		b.WriteString(` ac:macro-id="` + macroID + `"`)
	}
	b.WriteString(` ac:name="toc" ac:schema-version="1"/>`)
	return b.String()
}

// convertComment converts preserved HTML comments back to Confluence XML.
func (r *Renderer) convertComment(raw string) (string, bool) {
	trimmed := strings.TrimSpace(raw)


@@ 325,23 338,11 @@ func (r *Renderer) convertComment(raw string) (string, bool) {

	// TOC macro (in-p variant: wrap in <p> for original <p><toc/></p> shapes)
	case strings.HasPrefix(trimmed, "<!-- ac:toc-in-p"):
		macroID := extractCommentAttr(trimmed, "macro-id")
		macro := `<ac:structured-macro ac:name="toc" ac:schema-version="1"`
		if macroID != "" {
			macro += ` ac:macro-id="` + macroID + `"`
		}
		macro += "/>"
		return "<p>" + macro + "</p>", true
		return "<p>" + tocMacro(extractCommentAttr(trimmed, "macro-id")) + "</p>", true

	// TOC macro
	case strings.HasPrefix(trimmed, "<!-- ac:toc"):
		macroID := extractCommentAttr(trimmed, "macro-id")
		macro := `<ac:structured-macro ac:name="toc" ac:schema-version="1"`
		if macroID != "" {
			macro += ` ac:macro-id="` + macroID + `"`
		}
		macro += "/>"
		return macro, true
		return tocMacro(extractCommentAttr(trimmed, "macro-id")), true

	// Table attributes — store for next table
	case strings.HasPrefix(trimmed, "<!-- table-attrs:"):


@@ 535,6 536,15 @@ func (r *Renderer) renderRawHTML(w util.BufWriter, source []byte, node ast.Node,
	for i := 0; i < n.Segments.Len(); i++ {
		seg := n.Segments.At(i)
		raw := string(seg.Value(source))
		// A preserved macro marker comment can appear inline, not just at block
		// level — e.g. a TOC macro that lived inside a heading round-trips as
		// `# <!-- ac:toc ... -->`, where goldmark parses the comment as inline
		// RawHTML. Reconstruct the macro here too; otherwise it leaks back out as
		// a literal comment instead of `<ac:structured-macro ac:name="toc"/>`.
		if converted, ok := r.convertComment(raw); ok {
			w.WriteString(converted)
			continue
		}
		w.WriteString(r.convertRawSpan(raw))
	}
	return ast.WalkContinue, nil

M converter/md2xml_test.go => converter/md2xml_test.go +28 -0
@@ 690,6 690,34 @@ func TestRoundTrip_EmphasisInteriorSpaces(t *testing.T) {
	assert.NotContains(t, xmlOut, "**", "asterisks must not leak as literal text")
}

// TestRoundTrip_TocMacroInHeading guards against the TOC macro that lives inside
// a heading round-tripping lossily. xml2md emits it as `# <!-- ac:toc … -->`;
// goldmark parses that comment as *inline* RawHTML, so md2xml must reconstruct
// the structured-macro from the inline path (renderRawHTML), not only the block
// path — otherwise it leaks back out as a literal comment instead of a macro.
func TestRoundTrip_TocMacroInHeading(t *testing.T) {
	xmlIn := `<h1><ac:structured-macro ac:macro-id="f9579fa5-2198-4556-92de-04db447accec" ac:name="toc" ac:schema-version="1" /></h1>`

	md, err := ConfluenceToMarkdown(xmlIn)
	require.NoError(t, err)

	xmlOut, err := MarkdownToConfluence([]byte(md))
	require.NoError(t, err)
	assert.Contains(t, xmlOut, `ac:name="toc"`, "TOC macro must be reconstructed, not left as a comment")
	assert.Contains(t, xmlOut, `ac:macro-id="f9579fa5-2198-4556-92de-04db447accec"`)
	assert.NotContains(t, xmlOut, "<!-- ac:toc", "comment marker must not leak into the XML")
}

// TestRoundTrip_EmphasisNbspOnly guards against bold/italic applied to a lone
// non-breaking space (`<strong> </strong>`, Confluence copy-paste cruft)
// round-tripping as literal `** **`: nbsp must count as whitespace, so the
// emphasis markers are dropped entirely.
func TestRoundTrip_EmphasisNbspOnly(t *testing.T) {
	md, err := ConfluenceToMarkdown("<p>a<strong>\u00a0</strong>b</p>")
	require.NoError(t, err)
	assert.NotContains(t, md, "**", "nbsp-only bold must not emit asterisks")
}

// TestRoundTrip_NestedOrderedList guards against the regression where a
// nested <ol> inside an outer <ol><li> was flattened into a single flat list
// because the markdown was emitted with a 2-space indent (insufficient for

M converter/xml2md.go => converter/xml2md.go +6 -2
@@ 128,9 128,13 @@ func (c *xmlConverter) writeEmphasis(marker string, n *html.Node, depth int) {
	inner := c.buf.String()
	c.buf = saved

	body := strings.TrimLeft(inner, " \t\r\n")
	// The cutset includes U+00A0 (nbsp): Confluence stores stray bold applied to
	// a lone nbsp (`<strong> </strong>`), and treating it as content would
	// emit `** **`, which is not valid emphasis and leaks as literal `** **`.
	const emphWS = " \t\r\n\u00a0"
	body := strings.TrimLeft(inner, emphWS)
	lead := inner[:len(inner)-len(body)]
	core := strings.TrimRight(body, " \t\r\n")
	core := strings.TrimRight(body, emphWS)
	trail := body[len(core):]

	if core == "" {